Skip banner and navigation tools.

 |  site map

Skip sidebars of related page links.

Local Data Reprocessing

When you download quick-look or archived Swift data from the UKSDDC, you have the option to download the SDC processed data, or locally reprocessed data. The difference between these is only in the XRT data.

SDC-processed data

The SDC-processed data are those provided by the NASA Swift Data Center (SDC), which for uniformity are available from the SDC, ASDC (Italy) and our website. These are not guaranteed to have been processed with the most recent calibration and software. Also, older datasets are not routinely reprocessed, so the software and calibration versions used for the SDC data are not necessarily consistent across the archive. At the current time (2017 November) a ‘grand reprocessing’ of the entire archive using HEASoft v6.17 is underway, but will take years to complete.

The software and CALDB versions used in the SDC processing are stored as SOFTVER and CALDBVER FITS keywords in the data files; these will also be reported on the data download page, provided the version is the same for all of the datasets selected. The SDC standard advice is that users should reprocess the data themselves after downloading, using xrtpipeline.

Locally-processed data

At the UKSSDC, as well as serving the SDC-provided data we reprocess the XRT data using a more recent version of HEASoft and the CALDB, and allow you to download this. As part of this we also create exposure maps (with and without vignetting included) to download as part of the standard products. The version of HEASoft used for locally reprocessed data is reported on the download page. Normally the entire archive will be locally reprocessed using the same version of HEASoft, however when a new version of HEASoft is adopted at the UKSSDC, it takes several months for the entire archive to be fully reprocessed; during this time the locally-reprocessed archive will comprise a combination of the old and new software version. You will be warned of this on the download pages, and the LEICSVER keyword in the XRT files indicates the version used.

Normally the locally-reprocessed data will be completely up-to-date, however if for some reason the SDC data are newer than the locally-reprocessed data you will be warned on the download page. For new data, this may indicate a problem or delay in the UKSSDC processing. For older data this typically means that the SDC data have been reprocessed as part of the grand reprocessing. We do not generally reprocess the data locally after such SDC reprocessing because we have already used a newer version of HEASoft than was employed in the SDC reproecssing.

Windowed Timing (WT) mode data have only one-dimensional spatial information, thus to ensure that the source is placed at the correct RA and Dec, for ease of data analysis, xrtpipeline must be given the correct source co-ordinates. The local reprocessing supplies the values in the RA_OBJ and DEC_OBJ FITS keywords. If you are analysing WT mode data therefore you should verify that these values are correct for the source you are interested in, and if not, you should re-run xrtpipeline yourself, even if you downloaded the locally-processed data.

When HEASoft or the CALDB is updated

Not every update to HEASoft affects the XRT data processing, similarly not all CALDB updates affect all XRT data (for example, new gain calibration tends to affect only recent data). After a new release, UKSSDC scientists will assess whether the new version of the software merits a reprocessing of the archive, and whether this should be partial or complete. There is also a period of testing to verify that the new software interfaces as expected with our local analysis tools. Should we decide to reprocess at least some data, this is done in the following order.

Reprocess GRB data
The XRT data are reprocessed in a temporary disk area not available to the public, and the GRB products are rebuilt for all GRBs. These are also placed in temporary storage, not released to the public.
Assess the GRB data
We compare the reprocessed GRB products with the live ones, to verify that everything has worked as expected, and check for any unforeseen issues. We only proceed beyond this step when we are satisfied that the processing using the new software is working correctly.
Move the GRB data and products live
The temporary products are migrated to the live website and archive, so the GRB data and products available for download are the new versions.
Begin reprocessing the entire archive
We do not have sufficient storage to reprocess the entire archive into a temporary area, therefore the live archive is gradually reprocessed. This typically takes a few months, during which time the data on the locally-reprocessed archive will be a mixture of the old and new versions.

If you have any questions about this, please contact the UKSSDC helpdesk on swifthelp@le.ac.uk.