{
"cells": [
{
"cell_type": "markdown",
"id": "b4408cd8",
"metadata": {},
"source": [
"# The `swifttools.ukssdc.data.SXPS` module\n",
"\n",
"This module provides direct access to data in the [LSXPS catalogue](https://www.swift.ac.uk/LSXPS) and some functionality for [2SXPS](https://www.swift.ac.uk/2SXPS). Being part of the [`swifttools.ukssdc.data` module](../data.ipynb) it is designed for use where you already know which elements in the catalogue you want to get data for. If you don't know that, for example you want to select them based on their characteristics, you will need the [SXPS query module](../query/SXPS.ipynb).\n",
"\n",
"I've split the page into sections, linked to from the contents page, so you don't have to read this straight thought but can jump to the point of interest. And (as with the rest of this documentation), this is designed to teach you how to use the API, not to explain details of the SXPS catalogues. For that, you should read (and cite!) the catalogue papers, and view the documentation on the relevant catalogue web pages; both can be accessed via the links above.\n",
"\n",
"OK, first let's import the module, again using a short form to save our fingers and following the pattern adopted throughout:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "57188e36",
"metadata": {},
"outputs": [],
"source": [
"import swifttools.ukssdc.data.SXPS as uds"
]
},
{
"cell_type": "markdown",
"id": "675ba0f5",
"metadata": {},
"source": [
"## Page contents\n",
"\n",
"* [General notes](#intro)\n",
"* [Upper limits](#ul)\n",
"* [Getting the full catalogues](#fullCat)\n",
" * [Old versions of LSXPS tables](#oldTab)\n",
"* [Sources](#sources)\n",
" * [Source details](#sinfo)\n",
" * [Observations lists](#sobs)\n",
" * [Light curves](#lightcurves)\n",
" * [Spectra](#spectra)\n",
" * [Images](#sImages)\n",
" * [XRTProductRequests](#xpr)\n",
"* [Datasets](#datasets)\n",
" * [Dataset details](#dsinfo)\n",
" * [Images](#dsImages)\n",
" * [Superseded stacked images](#ssi)\n",
"* [Transients](#transients)\n",
" * [Transient details](#tsinfo)\n",
" * [Light curves](#tlightcurves)\n",
" * [Spectra](#tspectra)\n",
" * [Images](#tImages)\n",
" * [XRTProductRequests](#txpr)\n",
"\n",
"\n",
"----"
]
},
{
"cell_type": "markdown",
"id": "187afba2",
"metadata": {},
"source": [
"\n",
"## General notes\n",
"\n",
"There are a number of arguments which are common to many of the functions in this module and it's worth covering them here, so that after this you can dip into the sections you care about without having missed anything important.\n",
"\n",
"Firstly is `cat`. This argument lets you specify which catalogue you want to access, the default being 'LSXPS'; at the time of writing the only other one available is '2SXPS' and not all functionality exists for that catalogue. The name `cat` is deliberately chosen to make this tool ambivalent about which side of the Atlantic your spelling originates from.\n",
"\n",
"Secondly is how you identify the object for which you want data, and how those data are indexed when you get them. This is a bit more complicated than it was for [GRBs](GRB.ipynb), because for SXPS we can get data for sources, datasets or transients. The arguments by which the objects are identified are:\n",
"\n",
"* **Sources**: `sourceName` or `sourceID`\n",
"* **Transients**: `sourceName` or `sourceID`\n",
"* **Datasets** : `DatasetID` or `obsID`.\n",
"\n",
"You can only supply one argument, so if, for example, requesting a dataset, you supply either the `DatasetID` *or* the `obsID`, but not both: this would generate an error. Whichever argument you supply you can either give a single value, or a list/tuple of values. This affects how your data are returned, and we'll return to that in just a moment (the fourth point, below).\n",
"\n",
"Thirdly is the argument `bands`. This is not ubiquitous but appears sufficiently often that we should consider it here. It is used to define which of the SXPS energy bands you want data for. This can usually be either the string 'all' (the default) or a list/tuple of band names. These refer to the SXPS bands which are:\n",
"\n",
"* 'total' = 0.3-10 keV\n",
"* 'soft' = 0.3-1 keV\n",
"* 'medium' = 1-2 keV\n",
"* 'hard' = 2-10 keV\n",
"\n",
"In some cases the hardness ratios 'HR1' and 'HR2' can be specified in the `bands` argument as well.\n",
"\n",
"Fourthly we need to note how data are returned. Most functions in this module return a `dict` containing the data requested for the object; for example, `getSourceDetails(sourceID=1)` will return a `dict` of \"sourceProperty\": \"value\" pairs for source 1. But as I noted above, the argument specifying which object to get (`sourceID`, `sourceName` etc.) can be a list or tuple instead of a single value. In this case, the returned `dict` will contains keys for each requested object, and those entries will then contain the `dict` of results. So if we had run `getSourceDetails(sourceID=(1,5))` then the returned `dict` would have had keys '1' and '5', and those would have in turn been `dict`s with the source details. If we had supplied the `sourceName` argument instead of `sourceID`, the keys of our `dict`s would have been the source names. If this is a bit unclear in the abstract don't worry, it will be obvious what we mean when we get into the examples.\n",
"\n",
"For light curves and spectra, data can be saved to disk as well as / instead of being returned as a `dict`. You can control what happens using the arguments `returnData` and `saveData`:\n",
"\n",
"* `returnData` - (default `False`), whether or not the function should return a `dict` containing the data.\n",
"* `saveData` - (default: `True`), whether or not the function should download the data files from the UKSSDC website and write them to disk.\n",
"\n",
"You can set both to `True` to return and save. You can also set both to `False` and waste compute time, network bandwidth and carbon dioxide, but maybe don't do this. And you can `returnData` and then subsequently save what you got, or some subset thereof. If you've read the [GRB page](GRB.ipynb) this will all look familiar; if not, don't worry, I'm not going to assume you have. If you are using `saveData=True` you may also need the `clobber` parameter. This specifies whether existing files should be overwritten, and is `False` by default.\n",
"\n",
"And lastly, many of the functions have an argument `skipErrors` which is `False` by default. This -- and I know it's going to shock you -- tells the function to skip over certain errors and just continue. This is normally use for functions that can act on multiple objects, for example when saving light curves for several sources, and you don't want an error affecting one source or curve to cause the whole process to fail. I won't point this out when it occurs below, as it just adds too much information to an already-dense document, but you can always use `help()` to check whether a function supports this argument.\n",
"\n",
"Right, that was a lot of preliminary, so let's just get to work. We'll start off with calculating an upper limit."
]
},
{
"cell_type": "markdown",
"id": "b105d8e8",
"metadata": {},
"source": [
"\n",
"## Upper limits\n",
"\n",
"You can generate upper limits at any position covered by the SXPS catalogues using the cryptically-named `getUpperLimits()` function. There are quite a lot of parameters you can play with and we'll come to them, but let's jump straight in with a demonstation."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d757a7a1",
"metadata": {},
"outputs": [],
"source": [
"ul = uds.getUpperLimits(position='334.502058 -8.256481', cat='LSXPS')"
]
},
{
"cell_type": "markdown",
"id": "75d3461a",
"metadata": {},
"source": [
"`getUpperLimits()` always returns its result (there is no `saveData` option) so we captured it in the `ul` variable wich is a `dict`, let's explore it:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b2589719",
"metadata": {},
"outputs": [],
"source": [
"ul.keys()"
]
},
{
"cell_type": "markdown",
"id": "1d7cfe79",
"metadata": {},
"source": [
"The first few keys tell us how the `position` argument was parsed, and we'll come to that in a moment. The actual upper limit is contained in 'ULData', which we can examine:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "57666328",
"metadata": {},
"outputs": [],
"source": [
"ul['ULData']"
]
},
{
"cell_type": "markdown",
"id": "8d7e0c7b",
"metadata": {},
"source": [
"This is a pandas `DataFrame` object, and it gives you the upper limit (in the total band) along with some details about where the upper limit came from - see [the online documentation](https://www.swift.ac.uk/LSXPS/uldocs.php) for details.\n",
"\n",
"You may be wondering why this is a `DataFrame`, when there's only a single entry. The answer is that there is only a single entry because we just accepted the default parameters, which returns an upper limit calculated from the dataset with the deepest exposure at the source location (and only in the total band). We can change this behaviour using the parameters `whichData` and `bands`. The latter was discussed in the [General notes](#intro). `whichData` is a string and can be either:\n",
"\n",
"* 'deepest' - get the upper limit from the dataset with the most exposure at the input position (default)\n",
"* 'all' - get upper limits from every dataset covering the input position.\n",
"\n",
"So let's ask instead for upper limits in all bands and all datasets; this will take a few moments longer to calculate:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "21cc0a30",
"metadata": {},
"outputs": [],
"source": [
"ul = uds.getUpperLimits(position='77.969042, -62.324167',\n",
" cat='LSXPS',\n",
" bands='all',\n",
" whichData='all')"
]
},
{
"cell_type": "markdown",
"id": "2342061d",
"metadata": {},
"source": [
"And now let's see what the 'ULData' entry looks like:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f4d05e4f",
"metadata": {},
"outputs": [],
"source": [
"ul['ULData']"
]
},
{
"cell_type": "markdown",
"id": "5f0ddcdf",
"metadata": {},
"source": [
"This time we have more rows (12 when I'm writing this, but LSXPS is living, so this could increase before you run it). We also have a lot more columns because of the `bands='all'` argument, you may need to scroll to the right (or give the command `ul['ULData'].columns`) to see them all, but basically you have upper limits and associated details in all of the bands now.\n",
"\n",
"If you were paying close attention, you'll noticed that I gave a different position as well this time, which I did to show you something else. Let's have a look at our `ul` object returned by the above call:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "09b5709b",
"metadata": {},
"outputs": [],
"source": [
"ul.keys()"
]
},
{
"cell_type": "markdown",
"id": "3af95eaf",
"metadata": {},
"source": [
"Note the new key 'DetData'. If this is present, it means that the input position is near to (within 30\" of) a source in the catalogue you searched. If you already knew this, and wanted an upper limit close to the source then you don't need to worry about this. However, it may be telling you that your position corresponds to a known source, in which case, upper limits are probably not appropriate.\n",
"\n",
"If this key is set, its value tells you about the source(s) in question:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "79228ea3",
"metadata": {},
"outputs": [],
"source": [
"ul['DetData']"
]
},
{
"cell_type": "markdown",
"id": "0af280c0",
"metadata": {},
"source": [
"Here there is a single source 2.4\" from the input position (the \"Distance\" field is always in arcsec), and you would often want to get the light curve for the source using the `getLightCurves()` function ([described below](#lightcurves)), rather than getting upper limits. Of course, you may not want the light curve; maybe you really are placing limits on the flux of a source really near to the catalogued ones, for this reason the upper limits were still given.\n",
"\n",
"There are a few other options that let you control your upper limits:\n",
"\n",
"* `sigma` (default: 3) - sets the confidence level (in Gaussian sigmas) of the desired upper limit.\n",
"* `detThresh` - only used if `detectionsAsRates` is `True`. This sets the confidence level needed to report a detection. By default it is the same as `sigma` but doesn't have to be. For example, you can set `detThresh=2, sigma=3`. This will cause a 3-sigma upper limit to be reported, unless the source is detected at the 2-sigma level, in which case the count-rate will be provided (if you think this is weird, I agree, but this feature was specifically requested, and I'm not here to judge. Much).\n",
"* `detectionsAsRates` (default: `False`) - tells the upper limit server to check if the count-rate value corresponds to a detection, and if so, to report the count-rate and 1-sigma error as well as the upper limit.\n",
"* `skipDetections` (default `False`) - causes datasets in which there is a catalogue detection within 30\" of your input position to be skipped.\n",
"\n",
"Let's do a quick demonstration, because `detectionsAsRates` results in more columns in the output:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ad7d332f",
"metadata": {},
"outputs": [],
"source": [
"ul = uds.getUpperLimits(position='77.969042, -62.324167',\n",
" cat='LSXPS',\n",
" whichData='all',\n",
" detectionsAsRates=True)"
]
},
{
"cell_type": "markdown",
"id": "c417739d",
"metadata": {},
"source": [
"(Note, I removed the `bands` argument so we just get the total band again).\n",
"\n",
"Let's have a look at the results:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "16815120",
"metadata": {},
"outputs": [],
"source": [
"ul['ULData']"
]
},
{
"cell_type": "markdown",
"id": "c299941a",
"metadata": {},
"source": [
"Be sure to scroll to the right to see all the columns, or run the cell below:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4f1e7871",
"metadata": {},
"outputs": [],
"source": [
"ul['ULData'].columns.tolist()"
]
},
{
"cell_type": "markdown",
"id": "bca6ba56",
"metadata": {},
"source": [
"This time, the `DataFrame` has extra columns. 'Total_Rate', 'Total_RatePos' and 'Total_RateNeg' columns are are NaN when the source is undetected, otherwise contains the rate and 1-sigma error. There is also a column 'Total_IsDetected' which is a boolean column indicating whether or not the source was detected in each dataset (a source is 'detected' if the lower-limit on its count rate is >0, at the confidence level set by the `detThresh` parameter, above).\n",
"\n",
"We're nearly done, but there are two more things to mention. \n",
"\n",
"First, let's come back to those 'Resolved' keys in the returned dict:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5d6885b4",
"metadata": {},
"outputs": [],
"source": [
"ul.keys()"
]
},
{
"cell_type": "markdown",
"id": "b9619428",
"metadata": {},
"source": [
"Those keys starting \"Resolved\" tell you how the input position was parsed. Let's look at them:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d0d9473a",
"metadata": {},
"outputs": [],
"source": [
"print(f\"ResolvedRA: {ul['ResolvedRA']}\")\n",
"print(f\"ResolvedDec: {ul['ResolvedDec']}\")\n",
"print(f\"ResolvedInfo: {ul['ResolvedInfo']}\")"
]
},
{
"cell_type": "markdown",
"id": "ea0a5fe5",
"metadata": {},
"source": [
"As you may expect, 'ResolvedRA' and '-Dec' are just the RA and Dec (in decimal degrees), ResolvedInfo gives you some text.\n",
"\n",
"You don't have to use the `position` argument to give the upper limit position, the input options are the same as for a cone search in [the query module](../query.ipynb), that is:\n",
"\n",
"* `position` - as used above, a free-form coordinate string. It tries to support everything I can think of, but if you do something evil (and why would you?) like giving RA in sexagasimal and Dec in decimal then I can't promise it will work (and will be outrageously rude if you complain). But hey, that's why the Resolved* keys are returned, so you can check things.\n",
"* `name` - the name of a catalogued object, that the system will attempt to resolve using some standard services like SIMBAD, MARS and TNS.\n",
"* `RA` and `Dec` - You can explicitly provide the RA and Dec using these arguments, which can receive either a decimal number (assumed to be degrees) or an `astropy.coordinates.Angle` object.\n",
"\n",
"In the latter case the 'Resolved*' keys will not be present in the returned `dict` since no parsing/resolving was needed.\n",
"\n",
"I should also mention the `timeFormat` argument. In all the examples above, the 'StartTime' and 'StopTime' appeared as calendar dates, but you can change this by setting `timeFormat` to one of 'MJD', 'TDB', 'MET' (or 'cal', the default). MJD and TBD are standard timesystems and MET is Swift's Mission Elapsed Time.\n",
"\n",
"Of course, as with everything else, I have tried to provide useful docstrings, so `help(uds.getUpperLimits)` will give you more information."
]
},
{
"cell_type": "markdown",
"id": "aac65dd7",
"metadata": {},
"source": [
"### When upper limits timeout\n",
"\n",
"Calculation of upper limits is not instantaneous, and only a finite number can be found at once. The API waits 5 minutes for a response from the upper limit server. If the limit has not been found in that time, the `dict` returned by `getUpperLimits()` will contain the key 'TIMEOUT'. **Do not despair**! This doesn't mean you can't get an upper limitm it just means you need to wait a moment. The value of the 'TIMEOUT' entry is an identifier that can be used to get your upper limit. So, wait a minute or so, then try:\n",
"\n",
"```\n",
"ul = uds.getFailedUpperLimit(ul['TIMEOUT'])\n",
"```\n",
"\n",
"If the limit has now been calculated then `ul` will now be the normal upper limit `dict`. If not, then it will still have the 'TIMEOUT' key, and you can wait a bit and try again. There are two things to note about this:\n",
"\n",
"1. Upper limits are deleted from the server when retrieved, so `getFailedUpperLimit()` can only return the result once.\n",
"2. Don't let your code wait forever. If there is a server-side failure, your upper limit may always show as timed out. Put some sanity in your loops!\n",
"\n",
"----"
]
},
{
"cell_type": "markdown",
"id": "87557589",
"metadata": {},
"source": [
"### Merging upper limits\n",
"\n",
"If the `whichData='all'` argument was passed, we may then want to combine upper limits from some subset of the results. For this we can use the `mergeUpperLimits()` function. This is actually one of the handful of 'common' functions in the top-level module, so we will need to import that module:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5fbbcc24",
"metadata": {},
"outputs": [],
"source": [
"import swifttools.ukssdc as uk"
]
},
{
"cell_type": "markdown",
"id": "b5620784",
"metadata": {},
"source": [
"I'm not going to go into details here of how this function works, and what all the options are for it, for that you should see the [page describing the common functions](commonFunc.md). Let's just give a simple demo:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "eb693d70",
"metadata": {},
"outputs": [],
"source": [
"merged = uk.mergeUpperLimits(ul['ULData'],\n",
" verbose=True,\n",
" conf=0.95,\n",
" rows=ul['ULData']['ObsID']<\"10000000000\"\n",
" )\n",
"merged"
]
},
{
"cell_type": "markdown",
"id": "f3c205ab",
"metadata": {},
"source": [
"First, note the `rows` argument I supplied, which told the code to merge a subset of the data only. What you actually supply is a `pandas Series` but as demonstrated above, you can use `pandas` filter syntax to create this. In this case I wanted only the individual observations, so I filtered on ObsID because I know stacked images are the only things with obsIDs>1e10 (but as ObsID is a string column, to keep the leading zeroes, I did a string comparison).\n",
"\n",
"Also, just for fun, I requested a 95% confidence (~2 sigma) rather than the normal 3-sigma upper limit (which is the default).\n",
"\n",
"As you can see (because I turned on verbosity), only the total band was merged because we only have the total band data in our upper limit. And the returned data is a `dict` that looks very like a single row of the upper limits supplied."
]
},
{
"cell_type": "markdown",
"id": "8fc4a96c",
"metadata": {},
"source": [
"\n",
"## Getting the full catalogues\n",
"\n",
"If you want to simply download the full catalogue table, you can use the function `getFullCat()`. This requires, as a minimum, which table you want to get."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "960534df",
"metadata": {},
"outputs": [],
"source": [
"uds.getFullTable(table='sources',\n",
" saveData=True,\n",
" destDir='/tmp/APIDemo_SXPS_cat',\n",
" silent=False)"
]
},
{
"cell_type": "markdown",
"id": "4e80eb5c",
"metadata": {},
"source": [
"In the above I also turned off silent output, and specified the directory in which to save things.\n",
"\n",
"`getFullTable()` is one of the functions that takes the options `saveData` and `returnData`. In the above I explicitly gave `saveData=True` even though this is the default. \n",
"\n",
"We could instead have given `returnData=True` which actually downloads the file, reads it into a `DataFrame` and then (unless `saveData=True`), deletes it. By default (unless you specify the `destDir` argument) this file will be briefly stored in a temporary directory. Let's explore this:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a9489000",
"metadata": {},
"outputs": [],
"source": [
"data = uds.getFullTable(table='sources',\n",
" returnData=True,\n",
" saveData=False,\n",
" silent=False,\n",
" verbose=True)"
]
},
{
"cell_type": "markdown",
"id": "8e375317",
"metadata": {},
"source": [
"This is just about the only function in which the return type is not a `dict` but is just the `DataFrame`:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6b94c880",
"metadata": {},
"outputs": [],
"source": [
"data"
]
},
{
"cell_type": "markdown",
"id": "8313c600",
"metadata": {},
"source": [
"There are a few other options, which you can read about via the `help()` function:\n",
"\n",
"* `format` - defaults to 'csv' but can be changed to 'fits', provided that `returnData=False`.\n",
"* `subset` - some tables have subsets (e.g. the sources table has a 'clean' and 'ultra-clean' subset). You can request these. \n",
"\n",
"So:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d82a26b1",
"metadata": {},
"outputs": [],
"source": [
"data = uds.getFullTable(table='datasets',\n",
" destDir='/tmp/APIDemo_SXPS_cat',\n",
" saveData=True,\n",
" subset='ultra-clean',\n",
" format='fits',\n",
" silent=False,\n",
" verbose=True, )"
]
},
{
"cell_type": "markdown",
"id": "2f07e787",
"metadata": {},
"source": [
"I hope the result of this hasn't surprised you!\n",
"\n",
"Since these catalogues are dynamic, the website allows you to access old versions of the catalogues too. This is not yet possible through the API, but will likely be added in the near future."
]
},
{
"cell_type": "markdown",
"id": "db9fe79e",
"metadata": {},
"source": [
"\n",
"### Old versions of LSXPS tables\n",
"\n",
"LSXPS is a dynamic catalogue and as such the tables are in constant flux. For this reason, snapshots of the tables are taken every hour (stored for a limited period of time) and day (stored indefinitely). You may wish to grab one of these old versions. We can do that simply by adding the `epoch` argument to the call above, where `epoch` is the timestamp of the table we want. But how do we find that out? We can use the `listOldTables()` function to see what is available."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2b680dd6",
"metadata": {},
"outputs": [],
"source": [
"oldTabs = uds.listOldTables()\n",
"oldTabs.keys()"
]
},
{
"cell_type": "markdown",
"id": "6e22ffd4",
"metadata": {},
"source": [
"As you can see, this has returned us something with keys 'hourly' and 'daily', and those themselves are `dict`s containing keys giving the epochs for which snapshots exist. These can have a lot of entries so I'll just show the first 5 here:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "599283dd",
"metadata": {},
"outputs": [],
"source": [
"list(oldTabs['hourly'].keys())[0:5]"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e00f060f",
"metadata": {},
"outputs": [],
"source": [
"list(oldTabs['daily'].keys())[0:5]"
]
},
{
"cell_type": "markdown",
"id": "6471a807",
"metadata": {},
"source": [
"If you want to you can look inside those entries too, they list all of the tables available in each snapshot and their filenames, but since every table should be available every time, we shouldn't need to bother.\n",
"\n",
"Any of the keys we've just seen can be supplied as the `epoch` parameter to `getFullTable`, so '2022-08-16 21:00:02 UTC', for example, or '2022-08-16' are both valid; but they must be **exact** matches. So if we want to get a specific table we can do something like:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "00044c72",
"metadata": {},
"outputs": [],
"source": [
"data = uds.getFullTable(table='sources',\n",
" destDir='/tmp/APIDemo_SXPS_cat',\n",
" saveData=True,\n",
" subset='ultra-clean',\n",
" epoch='2022-08-17',\n",
" format='fits',\n",
" silent=False,\n",
" verbose=True, )"
]
},
{
"cell_type": "markdown",
"id": "c51e94aa",
"metadata": {},
"source": [
"Of course, the point of providing these old tables is for reproducibility: so you can come back to some old work and still get at the same catalogue table. This means that in reality, you will probably only run the `listOldTables()` function the first time you work on something that needs a static catalogue, and you will select the most recent version (it's probably best to take a 'daily' one, since they are kept forever). You can make a note of the epoch you obtained, and then when you come back to it later, you don't need to do a look up, you just use that epoch.\n",
"\n",
"---"
]
},
{
"cell_type": "markdown",
"id": "42cb6451",
"metadata": {},
"source": [
"\n",
"## Sources\n",
"\n",
"Before we get into the details of getting products for given a SXPS source, we need to address one thing affecting the LSXPS catalogue: it's dynamic. This means that source names and IDs can change. I'm not going into details now, you can read the paper / catalogue documentation, but we do need to discuss the consequences for this module.\n",
"\n",
"There are basically two ways a source can change.\n",
"\n",
"1. When more data are received, the position changes (hopefully, improves!)\n",
"1. When more data are received, a 'source' is actually resolved into multiple sources.\n",
"\n",
"In the first case, the source name will be changed to match the new position, and its LSXPS_ID will also change. This should not cause a problem; these changes are tracked and so, if you request a source whose identifier has changed, the server can still work out which source you meant, and send its data as requested. And, importantly, if you supplied a list of source names or IDs, the results will be indexed under the name/ID you supplied so as to avoid confusion. I will demonstrate this in a moment, under [source details](#sinfo).\n",
"\n",
"The second case is a bit more complicated, but also much rarer, it generally only occurs in crowded fields with faint sources, or in fields with diffuse emission/extended sources where essentially the detections are all spurious. Obviously, in this case the server cannot work out which source you actually wanted, because there are multiple options. Instead, in this case, the returned `dict` will contain the key 'newerSources', this will list some details of all of the sources that are 'descended' from the one you requested. Let's explore that briefly now -- don't worry about the function being called, just what it returns."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "04140bc7",
"metadata": {},
"outputs": [],
"source": [
"data = uds.getSourceDetails(sourceID=37562,\n",
" cat='LSXPS',\n",
" silent=True,\n",
" verbose=False\n",
" )\n",
"data.keys()"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c7d23a0d",
"metadata": {},
"outputs": [],
"source": [
"data['newerSources']"
]
},
{
"cell_type": "markdown",
"id": "6c89f62b",
"metadata": {},
"source": [
"As you can see, the function we called returned a `dict` with the 'newerSources' key, and that contains a list of `dicts`, each entry giving the source identifiers and positions.\n",
"\n",
"For the `getLightCurves()` and `getSpectra()` functions (discussed below), if called with `returnData=False` then of course you won't get this `dict`; instead you simply won't get any products saved for the affected sources; if `silent=False` you will get a printed warning, however.\n",
"\n",
"----"
]
},
{
"cell_type": "markdown",
"id": "829cb044",
"metadata": {},
"source": [
"\n",
"### Source details\n",
"\n",
"**Note: This functionality only exists for LSXPS**\n",
"\n",
"The first and easiest thing we can retrieve for a specific SXPS source, or set of sources, is the full set of information about that source. We do that very simply, using the `getSourceDetails()` function. This only has the standard set of arguments, `sourceID` or `sourceName` to specify the source(s) to get, `cat` to specify the catalogue (which, as noted, for now at least must be LSXPS, which is the default) and the usual `silent` and `verbose`.\n",
"\n",
"So, let's jump straight in with an example:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "353e96fa",
"metadata": {},
"outputs": [],
"source": [
"data = uds.getSourceDetails(sourceID=11375, cat='LSXPS')"
]
},
{
"cell_type": "markdown",
"id": "28318534",
"metadata": {},
"source": [
"(I said, truly, that `cat` is optional but I think it's helpful to make it explicit).\n",
"\n",
"Let's see what that gave us:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b18affb6",
"metadata": {},
"outputs": [],
"source": [
"data"
]
},
{
"cell_type": "markdown",
"id": "2f18dcdb",
"metadata": {},
"source": [
"Yowsers that's a lot of information! Not all of it will be of interest -- essentially this is everything needed to build the web page for the source.\n",
"\n",
"I'll let you explore it at your leisure, with just a few notes. Firstly, those things which appear as tables on the web page are `DataFrame`s in here. For example, let's look at the information about the datasets in which this source was detected."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8fb4f9e4",
"metadata": {},
"outputs": [],
"source": [
"data['Detections'].keys()"
]
},
{
"cell_type": "markdown",
"id": "7be72034",
"metadata": {},
"source": [
"This tells me how many stacks and normal observations the source was detected in, and then contains the `DataFrames` of the details. Given that there is an 'Observations' entry but no 'Stacks' entry, I can guess that 'NumStacks' will be 0, but let's prove it, then view the observations:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "83e5aa79",
"metadata": {},
"outputs": [],
"source": [
"data['Detections']['NumStacks']"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "db80aa27",
"metadata": {},
"outputs": [],
"source": [
"data['Detections']['Observations']"
]
},
{
"cell_type": "markdown",
"id": "ac56d252",
"metadata": {},
"source": [
"The other thing I want to point out about this relates to the notes above about when source identifiers change. You see, I somewhat sneakily chose a superseded source for this example. We asked for the details of source 11375, and we got source details but:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "acf8c7a5",
"metadata": {},
"outputs": [],
"source": [
"data['LSXPS_ID']"
]
},
{
"cell_type": "markdown",
"id": "9c993170",
"metadata": {},
"source": [
"This source is not source 11375 anymore! But that's OK, this is the source you were looking for, it's just got a new name/number.\n",
"\n",
"Another useful aspect of the data this function returns is that it contains the details of any sources from other catalogues that were found as part of the automated lookup. Unfortunately, source 11375 wasn't very helpful in this regard (there are no external matches) so let's get a different source:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "bca847fc",
"metadata": {},
"outputs": [],
"source": [
"data = uds.getSourceDetails(sourceID=106107, cat='LSXPS')\n",
"data['CrossMatch']"
]
},
{
"cell_type": "markdown",
"id": "f2927825",
"metadata": {},
"source": [
"Remember, this is not an exhaustive list of every catalogued source near the LSXPS soruce.\n",
"\n",
"There's not really much more to say for `getSourceDetails()`, except a reminder that you could have supplied the `sourceName` instead of `sourceID` parameter, and it could have been a list. Actually, let's quickly demo both of these points:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ba962600",
"metadata": {},
"outputs": [],
"source": [
"data = uds.getSourceDetails(sourceName=('LSXPS J163700.6+073914', 'LSXPS J163647.0+074206'),\n",
" cat='LSXPS')"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "eac9c16b",
"metadata": {},
"outputs": [],
"source": [
"data.keys()"
]
},
{
"cell_type": "markdown",
"id": "367a31a7",
"metadata": {},
"source": [
"This result should not be unexpected -- I warned you right at the top of this notebook and it's been seen for the GRB modules -- but if you've started with this notebook it's your first view, so I'll be explicit. We supplied multiple sourceNames, and so what we get back is a `dict` with one entry for each name we supplied, and the contents of each of those is the requested set of information. So:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cdb11fb8",
"metadata": {},
"outputs": [],
"source": [
"data['LSXPS J163700.6+073914']"
]
},
{
"cell_type": "markdown",
"id": "ce77b191",
"metadata": {},
"source": [
"If we had supplied `sourceID=(1,2)` instead of `sourceName=(...)` then the returned `dict` would have keys 1 and 2, instead of the names.\n",
"\n",
"\n",
"----\n",
"\n",
"#### Observation lists\n",
"\n",
"From the lists of detections, non-detections etc in the data returned above, it is possible to derive the list of observations (and targetIDs) corresponding to catalogued dataset covering the source. It's a little bit involved though, as it involves iterating through some of the tables so, because I'm nice, I've given you a function to get this information out:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "383ae7e5",
"metadata": {},
"outputs": [],
"source": [
"what = uds.getObsList(sourceName='LSXPS J073006.9-193710', useObs='allCat')\n",
"what"
]
},
{
"cell_type": "markdown",
"id": "ecd940ce",
"metadata": {},
"source": [
"And you can see it has given a `dict` containing `targList` and `obsList` which are lists of the targetIDs or obsIDs covering the source.\n",
"\n",
"This function can take the normal `sourceID` or `sourceName` arguments, or instead `sourceDetails=data`, where `data` is the value returned by a `getSourceDetails()` call.\n",
"\n",
"The only other argument (beyond the usual `cat`, `silent` and `verbose`) is `useObs`, which tells you which observations of the source you want included in the list. This can be one of (case insensitive):\n",
"\n",
"* `allCat` - get all catalogued observations of the source position.\n",
"* `blind` - get only observations with a blind detection of the source.\n",
"* `allDet` - get observations with blind or retrospective detections.\n",
"* `retro` - get only observations with retrospective detections.\n",
"* `nonDet` - get only observations with no detection (blind or retrospective).\n",
"\n",
"Let me give you one important warning that may sound counter-intuitive at first: it is possible for `useObs='blind'` to return no observations.\n",
"\n",
"This sounds impossible - how can a source be in the catalogue and yet have no blind detections? Well, of course, it can't. But a source can be in the catalogue and have *no single observations in which it is blindly detected*, if it is only detected in a stacked image. I did consider in this situation making `getObs()` return the list of observations making up the stack in which the source was detected, but I decided this would be incorrect: the source was *not* blindly detected in those. You can get that set of observations by setting `useObs='allCat'`, and so of course you can always do a simple check, as in this example:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "eee05f09",
"metadata": {},
"outputs": [],
"source": [
"for sid in (209851, 209220):\n",
" print(f\"Source {sid}:\")\n",
" what = uds.getObsList(sourceID=sid, useObs='blind')\n",
" if len(what['obsList']) == 0:\n",
" print(\"No blind detections found, getting allCat\")\n",
" what = uds.getObsList(sourceID=sid, useObs='allCat')\n",
" print(f\" * Targets: {what['targList']}\\n * Observations: {what['obsList']}\\n\\n\")"
]
},
{
"cell_type": "markdown",
"id": "83f4ea31",
"metadata": {},
"source": [
"#### Downloading the obs data\n",
"\n",
"Having got the obs list you may want to download the actual data, we can do that using the functions introduced the [parent `swifttools.ukssdc.data` module](../data.ipynb) which let us get data either by supplying obsID(s) or targetID(s). So having just got the obslist above, we could do:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "62b231b4",
"metadata": {},
"outputs": [],
"source": [
"import swifttools.ukssdc.data as ud\n",
"ud.downloadObsData(what['obsList'],\n",
" instruments=('xrt',),\n",
" destDir='/tmp/APIDemo_SXPS_data',\n",
" silent=False\n",
" )"
]
},
{
"cell_type": "markdown",
"id": "df592302",
"metadata": {},
"source": [
"Where I have taken advantage of the fact that `what` was set by the previous cell to something without too many observations.\n",
"\n",
"I have not added a wrapper function in `swifttools.ukssdc.data.SXPS` to automatically call this function for a set of objects, you'd have to do that in a loop yourself (it's not hard!). If you think that such an addition would really help you, drop me a line and, if enough people ask, I'll consider it."
]
},
{
"cell_type": "markdown",
"id": "964ad504",
"metadata": {},
"source": [
"\n",
"\n",
"### Light curves\n",
"\n",
"We will get light curves using a function called `getLightCurves()`.\n",
"\n",
"Sources in the SXPS catalogues have light curves created with two different binning methods (one bin per snapshot or one per obsid) and with three options for the units on the time axis (MJD, MET or TDB). This means that when we get light curves, using `getLightCurves()` we will have to specify which binning and time option we want. As well as specifying the object(s) we want to get light curves for of course, using the `sourceName` or `sourceID` argument, as ever.\n",
"\n",
"As noted in the [General notes](#intro), for light curves we have the option of saving them to disk, storing them in a variable, or both, controlled via the standard `saveData` and `returnData` arguments.\n",
"\n",
"We'll start off by exploring how the latter option, getting data into a variable.\n",
"\n",
"(If you have already read the [GRB documentation](GRB.ipynb) and are wondering why I've swapped the order over, it will make sense in a bit.)\n",
"\n",
"#### Storing light curves in a variable\n",
"\n",
"To get the light curve data into a variable we will need to set `returnData=True`, and we'll also set `saveData=False` so that we introduce one thing at a time.\n",
"\n",
"All light curves returned by anything in the `swifttools.ukssdc` module have a common structure which I call a \"light curve `dict`\", and you can [read about this here](https://www.swift.ac.uk/API/ukssdc/structures.md#the-light-curve-dict), but let's get ourselves a light curve `dict` shall we?"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "12331df4",
"metadata": {},
"outputs": [],
"source": [
"lcs = uds.getLightCurves(sourceName='LSXPS J062131.8-622213',\n",
" cat='LSXPS',\n",
" binning='obsid',\n",
" timeFormat='MJD',\n",
" saveData=False,\n",
" returnData=True\n",
" )"
]
},
{
"cell_type": "markdown",
"id": "69d501b2",
"metadata": {},
"source": [
"I'm not going to spend much time unpacking this, because you can [read about the light curve `dict` elsewhere](https://www.swift.ac.uk/API/ukssdc/structures.md#the-light-curve-dict) but we'll take a quick look."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4fb245ac",
"metadata": {},
"outputs": [],
"source": [
"list(lcs.keys())"
]
},
{
"cell_type": "markdown",
"id": "89306dff",
"metadata": {},
"source": [
"If you did read up on the light curve `dict` then this should be as expected. Let's look at some of the details of what we've downloaded."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "47d8ec4c",
"metadata": {},
"outputs": [],
"source": [
"print(f\"TimeFormat: {lcs['TimeFormat']}\")\n",
"print(f\"Binning: {lcs['Binning']}\")\n",
"print(f\"T0: {lcs['T0']}\")"
]
},
{
"cell_type": "markdown",
"id": "4eec3a76",
"metadata": {},
"source": [
"Well, that's a relief, the time format and binning method are what we requested. 'obsid' was a convenient abbreviation for 'Observation' which I provided because I'm nice. You can buy me a beer next time we meet.\n",
"\n",
"The 'Datasets' key, as I'm sure you know, tells you what light curves we actually have, and the entries in this list will correspond to other keys in the `dict`:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4254d225",
"metadata": {},
"outputs": [],
"source": [
"lcs['Datasets']"
]
},
{
"cell_type": "markdown",
"id": "915de674",
"metadata": {},
"source": [
"Hopefully these names are fairly self-explanatory; there are the four SXPS bands and then either '\\_rates' or '\\_UL'. The former contain count-rate bins and 1-sigma errors, the latter contain 3-sigma upper limits. \n",
"\n",
"Let's have a quick look at one each of these:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "86ed2fe9",
"metadata": {},
"outputs": [],
"source": [
"lcs['Total_rates']"
]
},
{
"cell_type": "markdown",
"id": "f11cc7f8",
"metadata": {},
"source": [
"We have a `pandas DataFrame` with the light curve data in it. The upper limit entries are similar:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "39e300b4",
"metadata": {},
"outputs": [],
"source": [
"lcs['Total_UL']"
]
},
{
"cell_type": "markdown",
"id": "c579cad0",
"metadata": {},
"source": [
"The basic difference here being that the 'UpperLimit' column has replaced the 'Rate' and errors.\n",
"\n",
"\n",
"In the above call we got all data in all bands, but we didn't have to, because of the `bands` argument (see the [General notes](#intro)). Let's use it:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7007339a",
"metadata": {},
"outputs": [],
"source": [
"lcs = uds.getLightCurves(sourceName='LSXPS J221755.4-082100',\n",
" cat='LSXPS',\n",
" binning='snapshot',\n",
" timeFormat='TDB',\n",
" bands=('total', 'hard', 'HR1'),\n",
" saveData=False,\n",
" returnData=True\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9f67951a",
"metadata": {},
"outputs": [],
"source": [
"lcs.keys()"
]
},
{
"cell_type": "markdown",
"id": "6d46d41d",
"metadata": {},
"source": [
"You'll note in this case there are no 'UL' datasets, and that's because for this object (FO Aqr, the subject of [my first ever paper](https://ui.adsabs.harvard.edu/abs/2004MNRAS.349..715E/abstract)), there are no upper limits, only detections.\n",
"\n",
"This leads us nicely onto the somewhat thorny question of when something is a rate, and when it's an upper limit and what we get. There are essentially 3 classes of bin in an SXPS light curve:\n",
"\n",
"* Blind detections - These correspond to a dataset in which the source was found by the blind detection used to create the catalogue.\n",
"* Retrospecitve detections - These correspond to a dataset in which the source was NOT found in the blind detection but where forced photometry at the source's location shows a detection (by which I mean that the 3-sigma confidence interval lower limit on the count-rate is >0).\n",
"* Non-detections - Neither of the above.\n",
"\n",
"Just like on the web pages for a given source, this API gives you some control over how the different classes of bin are returned. The default (like on the website) is that blind and retrospective detections are all given as count-rate bins and grouped together in the same curve (e.g. 'Total_rates'), whereas non-detections come as upper limits. If you are happy with this then I strongly advise you to skip ahead a bit, because we're about to dive into a rabbit hole.\n",
"\n",
"You read on? You brave soul.\n",
"\n",
"Seriously though, the above behaviour is a good default, but there are plenty of cases where you will want to drill down into the details a bit more, or customise how you handle data, and I've done my best to make this a) possible and b) easy. It is definitely much easier to do than to explain, but I'll do my best.\n",
"\n",
"As on the website, the API gives you some options about how the different class of bin should be given. Blind detections will always be returned as 'rates' in the light curve, i.e. a bin with a count-rate and 1-sigma confidence errors; for the other two categories, it's up to you - you can get rates or upper limits, or both!\n",
"For the hardness ratio the situation is a little different as we don't run source detection in a hardness ratio (a meaningless concept), and we don't get upper limits for HRs. Instead you have a choice: do you want to get HRs only for datasets with a blind detection of the source, or those with blind or retrospective detections?\n",
"\n",
"As well as choosing how each class of bin is returned, you can decide whether you want to group all rates and ULs together (the default) or not. i.e. in the example we showed above which used all the defaults, there was a single 'Total_rates' light curve and this contained both blind and retrospective detections, but you may well want to keep those separate.\n",
"\n",
"You can control these factors through a set of boolean arguments to `getLightCurves()`. \n",
"\n",
"The easiest is `getAllTypes`. If this is `True` then you will get all possible light curves, and all kept separately. That is, blind detections will be rates in their own dataset. Retrospective detections will be returned as rates, separate to the blind detections, *and* returned a second time, as upper limits. The same will be true of the non-detections, which will also be kept separate from the retrospective ones (this will make more sense when I demonstrate in a moment). `getAllTypes` is `False` by default.\n",
"\n",
"If you don't set `getAllTypes` you can instead control what you get with these arguments:\n",
"\n",
"* `groupRates` (default: `True`) - whether all of bins returned as count-rates and errors should be grouped into the same light curve.\n",
"* `groupULs` (default: `True`) - whether all upper limits (non-detection, and retrospective only) should be grouped into the same light curve.\n",
"* `groupHRs` (default: `True`) - whether all HRs (blind and retrospective dets) should be grouped into the same light curve.\n",
"* `retroAsUL` (default: `False`) - whether to return upper limits, instead of count-rate bins, for retrospective detections\n",
"* `nondetAsUL` (default: `True`) - whether to return upper limits, instead of count-rate bins, for non detections\n",
"* `getRetroHR` (default: `True`) - whether to include HRs from retrospective detections.\n",
"\n",
"Note that the `group*` parameters only group bins within an energy band, the different bands' data will always be kept separate.\n",
"\n",
"Before I move on to some examples, let me quickly outline three more arguments you can set:\n",
"\n",
"* showObs (default: `False`) - whether the obsID for each bin should also be returned as a column in the data.\n",
"* hideBadPup (default: `True`) - whether bins with the \"pupWarn\" flag set should be rejected.\n",
"* mergeAliases (default: `True`) - whether to include points from sources identified as aliases of the selected source.\n",
"\n",
"If you don't know what these mean, you'll have to read the SXPS documentation, or accept the defaults and trust me.\n",
"\n",
"Right, assuming you are still conscious, enthusiastic and ready to go, let's try a few of these options. For ease, I'm always only going to get the total band data in these examples. \n",
"\n",
"Let's start with `getAllTypes`, as it shows us everything!"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "05654f14",
"metadata": {},
"outputs": [],
"source": [
"lcs = uds.getLightCurves(sourceName='LSXPS J051152.3-621925',\n",
" cat='LSXPS',\n",
" bands=('total',),\n",
" binning='obsid',\n",
" timeFormat='MJD',\n",
" saveData=False,\n",
" returnData=True,\n",
" getAllTypes=True\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2497350a",
"metadata": {},
"outputs": [],
"source": [
"lcs['Datasets']"
]
},
{
"cell_type": "markdown",
"id": "8c2eeefa",
"metadata": {},
"source": [
"We have 5 datasets here for the total band which I hope is what you expected. The blind detections are given only as rates, and the retrospective and non-detections appear both as count-rates and as upper limits. **Please be aware** that this means we have duplicated time bins. That is, every dataset with a retrospective detection appears in both the 'Total_retro_rates' *and* 'Total_retro_UL' - because we set `getAllTypes=True`.\n",
"\n",
"If we don't set this flag, we won't get duplicated data, so we can instead select whether retrospective/non detections appear as rates or limit, and whether all rates or limits are grouped into one dataset or not. I'm not going to explore every option here, but let's really explicitly probe the grouping thing:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0e69bfef",
"metadata": {},
"outputs": [],
"source": [
"lcs = uds.getLightCurves(sourceName='LSXPS J051152.3-621925',\n",
" cat='LSXPS',\n",
" bands=('total',),\n",
" binning='obsid',\n",
" timeFormat='MJD',\n",
" saveData=False,\n",
" returnData=True,\n",
" retroAsUL = True,\n",
" groupULs = False \n",
" )"
]
},
{
"cell_type": "markdown",
"id": "77652e47",
"metadata": {},
"source": [
"Here we have asked for the retrospective detections to be given as upper limits. The non-detections will also be upper limits (the default), but we also said `groupULs=False`, so instead of having a `Total_UL` light curve containing the retrospective and non-detections, we expect these to have been kept separate. Let's confirm this:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d876a13c",
"metadata": {},
"outputs": [],
"source": [
"lcs['Datasets']"
]
},
{
"cell_type": "markdown",
"id": "f14caaab",
"metadata": {},
"source": [
"Great! Now, to hammer home this point about grouping: how many of each type of bin was there?"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e4bc4784",
"metadata": {},
"outputs": [],
"source": [
"print(f\"Retro: {len(lcs['Total_retro_UL'])}\")\n",
"print(f\"Nondet: {len(lcs['Total_nondet_UL'])}\")"
]
},
{
"cell_type": "markdown",
"id": "1303d6c2",
"metadata": {},
"source": [
"So if we give the same command as above, but with the grouping of upper limits enabled, we should get a single 'Total_UL' light curve, containing the bins from those two light curves above. Let's prove this:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5f60f40b",
"metadata": {},
"outputs": [],
"source": [
"lcs = uds.getLightCurves(sourceName='LSXPS J051152.3-621925',\n",
" cat='LSXPS',\n",
" bands=('total',),\n",
" binning='obsid',\n",
" timeFormat='MJD',\n",
" saveData=False,\n",
" returnData=True,\n",
" retroAsUL = True,\n",
" groupULs = True, ## This is the default, but I'm being explicit \n",
" )\n",
"lcs['Datasets']"
]
},
{
"cell_type": "markdown",
"id": "3276ecc9",
"metadata": {},
"source": [
"So we now just have a 'Total_UL' dataset with all of the upper limits in it. Let's really verify that, shall we?"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "894e3a21",
"metadata": {},
"outputs": [],
"source": [
"len(lcs['Total_UL'])"
]
},
{
"cell_type": "markdown",
"id": "fa3fe66d",
"metadata": {},
"source": [
"And unless something has broken between me writing this, and you running it, the number above should be the sum of the number from a few cells ago. Obviously, the analogue is true for the rate and HR grouping too but you can explore that in your own time, or we'll never finish this notebook. And there's lots still to get through, starting with:\n",
"\n",
"#### Plotting light curves\n",
"\n",
"Since we have our light curve data in a variable, we can make use of the [module-level `plotLightCurve()` function](https://www.swift.ac.uk/API/ukssdc/commonFunc.md#plotlightcurve) to give us a quick plot. I'm not going to repeat the `plotLightCurve()` documentation here, just show it in action:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0f50b108",
"metadata": {},
"outputs": [],
"source": [
"from swifttools.ukssdc import plotLightCurve"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a86f8639",
"metadata": {},
"outputs": [],
"source": [
"fig, ax = plotLightCurve(lcs, whichCurves=('Total_rates',),\n",
" ylog=True,\n",
" verbose=True)"
]
},
{
"cell_type": "markdown",
"id": "bbaa4c2c",
"metadata": {},
"source": [
"And I'll take this chance to show you something about that function; as well as returning `fig, ax` it can *receive* them if you want to add to the plot. So I may want to plot upper limits as well:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9f92c137",
"metadata": {},
"outputs": [],
"source": [
"fig, ax = plotLightCurve(lcs, whichCurves=('Total_UL',),\n",
" ylog=True,\n",
" verbose=True,\n",
" fig=fig,\n",
" cols={\"Total_UL\": \"blue\"},\n",
" ax=ax)\n",
"fig"
]
},
{
"cell_type": "markdown",
"id": "4c545d46",
"metadata": {},
"source": [
"For reasons I don't fully understand, Jupyter only plots this again if I put the `fig` line at the end of the cell, which is why I did it. I hope you noticed that this function automatically realised that the upper limits were, er, upper limits; I also used the 'cols' argument to specify the colour for my upper limits.\n",
"\n",
"---"
]
},
{
"cell_type": "markdown",
"id": "7c46cd06",
"metadata": {},
"source": [
"#### Saving directly to disk\n",
"\n",
"It may be that you didn't want to handle the data in a variable at all, but just grab the light curves into files on disk. We can do this by calling `getLightCurves(saveData=True)`, like this:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c7c17af9",
"metadata": {},
"outputs": [],
"source": [
"uds.getLightCurves(sourceName='LSXPS J062131.8-622213',\n",
" cat='LSXPS',\n",
" saveData=True,\n",
" binning='obsid',\n",
" timeFormat='MJD',\n",
" destDir='/tmp/APIDemo_SXPS_LC',\n",
" verbose=True,\n",
" )"
]
},
{
"cell_type": "markdown",
"id": "270209ad",
"metadata": {},
"source": [
"I turned `verbose` on so you could see what was happening, and feel free to go and look at those files.\n",
"\n",
"This function actually doesn't do very much itself, most of the work is done by another common function:\n",
"[`saveLightCurveFromDict()`](https://www.swift.ac.uk/API/ukssdc/commonFunc.md#savelightcurvefromdict), and most of the arguments\n",
"you may pass to `saveLightCurves()` are just keyword arguments that will get passed straight through. The\n",
"`uds.getLightCuvres` function will override the default values for the `timeFormatInFname` and `binningInFname`\n",
"arguments for `saveLightCurveFromDict()`, setting them both to ``False`` unless you explicitly specify them. \n",
"\n",
"As well as the arguments we pass to [`saveLightCurveFromDict()`](https://www.swift.ac.uk/API/ukssdc/commonFunc.md#savelightcurvefromdict),\n",
"we can use all of those complicated controls over how data are grouped and formatted that we discussed earlier as well, for example:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "830d9f26",
"metadata": {},
"outputs": [],
"source": [
"uds.getLightCurves(sourceName='LSXPS J062131.8-622213',\n",
" cat='LSXPS',\n",
" saveData=True,\n",
" binning='obsid',\n",
" timeFormat='MJD',\n",
" destDir='/tmp/APIDemo_SXPS_LC2',\n",
" verbose=True,\n",
" getAllTypes=True\n",
" )"
]
},
{
"cell_type": "markdown",
"id": "f84f193c",
"metadata": {},
"source": [
"Let me also demonstrate the `subDirs` issue which only comes into play when we get mutliple light curves. It is `True` by default but I'll be explicit:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8b7dd946",
"metadata": {},
"outputs": [],
"source": [
"uds.getLightCurves(sourceName=('LSXPS J062131.8-622213','LSXPS J051152.3-621925'),\n",
" cat='LSXPS',\n",
" saveData=True,\n",
" binning='obsid',\n",
" timeFormat='MJD',\n",
" destDir='/tmp/APIDemo_SXPS_LC3',\n",
" verbose=True,\n",
" subDirs=True\n",
" )"
]
},
{
"cell_type": "markdown",
"id": "3b05ce1b",
"metadata": {},
"source": [
"As you saw, I asked for light curves for two sources, and each was saved in its own subdirectory. Since I requested sources by name, the names are used to index the directories.\n",
"\n",
"You have probably noticed that the file names are the same for the two sources, which means that if I had `subDirs=False` we'd have an issue, so in that case the source names are prepended to the file names, thus:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8f9d7616",
"metadata": {},
"outputs": [],
"source": [
"uds.getLightCurves(sourceName=('LSXPS J062131.8-622213','LSXPS J051152.3-621925'),\n",
" cat='LSXPS',\n",
" saveData=True,\n",
" binning='obsid',\n",
" timeFormat='MJD',\n",
" destDir='/tmp/APIDemo_SXPS_LC4',\n",
" verbose=True,\n",
" subDirs=False\n",
" )"
]
},
{
"cell_type": "markdown",
"id": "b5f24a91",
"metadata": {},
"source": [
"Notice now that everything is in `/tmp/APIDemo_SXPS_LC3/` but with unique file names."
]
},
{
"cell_type": "markdown",
"id": "936f5370",
"metadata": {},
"source": [
"#### Saving light curves after downloading them.\n",
"\n",
"There is a third way ('yesss, yesss, we found it we did') to deal with light curves, which is that you can download them using the `getLightCurves()` function with `returnData=True` and then save them with a function`saveLightCurves()`. You may wonder why you would want to do this, instead of just letting `getLightCurves()` save the data for you -- the answer is that you may want to only save some of the data you downloaded.\n",
"\n",
"There are two extra arguments we can add when saving this way:\n",
"\n",
"* `whichSources` allows you to specify which of the sources in your lc variable to save.\n",
"* `whichCurves` allows you to specify which of the curves (entries in the 'Datasets' array) to save.\n",
"\n",
"Obviously, the former argument only makes sense if we downloaded a few light curves, so, let's plunge in with one example, covering everything. First, I'm going to download all light curve stuff for three objects."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d79a85b1",
"metadata": {},
"outputs": [],
"source": [
"lcs = uds.getLightCurves(sourceName=('LSXPS J051152.3-621925','LSXPS J221755.4-082100', 'LSXPS J062131.8-622213'),\n",
" cat='LSXPS',\n",
" bands='all',\n",
" binning='obsid',\n",
" timeFormat='MJD',\n",
" saveData=False,\n",
" returnData=True,\n",
" getAllTypes=True\n",
" )"
]
},
{
"cell_type": "markdown",
"id": "bbfd91f0",
"metadata": {},
"source": [
"Let's remind ourself what this looks like:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6000f4dd",
"metadata": {},
"outputs": [],
"source": [
"lcs.keys()"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8915d72d",
"metadata": {},
"outputs": [],
"source": [
"lcs['LSXPS J051152.3-621925']['Datasets']"
]
},
{
"cell_type": "markdown",
"id": "8a2800bf",
"metadata": {},
"source": [
"But now, for whatever reason, I only want to save the light curves of the first and third sources, and I only want to save the total-band blind rates, and the hard band non-detection upper limits."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1be7a3dc",
"metadata": {},
"outputs": [],
"source": [
"uds.saveLightCurves(lcs,\n",
" whichSources=('LSXPS J051152.3-621925','LSXPS J062131.8-622213'),\n",
" whichCurves=('Total_blind_rates', 'Hard_nondet_UL'),\n",
" destDir='/tmp/APIDemo_SXPS_LC5',\n",
" subDirs=True,\n",
" verbose=True\n",
" )"
]
},
{
"cell_type": "markdown",
"id": "b92e55aa",
"metadata": {},
"source": [
"I turned on verbosity so you can see what its doing. \n",
"\n",
"Just a few things to note:\n",
"\n",
"1. The first argument to `saveLightCurves` is the `dict` returned by `getLightCurves()`.\n",
"1. If our `getLightCurves()` call had specified `sourceID` instead of `sourceName` then in the above, `whichCurves` would have been a list of sourceIDs.\n",
"1. Both `whichCurves` and `whichSources` can be 'all', instead of a list; indeed, this is their default value.\n",
"\n",
"For those of you who had read the [GRB documentation](GRB.ipynb) and have been wondering for a while why I introduced getting things into a variable before saving to disk, I'm hoping you may have worked it out now… Essentially the reason is that for GRBs, `saveData=True` literally grabbed light curve files from the server and saved them to disk. You had no control over the format or anything of the files, you got what the server has. For SXPS, there are no such files (the light curves are compiled on the fly from some rather huge database tables), so `saveData=True` is completely identical to doing `returnData=True` and then calling `saveLightCurves()`. I'm sure you were desperate to know that.\n",
"\n",
"Right, that took a while but you'll be glad to know that the worst is out of the way. SXPS source light curves are probably the second most complicated thing in the entire `swifttools.ukssdc` module (the most complicated being the [GRB burst analyser data](GRB.ipynb#ban) and we've got to the end of it. So take a breath, get a drink (I'll have a Hobgoblin Ruby please) and then we can move on..."
]
},
{
"cell_type": "markdown",
"id": "4f6965e2",
"metadata": {},
"source": [
"\n",
"### Spectra\n",
"\n",
"What, we're still not done with sources? No, but nearly!\n",
"\n",
"SXPS sources deemed bright enough have automatically-created and fitted spectra. You can access the data for these in a manner analogous to the light curves: save them directly to disk, return as a variable (or both), or return as a variable and then save to disk!\n",
"\n",
"Spectra also differ a bit from light curves, in that the data we save and the data we get into a variable are not quite the same. When saving data to disk we save the spectral files (ready for use in `xspec`) but we don't return these raw data to a variable due to the specific nature of spectral data. Instead, the variable we return contains information about the spectrum, and its fit.\n",
"\n",
"If you've already read [GRB spectrum information](GRB.ipynb#spectra) you probably don't need to re-read all of this, you just need to know that for SXPS spectra there is only one rname, called 'interval0', and only one mode (PC) but there can be 2 models ('PowerLaw' and 'APEC'). If that didn't make sense to you, you need to read on.\n",
"\n",
"#### Saving spectra to disk\n",
"\n",
"We get spectra via a function called `getSpectra()` (bet you didn't see that coming), and as normal, the default behaviour is to save the spectral data to disk. Behind the scenes, this uses the [`saveSpectrum()` common function](https://www.swift.ac.uk/API/ukssdc/commonFunc.md#commonFunc).\n",
"\n",
"For spectra we can choose to save a `gif` image of the spectrum and fit, and/or a `.tar.gz` archive containing the spectral files. There are various arguments you can provide to control how things are saved:\n",
"\n",
"* `extract` (default `False`) - whether to extract the spectral data from the tar file they are downloaded in.\n",
"* `removeTar` (default `False`) - whether to delete the tar file after extracting (this is ignored if `extract` is `False`)\n",
"* `saveImages` (default `True`) - whether to downoad the `gif` images of the automated spectral fits.\n",
"* `prefix` (default `None`) - an optional prefix to prepend to the names of the downloaded files. NOTE: this is not prepended to things extracted from the tar archive.\n",
"\n",
"The usual `subDirs` parameter is not listed here (you can supply it; you will be ignored). Because the contents of each tar file have the same name, if downloading spectra from multiple sources they will always be put in their own subdirectory, you cannot disable this.\n",
"\n",
"Anyway, \"A little less conversation a little more action\" seems appropriate here, so let's get to work:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "702c3a46",
"metadata": {},
"outputs": [],
"source": [
"uds.getSpectra(sourceName='LSXPS J221755.4-082100',\n",
" cat='LSXPS',\n",
" destDir='/tmp/APIDemo_SXPS_spec',\n",
" verbose=True,\n",
" extract=True,\n",
" removeTar=True\n",
" )"
]
},
{
"cell_type": "markdown",
"id": "0028f6c5",
"metadata": {},
"source": [
"I turned on verbose mode so that you can see what's happening. Because we only requested one object the data were saved directly into the `destDir`, and then extracted as requested. If we had requested multiple spectra they would have been saved in multiple directories, named for the sourceID or name, depending on how we called the function, so:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1dca95a5",
"metadata": {},
"outputs": [],
"source": [
"uds.getSpectra(sourceName=('LSXPS J221755.4-082100', 'LSXPS J033112.0+435414'),\n",
" cat='LSXPS',\n",
" destDir='/tmp/APIDemo_SXPS_spec2',\n",
" verbose=True,\n",
" extract=True,\n",
" removeTar=True\n",
" )"
]
},
{
"cell_type": "markdown",
"id": "dccaf186",
"metadata": {},
"source": [
"Hopefully you spotted the subdirectories above.\n",
"\n",
"You can explore those files yourselves, I'm not giving an X-ray spectroscopy primer here!\n",
"\n",
"One little note: if you want just the images and no tar file, you can set `saveData=False`, you will still get the images unless you also say `saveImages=False`.\n",
"\n",
"#### Storing spectra in variables\n",
"\n",
"As with light curves, we can grab the spectral information into a variable instead of (or as well as) saving to disk. In this case most of what we get is information about the spectral fit. As with pretty much everything in this API, when we do this we get a `dict`, and, in a shocking feat of imaginative naming, I call this a 'spectral `dict`', and [it is documented with the other common data structures in this module](https://www.swift.ac.uk/API/ukssdc/structures.md#the-spectrum-dict).\n",
"\n",
"\n",
"So, let's get an spectrum:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "bbd078c6",
"metadata": {},
"outputs": [],
"source": [
"specData = uds.getSpectra(sourceName='LSXPS J221755.4-082100',\n",
" cat='LSXPS',\n",
" saveData=False,\n",
" saveImages=False,\n",
" returnData=True,\n",
" )"
]
},
{
"cell_type": "markdown",
"id": "792fa8bb",
"metadata": {},
"source": [
"Note that I have set both `saveData` and `saveImages` to `False` so all I have got is the spectral data. This follows the spectral `dict` and I'm not going to detail this much because of the [dedicated documentation](https://www.swift.ac.uk/API/ukssdc/structures.md#the-spectrum-dict), but let's at least look at something. I want to know what the results of the APEC fit to this object was, and I know that, for all SXPS spectra, we only have one time interval (called interval0) and only one mode (PC), so I can jump straight to it:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "345ffc84",
"metadata": {},
"outputs": [],
"source": [
"specData['interval0']['PC']['APEC']"
]
},
{
"cell_type": "markdown",
"id": "3e8355bb",
"metadata": {},
"source": [
"The only other thing to say here is a reminder that if you supplied a list/tuple of source IDs or names in the `getSpectra()` call then `specData` will have an extra level - the top level will have one entry for each source you requested."
]
},
{
"cell_type": "markdown",
"id": "94a9c7af",
"metadata": {},
"source": [
"#### Saving after download\n",
"\n",
"As with light curves, you can separate the `get` and `save` phases, by calling `getSpectra(returnData=True)` and then calling `saveSpectra()`. The arguments for `saveSpectra()` are as discussed above when saving via the `getSpectra()` function, with one addition, `whichSources`. As for light curves, this lets you choose to save the spectra only of a subset of the sources you downloaded, thus:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "43c2b4ba",
"metadata": {},
"outputs": [],
"source": [
"specData = uds.getSpectra(sourceName=('LSXPS J051152.3-621925','LSXPS J221755.4-082100', 'LSXPS J062131.8-622213'),\n",
" cat='LSXPS',\n",
" saveData=False,\n",
" saveImages=False,\n",
" returnData=True,\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f58acb36",
"metadata": {},
"outputs": [],
"source": [
"uds.saveSpectra(specData,\n",
" whichSources=('LSXPS J051152.3-621925','LSXPS J221755.4-082100'),\n",
" destDir='/tmp/APIDemo_SXPS_spec3',\n",
" verbose=True,\n",
" extract=True,\n",
" removeTar=True\n",
" )"
]
},
{
"cell_type": "markdown",
"id": "9b2cc2ab",
"metadata": {},
"source": [
"----\n",
"\n",
"### Source Images\n",
"\n",
"You can also download the png-format thumbnail images of the sources. These can only be saved to disk, not returned, and therefore the function is `saveSourceImages()`. Its arguments should be familiar by now, so let's just get on with it:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "49867262",
"metadata": {},
"outputs": [],
"source": [
"uds.saveSourceImages(sourceName='LSXPS J051152.3-621925',\n",
" cat='LSXPS',\n",
" destDir='/tmp/APIDemo_SXPS_image',\n",
" verbose=True)"
]
},
{
"cell_type": "markdown",
"id": "573e1d51",
"metadata": {},
"source": [
"As usual, `cat` is not mandatory but I think it's helpful, and I turned on verbosity so you can see what was happening.\n",
"\n",
"One thing which I hope you noted above: as well as the usual bands (total, soft, etc) there is \"Expmap\" - which is the exposure map. So if you set the `bands` argument to something other than `all` then you can include 'expmap' (case insensitive), as well as the usual 'hard', 'total' etc.\n",
"\n",
"As usual, this function includes a `subDirs` parameter (default: `True`) which is used if you supply a list of sourceNames or IDs. If `True` the images go into a separate directory for each source; if `False`, the source name or ID is prepended to the file name, as below:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "68d29393",
"metadata": {},
"outputs": [],
"source": [
"uds.saveSourceImages(sourceName=('LSXPS J051152.3-621925','LSXPS J221755.4-082100'),\n",
" cat='LSXPS',\n",
" destDir='/tmp/APIDemo_SXPS_image2',\n",
" subDirs=False,\n",
" verbose=True)"
]
},
{
"cell_type": "markdown",
"id": "ea216a0c",
"metadata": {},
"source": [
"Note the \"Saving file\" lines - all the files went into the same directory, but with the source name in the file. (Random aside: I hate spaces in directories and filenames, so I always supply 'sourceID' when I actually use this API myself).\n",
"\n",
"----\n"
]
},
{
"cell_type": "markdown",
"id": "8649a8f7",
"metadata": {},
"source": [
"\n",
"### XRTProductRequest objects for sources\n",
"\n",
"You may want to build some custom products of a given SXPS source, using the `XRTProductRequest` object in the [`swifttools.ukssdc.xrt_prods` module](../xrt_prods.md). To aid in this, there is a function `makeProductRequest()`, which will generate `XRTProductRequest`s for you, supplying what information it can from the catalogue. \n",
"\n",
"**NOTE** The function aims to return a submittable `XRTProductRequest` object for you, based on the catalogue information. This means that if you don't specify the T0 or observation list via the arguments below, the global parameters `getT0` or `getTargs` will be set to `True`. Obviously, the `XRTProductRequest` can be manipulated before submission. If you don't know what I'm talking about, you should read the [xrt_prods documentation](../xrt_prods.md) before using this function.\n",
"\n",
"In order to get this information, the function needs some information from you, so you must provide either the result of an earlier `getSourceDetails()` function call, or the name of the catalogue so that `getSourceDetails()` can be called internally. Or, if you are requesting multiple sources, you can always supply both and then `getSourceDetails()` will be called for any sources not in your sourceDetails dataset.\n",
"\n",
"The information automatically applied to the product request is some subset of:\n",
"\n",
"* Name\n",
"* Position\n",
"* Whether to centroid\n",
"* T0\n",
"* TargetID list\n",
"* Observation list\n",
"\n",
"The first two of these are self-explanatory. \n",
"\n",
"\"Whether to centroid\" is set to `False`, since the position determined from the catalogue should be the best position (the analysis tools will still check for snapshot-to-snapshot variability).\n",
"\n",
"For T0, you have a few options, which are controlled by the `T0` parameter to `makeProductRequest()`. This can be:\n",
"\n",
"* `None` (default): Let the user objects software set T0. This leaves the `T0` global parameter in the `XRTProductRequest` object unset, but sets `getT0` to `True`.\n",
"* \"firstObs\": Set `T0` the start time of the first SXPS observation of the source location.\n",
"* \"firstBlindDet\": Set `T0` the start time of the first SXPS observation in which this source was blindly detected.\n",
"* \"firstDet\": Set `T0` the start time of the first SXPS observation in which this source was detected, blindly or retrospectively.\n",
"\n",
"The targetID list and observation list are correlated, they determine which data should be used to build the product. This can be controlled by the `useObs` argument to `makeProductRequest()`. The behaviour of this is slightly more complex than T0. It sets the `getTargs` and `targ` global parameters in the `XRTProductRequest`, and it will also set the `whichData` and `useObs` arguments of any products you request (discussed in a minute). **But please note** those keys are only set for products you request in the `makeProductRequest()` call. If you then edit the returned `XRTProductRequest` object to add more products, those parameters will not be set, you will have to do this yourself (I will note how, below).\n",
"\n",
"If not specified, the `useObs` argument is set to \"all\". This is the easiest case, and SXPS data are basically not used, because \"all\" means \"use all data, including any not in SXPS\". In this case the `getTargs` global parameter in the `XRTProductRequest` is set to `True`, and the `targ` value will be unset. All requested products will have `whichData` set to \"all\". \n",
"\n",
"For all of the other `useObs` values, `getTargs` will be set to `False`, `targ` will be set to specific values, and for the requested products, `whichData` will be \"user\" and the product's `useObs` parameter will be set to the string giving the requested observations.\n",
"\n",
"In full, the possible values for the `useObs` argument to `makeProductRequest()` are:\n",
"\n",
"* \"all\": as described above.\n",
"* \"allCat\": use all observations in the SXPS catalogue that cover the source position.\n",
"* \"allDet\": use all observations in the SXPS catalogue in which the source was detected, blindly or retrospectively.\n",
"* \"blind\": use all observations in the SXPS catalogue in which the source was blindly detected.\n",
"\n",
"I have mentioned products several times; by default the `XRTProductRequest` will be returned with the global parameters set, but nothing else. However, the `addProds` argument to `makeProductRequest()` allows you to specify some products as well. If you do this, those products are added to the request, and their `whichData` and `useObs` parameters set as discussed above.\n",
"\n",
"If you do not specify products, or don't specify all of them, then the \"useObs\" parameter is essentially pointless if it is not \"all\" (the default), since the restricted obs list is only applied to products. However, I have given an argument `returnObsList`. This is `False` by default, but if set to `True` then instead of returning an `XRTProductRequest` object, `makeProductRequest()` will return a `dict` with two keys: `request` will be the `XRTProductRequest` object, and `useObs` will be the string to pass to any products' `useObs` parameter if you add parameters.\n",
"\n",
"Of course, the requests are not submitted, so you can view and play with them. And, like with all of these functions, you can supply a single sourceID (or name) and get back a single `XRTProductRequest`, or you can supply a list of IDs (or names) and get back a `dict`, indexed by ID or name. \n",
"\n",
"Right, that is comprehensively far too much talk, so let's get to some demos:\n",
"\n",
"First, a simple one:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "23a0918e",
"metadata": {},
"outputs": [],
"source": [
"myReq = uds.makeProductRequest('MY_EMAIL_ADDRESS',\n",
" cat='LSXPS',\n",
" sourceID=17092,\n",
" useObs='all',\n",
" silent=False\n",
" )\n",
"myReq.getGlobalPars()"
]
},
{
"cell_type": "markdown",
"id": "16c910c1",
"metadata": {},
"source": [
"Don't forget that you have to be [registered](https://www.swift.ac.uk/user_objects/register.php) to use the on-demand product tools, and to put the correct email address above.\n",
"\n",
"In the above call, I did not provide any source information, instead I provided the catalogue, so the data lookup happened in the background. Let's see what it gave me:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "223ce459",
"metadata": {},
"outputs": [],
"source": [
"print(myReq)\n",
"myReq.getGlobalPars()"
]
},
{
"cell_type": "markdown",
"id": "7e1b6ac4",
"metadata": {},
"source": [
"OK, good, this made us a single request, and the default parameters have been set. Obviously, you can now edit this object, add products, submit it etc., but we're not doing that now (if you don't know how to, you need to [read the xrt_prods documentation](../xrt_prods.md).\n",
"\n",
"Let's explore a few more ways of using this function. First, we will specify that the product should only be built using observations in which our source was blindly detected, and we'll also say what products we want. Oh, and let's set `silent` to suppress all the output."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ed520445",
"metadata": {},
"outputs": [],
"source": [
"myReq = uds.makeProductRequest('MY_EMAIL_ADDRESS',\n",
" cat='LSXPS',\n",
" sourceID=17092,\n",
" T0='firstBlindDet',\n",
" useObs='blind',\n",
" addProds=['LightCurve', 'StandardPos'],\n",
" silent=True, \n",
" )\n",
"print(myReq)\n",
"print(f\"Globals: {myReq.getGlobalPars()}\\n\")\n",
"for p in myReq.productList:\n",
" print(f\"{p}:\\t{myReq.getProductPars(p)}\")"
]
},
{
"cell_type": "markdown",
"id": "eb398c32",
"metadata": {},
"source": [
"This all looks pretty good; notice how having supplied `useObs='blind'`, the API has worked out for us which observations correspond to a blind detection. There is a problem, however: this information is only applied to products requested via `uds.makeProductRequest()`. The `xrt_prods` module does not copy information about observations between products (because it's easy to do yourself). So, for example, if we now decide to add a spectrum:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1d3b8c88",
"metadata": {},
"outputs": [],
"source": [
"myReq.addSpectrum()\n",
"for p in myReq.productList:\n",
" print(f\"{p}:\\t{myReq.getProductPars(p)}\")"
]
},
{
"cell_type": "markdown",
"id": "f5f39fe8",
"metadata": {},
"source": [
"You can see that the spectrum parameters don't include the observation list. Of course, you can set it by yourself easily enough:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b9693020",
"metadata": {},
"outputs": [],
"source": [
"myReq.setSpectrumPars(whichData='user', useObs = myReq.getLightCurvePars()['useObs'])\n",
"for p in myReq.productList:\n",
" print(f\"{p}:\\t{myReq.getProductPars(p)}\")"
]
},
{
"cell_type": "markdown",
"id": "7285fc6f",
"metadata": {},
"source": [
"OK, two more things. First, I said above you don't have to supply the `cat` argument, you can instead get the source information yourself and then pass that directly to `makeProductRequest()`. We do that via the `sourceDetails` argument, as below:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "47612cf0",
"metadata": {},
"outputs": [],
"source": [
"info = uds.getSourceDetails(sourceID=17092,\n",
" cat='LSXPS',\n",
" silent=True,\n",
" verbose=False\n",
")\n",
"\n",
"myReq = uds.makeProductRequest('MY_EMAIL_ADDRESS',\n",
" sourceDetails=info,\n",
" sourceID=17092,\n",
" T0='firstBlindDet',\n",
" useObs='blind',\n",
" addProds=['LightCurve','Spectrum', 'StandardPos'],\n",
" silent=True, \n",
" )\n",
"print(myReq)\n",
"print(f\"Globals: {myReq.getGlobalPars()}\\n\")\n",
"for p in myReq.productList:\n",
" print(f\"{p}:\\t{myReq.getProductPars(p)}\")"
]
},
{
"cell_type": "markdown",
"id": "f17b60ce",
"metadata": {},
"source": [
"Not surprisingly, this worked just as above. One note: if you do this - if you use the `sourceDetails` argument - and you gave a single value for `sourceID` (rather than a list) then it will be this source for which the request is built *even if the sourceID parameter passed to makeProductRequest is different*. i.e. `sourceDetails` trumps `sourceID`. If you're worried about making an error because of this, either don't use `sourceDetails` or always use a tuple. i.e. if you replace the `sourceID` argument (in both function calls) with `sourceID=(17092,)` then everything will be all right.\n",
"\n",
"The last thing to show, related to what I've just said, is that we can of course pass in multiple sources as I said before. I'm actually going to combine one other thing in this demo: if you pass in sourceInfo, but it does not contain the info for the source you submitted, then a `getSourceDetails()` call will again be done in the background. I will turn verbosity on to make this clear:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "060e0fc5",
"metadata": {},
"outputs": [],
"source": [
"data = uds.getSourceDetails(sourceID=(17092,),\n",
" cat='LSXPS',\n",
" silent=True,\n",
" verbose=False\n",
")\n",
"\n",
"rlist = uds.makeProductRequest('MY_EMAIL_ADDRESS',\n",
" sourceDetails=info,\n",
" sourceID=(17092,128791),\n",
" cat='LSXPS', # This is actually the default, but explicit is good\n",
" T0='firstBlindDet',\n",
" useObs='blind',\n",
" addProds=['LightCurve'],\n",
" silent=False, \n",
" verbose=False\n",
" )"
]
},
{
"cell_type": "markdown",
"id": "72860151",
"metadata": {},
"source": [
"As you see, we did a sourceInfo look up for 17006. You'll also notice that when I turned `silent` off, it was disabled for the `XRTProductRequest` objects as well, hence the extra blurb. We'll sort that in a second, first, let's look at what we got:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "933b9186",
"metadata": {},
"outputs": [],
"source": [
"rlist"
]
},
{
"cell_type": "markdown",
"id": "8c2ca95e",
"metadata": {},
"source": [
"Hopefully you were expecting that: we got a `dict`, and the keys were the supplied sourceIDs. If we have given `sourceName` not ID then the keys would be the names -- and the sourceInfo object would also need to be indexed by name or look ups would be done in the background.\n",
"\n",
"I mentioned that we may want to make the actual `XRTProductRequest`s be silent. Let's do that, and also just check them out:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c193b21a",
"metadata": {},
"outputs": [],
"source": [
"for sourceID in rlist.keys():\n",
" print(f\"\\n{sourceID}\\n======\")\n",
" rlist[sourceID].silent=True\n",
" print(f\"{rlist[sourceID]}\")\n",
" print(f\"Globals: {rlist[sourceID].getGlobalPars()}\\n\")\n",
" for p in rlist[sourceID].productList:\n",
" print(f\"{p}:\\t{rlist[sourceID].getProductPars(p)}\")\n"
]
},
{
"cell_type": "markdown",
"id": "3d67b002",
"metadata": {},
"source": [
"And you can see that we have two separate requests with their own parameters.\n",
"\n",
"**IMPORTANT REMINDER**\n",
"\n",
"This system allows you to easily create large numbers of `XRTProductRequest` objects, but please do not submit large numbers at once. [The xrt_prods documentation](https://www.swift.ac.uk/user_objects/API) gives advice on how to throttle your submissions. Users who overload the system will be given a stern talking to...\n",
"\n",
"----\n"
]
},
{
"cell_type": "markdown",
"id": "2868d3ce",
"metadata": {},
"source": [
"\n",
"## Datasets\n",
"\n",
"Datasets can be directly accessed through this module, either by their ObsID, or their DatasetID (LSXPS only). ObsIDs are the Swift standard 11-digit identifier of an observation, except for stacked images (which are built by combining observations); these have been given a 'fake' ObsID which begins with a 1 (normal Swift observations begin with 0), so if an obsID is >1e10, then it is a stacked image.\n",
"\n",
"At present there are two things you can get for a dataset: the [full set of information about it](#dsinfo), and [the images](#dsImages)\n",
"\n",
"Because LSXPS is dynamic, an observation may be analysed several times as new data are received on the quicklook site, thus each version of an observation is assigned a unique DatasetID for LSXPS. For normal observations, only a single DatasetID will enter the catalogue. Stacked images in LSXPS are more complex, because they can change as new data are added, as discussed in Sections 2.2 and 3.3 of [the LSXPS paper](https://www.swift.ac.uk/LSXPS/paper.php). If you request products for a stacked image which has been superseded then you will get back details of the superseding stacks, rather than the products you asked for. We will return to this [below](#superseded-stacked-images).\n"
]
},
{
"cell_type": "markdown",
"id": "6a6175ac",
"metadata": {},
"source": [
"\n",
"### Dataset details\n",
"\n",
"**Note: This functionality only exists for LSXPS**\n",
"\n",
"To get information about a dataset, we simply call the function `getDatasetDetails()`. As for sources, this does not have an option to save data to disk, simply always returning a `dict`. It has only the standard arguments (`DatasetID` or `ObsID`, `cat`, `silent` and `verbose`). So, let's go exploring:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0f0f1085",
"metadata": {},
"outputs": [],
"source": [
"dsInfo = uds.getDatasetDetails(ObsID='00282445000', cat='LSXPS')"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "bd4561dc",
"metadata": {},
"outputs": [],
"source": [
"dsInfo"
]
},
{
"cell_type": "markdown",
"id": "fc5b579b",
"metadata": {},
"source": [
"As you can see from the above, this is largely a `dict` containing the information that is provided in the dataset web pages (in fact, those web pages are built from this `dict`). Let's explore a few of the entries here. \n",
"\n",
"If you scroll down in the above you'll see one thing that look like a tables -- 'Sources'. This is indeed a table (the list of sources in the dataset you can see on the webpage) and this has been converted into a `DataFrame`:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7589c993",
"metadata": {},
"outputs": [],
"source": [
"dsInfo['Sources']"
]
},
{
"cell_type": "markdown",
"id": "09d63f42",
"metadata": {},
"source": [
"And you may note that the columns 'Total_DetectionDetails', 'Soft_DetectionDetails' etc are `dict`s giving summary details of the detection of this source in this dataset in the named band. I have to admit, having a `dict` inside a `DataFrame` seems a little inelegant to me, but I didn't like the idea of expanding these properties out and making the `DataFrame` really wide. If any readers have suggestions as to how to handle this better: I'm listening! As it is, it's not so hard to access the details, e.g. to see the total-band details for the first source:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3d290bc4",
"metadata": {},
"outputs": [],
"source": [
"dsInfo['Sources']['Total_DetectionDetails'][0]"
]
},
{
"cell_type": "markdown",
"id": "b7dee587",
"metadata": {},
"source": [
"And of course we can access a specific property:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "af310b6b",
"metadata": {},
"outputs": [],
"source": [
"dsInfo['Sources']['Total_DetectionDetails'][0]['SNR']"
]
},
{
"cell_type": "markdown",
"id": "a7f0c19f",
"metadata": {},
"source": [
"As with all of the data access functions (see the [General notes](#intro)) we can get details of multiple objects by supplying a list/tuple to `getDatasetDetails()`. In this case, we get a `dict` of `dict`s back, the outer layer being indexed by the obsID or datasetID, whichever we supplied, as we see here:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7cec145f",
"metadata": {},
"outputs": [],
"source": [
"dsInfo = uds.getDatasetDetails(ObsID=('00282445000','00015231001'),\n",
" cat='LSXPS')\n",
"dsInfo.keys()"
]
},
{
"cell_type": "markdown",
"id": "1ea149d3",
"metadata": {},
"source": [
"\n",
"### Dataset images\n",
"\n",
"You can also download the FITS images and source regions (ds9 format) for your datasets, using the `saveDatasetImages()` function. As with source images, this is called `save` not `get` because you can only save data to disk.\n",
"\n",
"Let's start with a quick demo:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "dd96914b",
"metadata": {},
"outputs": [],
"source": [
"uds.saveDatasetImages(ObsID='00282445000',\n",
" cat='LSXPS',\n",
" destDir='/tmp/APIDemo_SXPS_Im', \n",
" verbose=True,\n",
" getRegions=True)"
]
},
{
"cell_type": "markdown",
"id": "0c1de8f8",
"metadata": {},
"source": [
"As ever, I've turned on verbosity so you can see what is going on.\n",
"\n",
"The default settings (used above) get all of the images for the specified dataset(s), and I added `getRegions=True` (it defaults to `False`) to get the ds9 region files as well.\n",
"\n",
"Apart from the normal arguments, (`subDirs`, `destDir` etc), note that like for source images, `bands` can include \"Expmap\" - which is the exposure map.\n",
"\n",
"The other useful parameter is 'types', which specifies which images you want to get. The default is 'all', but can instead be a (case insensitive) list/tuple of:\n",
"\n",
"* 'Image' - the actual image\n",
"* 'BackgroundMap' - the background map ('bgmap' is also accepted).\n",
"* 'BackgroundMapWithSources' - the background map ('bgmap_wsrc' is also accepted).\n",
"\n",
"Let's do one more quick demo, exploting these features and reminding you that you can get more than one dataset in a single request."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "93e07869",
"metadata": {},
"outputs": [],
"source": [
"uds.saveDatasetImages(ObsID=('00282445000','00221755001'),\n",
" cat='LSXPS',\n",
" destDir='/tmp/APIDemo_SXPS_Im2', \n",
" types=('image','backgroundmap'),\n",
" bands=('total', 'soft'),\n",
" verbose=True,\n",
" getRegions=True)"
]
},
{
"cell_type": "markdown",
"id": "78e177dc",
"metadata": {},
"source": [
"\n",
"### Superseded stacked images\n",
"\n",
"As mentioned above, in the dynamic LSXPS catalogue, stacked images can be superseded (see Sections 2.2 and 3.3 of [the LSXPS paper](https://www.swift.ac.uk/LSXPS/paper.php)). So, what happens if you try to get the products for a superseded dataset?\n",
"\n",
"First, we have to remember that there are two types of superseded stacks:\n",
"\n",
"1. Those which, being superseded, have been removed from the catalogue.\n",
"1. Those which, although superseded, are the only dataset in which a source was detected, so are retained but marked as 'obsolete'.\n",
"\n",
"What happens if you try to access one of these images depends on which function you called. \n",
"\n",
"`saveDatasetImages()` has no return type, so if you try to get a non-existent dataset (including a superseded stack) you will just get an error. If you try to get an obsolete stacked image then you will get the data, but with a warning message.\n",
"\n",
"For `getDatasetDetails()` things are slightly different. Again, if you request a non-existent data you will just get an error, but if you ask for a superseded or obsolete one, the returned `dict` will contain the key \"SupersedingStacks\", which tells us that the stack is superseded, and by what.\n",
"\n",
"I will demonstrate these points but **be aware** due to the above-mentioned dynamic nature of the catalogue, the stacks I've used in the demos below may themselves change. If you're reading the web page, then the output for these examples is fixed so all is well. If you are using the Jupyter notebook, you may not get the same results that I did!\n",
"\n",
"Let's start by getting images. I'm going to request an image which I know is a superseded stack."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5ac5e126",
"metadata": {},
"outputs": [],
"source": [
"uds.saveDatasetImages(ObsID='10000000668',\n",
" cat='LSXPS',\n",
" destDir='/tmp/APIDemo_SXPS_Im3', \n",
" verbose=True,\n",
" getRegions=True)"
]
},
{
"cell_type": "markdown",
"id": "87528cd6",
"metadata": {},
"source": [
"As predicted, I got an error, and the error message tells me that the dataset cannot be found. What if I'd tried an obsolete stack?"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b758740d",
"metadata": {},
"outputs": [],
"source": [
"uds.saveDatasetImages(ObsID='10000000189',\n",
" cat='LSXPS',\n",
" destDir='/tmp/APIDemo_SXPS_Im3', \n",
" verbose=True,\n",
" getRegions=True)"
]
},
{
"cell_type": "markdown",
"id": "7c235b76",
"metadata": {},
"source": [
"Hopefully you can see the warning line above -- with `verbose` mode enabled, it can get lost, so be careful!\n",
"\n",
"Now lets explore these same two datasets with `getDatasetDetails()`:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "73f4dfda",
"metadata": {},
"outputs": [],
"source": [
"dsInfo = uds.getDatasetDetails(ObsID='10000000668', cat='LSXPS')\n",
"dsInfo.keys()"
]
},
{
"cell_type": "markdown",
"id": "4b04b006",
"metadata": {},
"source": [
"This was a case of a stack that has been superseded and removed, and we know this because the returned `dict` has the key \"SupersedingStacks\". Let's see what's in there:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f3530f3d",
"metadata": {},
"outputs": [],
"source": [
"dsInfo['SupersedingStacks']"
]
},
{
"cell_type": "markdown",
"id": "0fd56e3c",
"metadata": {},
"source": [
"We have the key \"OtherStacks\", which contains a list giving some details of which stacks have superseded the one we requested; in this case there was only one of them. The key \"OtherStacks\" tells us that the superseding images are actually different stacks, that is, stack 668 is dead, and has been replaced with 25185. Sometimes when a stack is superseded it can be split up into multiple stacks, hence \"OtherStacks\" is a list.\n",
"\n",
"I don't want to get bogged down here in the details of stacked image management in LSXPS, but you should be aware that if you request a stack by its ObsID, as here, the only way you can get this \"SupersedingStacks\" key is because the stack itself has been replaced, because querying by ObsID will always get you the latest version of a stack. But if you query by DatasetID, which is tied to a specific version of a stack, you can find that your requested image has been superseded not by a new stack, just by a new version of it. Like in this case:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ace0b59f",
"metadata": {},
"outputs": [],
"source": [
"dsInfo = uds.getDatasetDetails(DatasetID=229018, cat='LSXPS')\n",
"dsInfo['SupersedingStacks']"
]
},
{
"cell_type": "markdown",
"id": "63000f4f",
"metadata": {},
"source": [
"Here, instead of the \"OtherStacks\" key we have the \"LatestVersion\" entry. This tells us that the stack that dataset 229018 was in still exists, but there is a newer version.\n",
"\n",
"If we asked for the dataset details for an obsolete stack we would get the full set of data - that stack is still in LSXPS - but the \"IsObsoleteStack\" value will be 1, rather than 0, which tells us that the stack is obsolete."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "80e5e9a6",
"metadata": {},
"outputs": [],
"source": [
"dsInfo = uds.getDatasetDetails(DatasetID=228895, cat='LSXPS')\n",
"dsInfo[\"IsObsoleteStack\"]"
]
},
{
"cell_type": "markdown",
"id": "51e1a144",
"metadata": {},
"source": [
"------\n",
"\n",
"## Transients\n",
"\n",
"The final SXPS product currently supported for direct access are transients, which are an **LSXPS product only**. To obtain transient data, we use the same functions as for sources, but pass the argument `transient=True` (except in one case). Since these functions were all discussed at length in the [sources section](#sources) I am not going to go into huge detail here, beyond identifying some transient-specific arguments, **so I advise you to read the [sources section](#sources)** if you want to understand what follows (yes, I know, it's long. Sorry).\n",
"\n",
"The first of these is `transAsSource`, which defaults to `False` and needs a few words of explanation. XRT data are searched for transients as soon as they are received, and transients made public as soon as possible, so the transients can (and usually will) be made public before the dataset they were discovered in is marked as 'complete' and added to LSXPS. Thus, transients are given a Transient ID and a \"Swift J\" name, rather than an LSXPS ID and \"LSXPS J\" name. However, the transient source will of course appear in LSXPS when its dataset is complete, so at this point the transient record is updated to point to the relevant LSXPS entry -- and the transient light curve and spectra stop being updated (since these products are now curated for the source entry instead) unless the XRT team explicitly request otherwise.\n",
"\n",
"The `transAsSource` argument lets you obtain information from the LSXPS source, accessed via its transient ID. That is, if I know I am interested in transient \"Swift J123456.7+891011\", but I actually want the spectra created once it entered LSXPS, I can say `getSpectra(sourceName='Swift J123456.7+891011', transient=True, transAsSource=True)`. **But beware** if this transient is not yet in LSXPS, you will get an error (unless you also said `skipErrors=True`, in which case you'll just get an empty `dict`). **Also beware** if you supply a list of sourceID/sourceName values with `transAsSource=True` then the products saved/returned will be indexed by their *LSXPS* ID or name, not the transient ID you supplied. Generally, I will not demonstrate this argument, since it literally reproduces the behaviour already shown in the [sources section](#sources).\n",
"\n",
"Right, with that introduction out of the way, let's have a brief ramble through the transient products.\n"
]
},
{
"cell_type": "markdown",
"id": "358dc2df",
"metadata": {},
"source": [
"\n",
"### Transient details\n",
"\n",
"The one exception to the 'just use the sources function' is the function to get the `dict` of transient information. For that we have a dedicated transients function, but I think you can probably guess both its name, and its action."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "caaea1d0",
"metadata": {},
"outputs": [],
"source": [
"data = uds.getTransientDetails(sourceID=576)"
]
},
{
"cell_type": "markdown",
"id": "bad79973",
"metadata": {},
"source": [
"You'll note that I didn't bother giving `cat='LSXPS'` in this case, since transients are only in LSXPS. Let's have a look at what we got:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "dc01c45a",
"metadata": {},
"outputs": [],
"source": [
"data"
]
},
{
"cell_type": "markdown",
"id": "f5cd943e",
"metadata": {},
"source": [
"There are no tables/`DataFrame`s in there to worry about, just information. You will also note that this has a valid 'LSXPS_ID' entry, i.e. it is now in LSXPS, so we could have used that `transAsSource` argument. Let's do that:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "97ae0834",
"metadata": {},
"outputs": [],
"source": [
"data = uds.getTransientDetails(sourceID=576, transAsSource=True)\n",
"data"
]
},
{
"cell_type": "markdown",
"id": "a33c1cb7",
"metadata": {},
"source": [
"You can see that what we got here was the same as if we'd done `getSourceDetails(sourceID=349111)`."
]
},
{
"cell_type": "markdown",
"id": "6bb605cd",
"metadata": {},
"source": [
"----\n",
"\n",
"### Light curves\n",
"\n",
"Transient light curves are obtained and saved as for [source light curves](#lightcurves); apart from the addition of the `transient=True` argument (essential to explain we are getting a transient), the only changes relate to the differences between transient light curves and source light curves. Transient light curves are built for rapid analysis, using the [on-demand XRT analysis tools](https://www.swift.ac.uk/user_objects). This results in a few differences to how we call `getLightCurves()`:\n",
"\n",
"* `bands` is ignored (only total-band curves are made for transients)\n",
"* `timeFormat` is ignored; only MJD is available for transients;\n",
"* `binning` supports an extra option, 'counts', which gives light curves binned to a minimum number of counts per bin.\n",
"\n",
"For the first two, you will not get an error if you supply an unsupported band or timeFormat, you'll just get the total-band, MJD data. If `silent=False` you will at least be warned that your input is being ignored. All the arguments about whether to get limits or bins and grouping them etc are silently ignored, as they are meaningless in the context of transients.\n",
"\n",
"As usual, the files are either written to disk (default), returned as a variable, or both. We will just explore the return option, so that I can show you what the returned object looks like:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "39d285e6",
"metadata": {},
"outputs": [],
"source": [
"lcs = uds.getLightCurves(sourceName=('Swift J073006.8-193709', 'Swift J175737.4-070600'),\n",
" transient=True,\n",
" cat='LSXPS',\n",
" binning='counts',\n",
" saveData=False,\n",
" returnData=True,\n",
" )\n",
"lcs.keys()"
]
},
{
"cell_type": "markdown",
"id": "7a51829f",
"metadata": {},
"source": [
"I got two transients, just to remind you that we can, so as you can see, the top-level of the returned `dict` has an entry per source. Those entries are each a [standard light curve `dict`](https://www.swift.ac.uk/API/ukssdc/structures.md#the-light-curve-dict), so for example we can view the list of light curves:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "dce2f835",
"metadata": {},
"outputs": [],
"source": [
"lcs['Swift J073006.8-193709']['Datasets']"
]
},
{
"cell_type": "markdown",
"id": "3463d0e1",
"metadata": {},
"source": [
"This automatically gave us all of the possible datasets from the transient light curves. The dataset name components '\\_nosys' and '\\_incbad' indicate that those datasets have had systematic errors removed, or times with poor/no centroid information included. For more details you will need to read up on the product generator.\n",
"\n",
"The other thing to note -- sorry -- is that \"hard\", \"soft\" and \"HR\" here are again as used in the on-demand tool, not the catalogue. So \"soft\" is 0.3-1.5 keV, \"hard\" is 1.5-10 keV and the HR is just hard/soft.\n",
"\n",
"If you find this annoying, there is a reason, but it's too boring to write here. Similarly let's check an actual light curve:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3148c1cf",
"metadata": {},
"outputs": [],
"source": [
"lcs['Swift J073006.8-193709']['PC_incbad']"
]
},
{
"cell_type": "markdown",
"id": "99452e06",
"metadata": {},
"source": [
"As you can see, this light curve has a few extra columns compared to the source light curves, but the fundamentals are the same."
]
},
{
"cell_type": "markdown",
"id": "7a66c59d",
"metadata": {},
"source": [
"\n",
"### Spectra\n",
"\n",
"Again the spectra are very similar to [source spectra](#spectra), with just two differences:\n",
"\n",
"* There is only a power-law model fitted\n",
"* There is a new argument to `getSpectra()`: `specType`.\n",
"\n",
"The latter exists because two spectra are created for transients, one using only the dataset in which the transient was discovered, and one using also any subsequent observations covering the transient. `specType` is a string, and should be one of 'discovery' 'full' or 'both'. In the latter case, the returned `dict` will contain a new top-level per transient, with keys 'Discovery' and 'Full' and the contents being the spectrum `dict`.\n",
"\n",
"Let's explore this:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "515dc8b4",
"metadata": {},
"outputs": [],
"source": [
"specSet = uds.getSpectra(sourceID=(30, 576),\n",
" destDir='/tmp/APIDemo_transSpec2',\n",
" transient=True,\n",
" silent=False,\n",
" specType='both',\n",
" verbose=True,\n",
" returnData=True\n",
" )"
]
},
{
"cell_type": "markdown",
"id": "23a4019f",
"metadata": {},
"source": [
"Note that I set `returnData=True` and did not set `saveData` to `False`, so we saved and returned. You can see from the above the structure of saved files; remember I got multiple transients which is where there is a separate directory for each. Let's just confirm what the returned data look like."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c96d3b16",
"metadata": {},
"outputs": [],
"source": [
"specSet.keys()"
]
},
{
"cell_type": "markdown",
"id": "4e9d81d7",
"metadata": {},
"source": [
"OK, that's expected, an entry per transient."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cd560bd5",
"metadata": {},
"outputs": [],
"source": [
"specSet[30].keys()"
]
},
{
"cell_type": "markdown",
"id": "cf780fc6",
"metadata": {},
"source": [
"As I warned you, `specSet[30]` is not [a spectrum `dict`](https://www.swift.ac.uk/API/ukssdc/structures.md#the-spectrum-dict) as it would have been for sources, but simply a `dict` indexing the two types of spectra we got. Those two entries, however, *are* spectrum `dict`s."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "aea28abf",
"metadata": {},
"outputs": [],
"source": [
"specSet[30]['Discovery'].keys()"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "64f54419",
"metadata": {},
"outputs": [],
"source": [
"specSet[30]['Full'].keys()"
]
},
{
"cell_type": "markdown",
"id": "2455697d",
"metadata": {},
"source": [
"There was no full spectrum for source 30 (probably meaning that there were no observations apart from the discovery one), so we get the null spectral `dict` instead.\n",
"\n",
"The only other thing to tell you about transient spectra is that, should you decide to separate the download and save steps you have to provide `transient=True` to `saveSpectra` as well.\n",
"\n",
"----"
]
},
{
"cell_type": "markdown",
"id": "6e7750e7",
"metadata": {},
"source": [
"\n",
"### Images\n",
"\n",
"Let's see if you can guess how we get the images of a transient. I will request only the soft band and exposure map, just for fun (look, it's 4pm on a Friday, anything feels like fun now)."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e16b1c90",
"metadata": {},
"outputs": [],
"source": [
"uds.saveSourceImages(sourceName='Swift J073006.8-193709',\n",
" transient=True,\n",
" bands=('soft', 'expmap'),\n",
" destDir='/tmp/APIDemo_SXPS_image2',\n",
" verbose=True)"
]
},
{
"cell_type": "markdown",
"id": "34747af1",
"metadata": {},
"source": [
"----\n",
"\n",
"\n",
"### XRTProductRequest\n",
"\n",
"The final thing is to be able to make an XRTProductRequest. I think you can guess how we do this as well:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ad9209e4",
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"myReq = uds.makeProductRequest('MY_EMAIL_ADDRESS',\n",
" transient=True,\n",
" sourceID=576,\n",
" T0='Discovery',\n",
" useObs='new',\n",
" addProds=['LightCurve',]\n",
" )\n",
"print(myReq)\n",
"print(f\"Globals: {myReq.getGlobalPars()}\\n\")\n",
"for p in myReq.productList:\n",
" print(f\"{p}:\\t{myReq.getProductPars(p)}\")"
]
},
{
"cell_type": "markdown",
"id": "3f0589a9",
"metadata": {},
"source": [
"Where, above, we created an XRTProductRequest with a light curve.\n",
"\n",
"The two (optional) parameters to set the T0 and observation list differ slightly for transients compared to for sources. \n",
"\n",
"`T0` can only be 'Discovery' or `None` (the default): 'Discovery' this will set it to the start time of the observation in which the transient was discovered, `None` will not set the T0 in the `XRTProductRequest`.\n",
"\n",
"`useObs` can be (case insensitive):\n",
"\n",
"* 'discovery' = Only the discovery observation.\n",
"* 'all' = all observations covering the source.\n",
"* 'new' = all observations covering the source, from the discovery time onwards.\n",
"\n",
"**Important note** in this case 'all' and 'new' will get their observations from the full set of Swift observations, *not* only those in the LSXPS catalogue (as when calling this function for sources). This is because transients may not yet be in the catalogue, as already discussed above.\n",
"\n",
"----"
]
},
{
"cell_type": "markdown",
"id": "15cdb4bd",
"metadata": {},
"source": [
"Well, you made it! Well done. Hopefully you didn't die of boredom during the above. If you spotted any mistakes, do let me know and I'll try not to curse you under my breath while I fix them."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3.9.13 ('python')",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.13"
},
"vscode": {
"interpreter": {
"hash": "f513bf85d22a3153060d5d107ef496d0a3f1f44029ac3c98802cda3a73891884"
}
}
},
"nbformat": 4,
"nbformat_minor": 5
}