US Site | Italian Site | UK site

Search:

 |  site map

• Home
• About
  The Swift Mission UK involvement Instruments

  ---

  GRBs before Swift GRBs in the Swift era Non-GRB science with Swift GRB research at Leicester

  ---

  Spacecraft status Swift in the press Swift Videos
• Support
  Helpdesk TOO requests Guest Investigator program Target visibility checker Search transient database
• Data Access
  Overview Quick Look GCN Notices Download obs data Search obs catalogue The 2SXPS catalogue The LSXPS catalogue
• Data Analysis
  Data analysis overview Swift software

  ---

  Build XRT products

  ---

  BAT data analysis guide XRT data analysis guide XRT digest UVOT data analysis guide
• Data Products
  GRBs Full dataset Swift XRT products for GRBs XRT positions XRT light curves XRT spectra XRT GRB catalogue XRT GRB images Burst analyser Prompt data GRB field info XRT prompt analysis External triggers Einstein Probe follow up SVOM follow up Neutrino follow up Gravitational Wave follow up
• Publications
  Publication index Published papers Submitted papers XRT instrument papers ATels GCNs Conferences
• Links
  Databases NED Simbad GCN circulars archive GRB data table Software & Tools Swift Software (HEASoft) Xanadu WebPIMMS Institutional Swift Sites GSFC PSU OAB SSDC MSSL University of Leicester

The swifttools.ukssdc.query module

Jupyter notebook version of this page

Latest version v1.0, released in swifttools v3.0

The query module provides tools to query various catalogues held at the UKSSDC (and a couple of others) and provides wrappers to get data for the objects we find. There are two types of query supported, a cone search and a more complex search applying filters to different fields. These are analogous to the 'simple' and 'advanced' searches available on the website.

Unlike the the data module (which provided functions to get specific products), the query module provided classes: we create query objects and then manipulate and use these to perform our query and play with the results. There are three query classes provided:

• ObsQuery - This lets us query the databases of Swift observations.
• GRBQuery - This lets us query several GRB catalogues.
• SXPSQuery - This lets us query the SXPS catalogues.

Whichever class you want, I strongly advise you to read this page first. Although it deals with the ObsQuery class, that class only contains functionality common to all of the classes, and so all the common concepts are introduced here, and not necessarily repeated on the other pages.

The query module also provides access to data or data products, by wrapping functions from the data module.I will not go into much detail about those functions and what they produce, since this has been covered in the relevant pages for the data module already. One crucial point, however, does need discussing: how the query module makes any requested data available to you (if you haven't read about the data module, ignore this point because it will just confuse you - you'll see how we get data shortly). In the data module the default behaviour when getting a product was to download files to disk, but you could instead request the data be returned from the function so you could capture it in a variable. For the query module, the requested data are always stored in variables inside your query object. You can still also request that they be saved to disk, or returned from the function that got them (using the same returnData and saveData arguments as in the data module) but these are not done by default.

In this notebook we will demonstrate the query interface for observation data, which is quite straightforward and also gets us familiar with the module syntax.

We will import the query module as uq:

import swifttools.ukssdc.query as uq

Contents

• The ObsQuery class
  • Selecting a table
• Simple (cone) searches
• Selecting columns to retrieve
• Advanced queries
• Extra query settings
• Retrieving products
  • For only some rows

The ObsQuery Class

The ObsQuery class allows us to query the Swift observation database, it is the API equivalent of this UKSSDC webpage. The only functionality it provides is that which is common to the entire swifttools.ukssdc.query module and all subclasses, so it's the perfect one to use as an introduction.

As discussed above, this module is built around classes, so the first thing we have to do is create an ObsQuery object:

q = uq.ObsQuery(silent=False)

Resetting query details

I set silent=False because in an interactive case like this, it can be helpful to get some textual feedback. If you want even more feedback you can set verbose=True. These can be set in the constructor, or via simple calls:

q.verbose = True
q.verbose = False # Turn it off again!

Selecting a table

There are several database tables relating to Swift observations. Just like the website, the API gives you the most useful (I think) one by default, but of course, you can change this. But how? Well, first, we want to know what tables we have to choose from, and this is stored in the tables variable of our query object:

q.tables

('swiftmastr', 'swiftxrlog', 'swiftbalog', 'swiftuvlog', 'swifttdrss')

We can also check which one is currently selected, or change it, via the table variable:

q.table

'swiftmastr'

q.table = 'swiftxrlog'

Resetting query details

You'll notice the cell that changed the table warned us that it was resetting the query details. This only appeared because we have silent=False and don't worry about it too much now. Basically it is warning us that any filters set or results retrieved (all covered below) have been wiped because we changed table. We could also have set the table we wanted in the constructor:

q = uq.ObsQuery(table='swiftbalog',
                silent=False
               )
q.table

Resetting query details

'swiftbalog'

I won't keep saying this but I'll remind you here: ObsQuery only contains behaviour common to the entire query module, so the syntax for checking and changing tables is the same for the GRBQuery and SXPSQuery modules (and for any others I may add in the future).

Simple (cone) searches

A simple search is just a cone search, and rather than pontificating, let's just demonstrate. I will stick with the default table, "swiftmastr" for all the demonstrations below.

q = uq.ObsQuery(silent=False)
q.addConeSearch(name='GK Per', radius=300, units='arcsec')
q.submit()

Resetting query details
Need to get the metadata to check the query is valid.
Calling DB look-up for rows 0 -- 1000
Resolved 'GK Per' as (52.80004119305, 43.90429720867) via SIMBAD
Received 145 rows.

That was pretty straightforward wasn't it? We introduced two functions here, addConeSearch() and submit(). The latter is very easy: it takes no arguments and just submits our query object for execution.

addConeSearch() should also be pretty clear, but I'll elaborate on its arguments in a moment.

Because we disabled silent mode, we got a bit of information about what was going on, and we can see, firstly, that "GK Per" was resolved into coordinates (thank you SIMBAD), and secondly, that we got 145 results. But where are those results? They are stored inside your query object (q) and the variable holding them is cunningly named results. This is a pandas DataFrame and we can have a look at it:

q.results

You can explore this at your leisure, but let me highlight one point regarding coordinates. In the databases we query, coordinates are stored in decimal degrees (J2000), but this may not be how you want them. So, when you perform a query that gets coordinates, the query module will do a bit of extra work. It identifies all of the coordinate columns and creates sexagesimal versions of the coordinates to (in the format of strings). To identify these, "_s" is appended to the column name (so in the above, "ra" and "decl" were part of the database, and "ra_s", "decl_s" have been added. Also, if you have the astropy module installed then the coordinates will be converted into astropy.coordinate.Angle objects and identied by "_apy" ("ra_apy" and "decl_apy" in the above.

When we executed the query above, we were told how the name supplied had been resolved; but only because we set silent=False, and not in a way that we could have readily captured in our script. Despair not, the details of the name resolution are also in class variables:

q.resolvedInfo

"Resolved 'GK Per' as (52.80004119305, 43.90429720867) via SIMBAD"

Or indeed:

print(q.resolvedRA)
print(q.resolvedDec)

52.80004119305
43.90429720867

We're going to go back and look at addConeSearch() a bit more and explore some of its syntax, but if we just ran q.addConeSearch() again now, we'd get an error telling us our query was locked. This is because when a query is submitted it is locked to prevent ending up in a confused state (a trivial example; imagine you ran the query above and then ran q.addConeSearch(name='FO Aqr') but due to an error, didn't submit it; it would be easy to mistakenly think that q.results was related to the FO Aqr search. It isn't).

So, if we want to do another query we either need to make a new ObsQuery object, or reset the one we have. Let's do the latter:

q.reset()

Resetting query details

Now we can make any changes we like. Before I move on I'll note that q.reset() has some options, you don't have to reset literally everything, but you can read about those via the help command if you want.

So, now we can do another cone search if we want. Before we do, let's look at the arguments that this function takes. You could do this via help (q.addConeSearch) but I'll be nice and elaborate here.

A cone search needs to know two things:

1. The centre of the cone
2. The radius of the cone.

The latter is easy, it is managed by two arguments:

• radius - a number giving the radius.
• units - the units of radius, must be 'arcsec', 'arcmin' or 'deg' (default: 'arcsec').

The centre of the cone can be specified in a few ways, via these arguments:

• name - a string giving an object name which we will attempt to resolve.
• position - a free-form string giving the position, which we will attempt to parse.
• ra and dec - Two arguments that can either be floats or astropy.coordinates.Angle objects.

You should only provide one of these arguments (OK, two if ra and dec) or you will get an error. name was used above. position is a free-form string and we have tried to accept almost any sane way in which you may enter coordinates (provided they are in J2000). So, all of the examples below will work:

q.addConeSearch(ra=123.456, dec=-43.221, radius=1, units='deg')
q.addConeSearch(position='12 13 15, -15 16 17', radius=12, units='arcmin')

from astropy.coordinates import Angle
ra = Angle('12h 13m 14s')
dec = Angle('-13d 14m 15s')
q.addConeSearch(ra=ra, dec=dec, radius=300, units='arcsec')

If you provided a name or position, it will be resolved when the query is submitted, so you can check the details of the resolution using the q.resolvedRA etc. variables already introduced. If you supplied ra and dec you can actually read these back; for example if you don't trust astropy as used above, we can check that the correct decimal values were extracted:

print(q.coneRA)
print(q.coneDec)

183.3083333333333
-13.237499999999999

Note that these variables are read-only so don't try to edit them. If you made a mistake you can use q.editConeSearch(), which takes exactly the same arguments as addConeSearch().

We can also change our minds completely and remove the cone search settings from our query:

q.removeConeSearch()

That's all there is to setting up the cone search aspect of the query.

Selecting which columns to retrieve

When we ran our demo query above, we got a lot of columns containing data, but what if we didn't want them all? Or if we wanted one not included? The default set of columns returned is not exhaustive. Well, of course, you can change what set of columns you get as we'll now discuss. First of all, it would be nice to know what we are going to get by default. These are stored in the defaultCols variable:

q = uq.ObsQuery(silent=False)
q.defaultCols

Resetting query details
Need to get the metadata.

['name',
 'target_id',
 'ra',
 'decl',
 'roll_angle',
 'start_time',
 'stop_time',
 'obs_segment',
 'obsid',
 'bat_exposure',
 'xrt_exposure',
 'xrt_expo_wt',
 'xrt_expo_pc',
 'uvot_exposure']

As you can see, this is just a list, and it's a list of column names. Which is nice and all, but what actually are these columns, and what extra ones are available?

Because we have silent=False you can see from the above that in order to identify the default columns, the Python module grabbed some metadata. It is this metadata which tells us all about the table. Let's take a look at it:

q.metadata

Table metadata is made available in the form of a pandas DataFrame. The exact columns in this do differ slightly for the different tables (and modules; 'LongDescription' is only present for the ObsQuery class), and not all of them are relevant for you - some are needed internally by the Python module, and some are needed by the web interfaces to the databases. The columns you really care about are "ColName" and "Description" which should provide enough details for you to choose whether the default columns are enough, or if you want to edit them.

So, knowing what columns exist, we can now decide which ones we want to obtain, if we didn't just want the default. We add columns using the addCol() function. This takes either * (to get everything), a column name, or a list/tuple of column names. So:

q.addCol('name')
q.colsToGet

['name']

As shown above, the colsToGet variable (which you cannot edit directly) tells you what columns are selected, and you may have noticed that now we've explicitly said what we want, all the default columns are not there. We can easily add them back in though:

q.addCol(q.defaultCols)
q.colsToGet

Cannot add column name; it is already selected.

['name',
 'target_id',
 'ra',
 'decl',
 'roll_angle',
 'start_time',
 'stop_time',
 'obs_segment',
 'obsid',
 'bat_exposure',
 'xrt_exposure',
 'xrt_expo_wt',
 'xrt_expo_pc',
 'uvot_exposure']

You note that we got a warning because we tried to add an existing column in, but it was just a warning and did not stop the other columns being added. We can also remove columns with removeCol, or removeAllCols. So here are some examples:

q.removeCol('name')
q.colsToGet

['target_id',
 'ra',
 'decl',
 'roll_angle',
 'start_time',
 'stop_time',
 'obs_segment',
 'obsid',
 'bat_exposure',
 'xrt_exposure',
 'xrt_expo_wt',
 'xrt_expo_pc',
 'uvot_exposure']

q.removeCol(('obsid', 'stop_time'))
q.colsToGet

['target_id',
 'ra',
 'decl',
 'roll_angle',
 'start_time',
 'obs_segment',
 'bat_exposure',
 'xrt_exposure',
 'xrt_expo_wt',
 'xrt_expo_pc',
 'uvot_exposure']

q.addCol(['cycle', 'soft_version']+q.defaultCols)

Cannot add column target_id; it is already selected.
Cannot add column ra; it is already selected.
Cannot add column decl; it is already selected.
Cannot add column roll_angle; it is already selected.
Cannot add column start_time; it is already selected.
Cannot add column obs_segment; it is already selected.
Cannot add column bat_exposure; it is already selected.
Cannot add column xrt_exposure; it is already selected.
Cannot add column xrt_expo_wt; it is already selected.
Cannot add column xrt_expo_pc; it is already selected.
Cannot add column uvot_exposure; it is already selected.

The latter was a slightly gratuitous demonstration of a way that you can add the default columns, and some others, all in one go. We can also remove the whole lot in one go:

q.removeAllCols()
q.colsToGet

If we were to submit this now we would not get an empty result set, but rather the default columns, which are always request if colsToGet is empty.

If you've been reading everything very carefully, you may be wondering why the results from our demonstration cone search contained the column _r, which has not appeared anywhere in the metadata or our requests. _r is a special column only created for cone searches, and it contains the angular distance of each row from the centre of the search. The units are whatever was given as the units argument to addConeSearch() (arcsec by default).

---

'Advanced' searches

'Advanced' searches are those where we select data based on filters applied to specific columns. This is actually fairly simple, we just define a series of filters to apply to the query and then submit it. These can be in addition to, or instead of the cone search. There are a few things to note before we start:

1. Filters are combined with a logical AND; that is if you define multiple filters, only rows matching all of them will be returned.
2. Each filter applies to a single column but can have two clauses, combined with AND or OR.

This will hopefully all make sense as we go on.

Filters

We add filters using the imaginatively-named addFilter() function, which takes a single argument, a filter definition. A filter definition can either be a dict or a list and has the following components:

• column name
• filter
• value (if applicable)
• combiner (optional)
• filter2 (optional)
• value2 (optional)

I will unpack these in more detail but an example is probably a better helper so, here are two filters created using the two approaches:

filter1 = ('xrt_exposure', '>', 1000, 'OR', '<', 200)

filter2 = {
    "colName": "ra", 
    "filter": ">", 
    "val": 123,
    "combiner": "and" ,
    "filter2": "<",
    "val2": 200
}

If these filters were converted to SQL they would be written as

• filter1: xrt_exposure>1000 OR xrt_exposure<200
• filter2: ra>123 AND ra<200

And if we submitted this query they would combine as:

(xrt_exposure>1000 OR xrt_exposure<200) AND (ra>123 AND ra<200)

If we wanted to create a query using these filters we would do:

q = uq.ObsQuery(silent=False)
q.addFilter(filter1)
q.addFilter(filter2)

Resetting query details
Need to get the metadata.

A filter doesn't have to have all of the elements we used above. You may not want the second part of the filter, so everything from combiner onwards can simply be omitted. Also, some of the filters do not require arguments, so in that case, val (or val2) can be left out. Conversely, the 'BETWEEN' filter requires two arguments and so val must take a 2-element tuple/list.

Some more examples:

q.removeAllFilters()

q.addFilter ( ('xrt_exposure', '<', 2000))

q.addFilter ( ('ra', 'BETWEEN', [100,200]))

q.addFilter ({
    'colName': 'target_id',
    'filter': 'IS NULL',
    'combiner': 'or',
    'filter2': '<',
    'val2': 10000    
})

Here I've introduced the removeAllFilters() function (you can probably guess what it does) and a few more examples of adding filters, including some where we don't need all elements of the filter definition.

The following values are permitted for the 'filter' and 'filter2' keys:

• "<"
• ">"
• "="
• "<="
• ">="
• "LIKE"
• "BETWEEN"
• "NOT LIKE"
• "!="
• "IS NULL"
• "IS NOT NULL"

All of these require a single value, except for 'BETWEEN', which requires 2, and 'IS NULL' and 'IS NOT NULL' which take no values.

Having added filters we can check them:

q.showFilters()

0:  xrt_exposure < 2000
1:  ra BETWEEN [100, 200]
2:  target_id IS NULL OR < 10000

The outputs as you can see are strings.

We can also remove specific filters by their index. Please bear in mind indices will change when you remove a filter!

q.removeFilter(1)

0:  xrt_exposure < 2000
1:  target_id IS NULL OR < 10000

Because silent=False the above function printed out the revised set of filters, if it was True you'd have to do this yourself with another showFilters() call.

Right, enough messing around, let's do an actual query. I'm going to do both a cone and advanced search together here, just to prove you can (and to limit how many rows we get), but of course, you don't have to.

q=uq.ObsQuery(silent=False)
q.addConeSearch(name='GK Per', radius=12, units='arcmin')
q.addFilter( ('xrt_exposure', '>', 3000))
q.isValid()

Resetting query details
Need to get the metadata.

True

Here, we have created a query, requested a cone search of 12' radius, centred on "GK Per", and we have asked to only get those rows where the "xrt_exposure" field is above 3000. I also made use of a function, isValid(). This just checks that the syntax is OK and we haven't done anything silly. Let's submit the query:

q.submit()

Calling DB look-up for rows 0 -- 1000
Resolved 'GK Per' as (52.80004119305, 43.90429720867) via SIMBAD
Received 21 rows.

Again, because we were not silent we got some information printed, but this is also available in variables. I have already discussed accessing details of the name resolution, but how could I find out that 21 rows were returned without having to read from the screen? Like this:

q.numRows

21

Or indeed:

len(q.results)

21

if you prefer. They should be the same. And let's just prove that our filters worked:

q.results

Extra query settings

There are one or two further things to discuss: sorting the results, and controlling how many rows we get. Both of these can be done after the query is complete using pandas functions, or they can be done at query time. Let's start with sorting.

Sorting results

By default, if you did a cone search the results are ordered by increasing distance from the cone centre. If you didn't do a cone search, they are ordered by however they come out of the database. To control this sorting, we use two variables in our class sortCol and sortDir. The former is the name of a column in the table (which we can get via q.metadata, as above), the latter is either "ASC" or "DESC", (for ascending or descending).

Let's have a quick demo, and while I'm here I'll show you that you can unlock a query, rather than fully resetting it, if you just want to change something and resubmit:

q.unlock()
q.sortCol='xrt_exposure'
q.sortDir='DESC'
q.submit()
q.results

Calling DB look-up for rows 0 -- 1000
Resolved 'GK Per' as (52.80004119305, 43.90429720867) via SIMBAD
Received 21 rows.

As you can see, this time, the results are ordered by the "xrt_exposure" column, in descending order. Incidentally, you don't have to actually retrieve the column you sort on, if you don't want to!

How many rows to get

The default behaviour of this module is to get all rows in the database that match your query. This can be a lot, and maybe you don't want them all. You can limit how many are returned using the maxRows variable:

q.unlock()
q.maxRows=3
q.submit()

Calling DB look-up for rows 0 -- 3
Resolved 'GK Per' as (52.80004119305, 43.90429720867) via SIMBAD
Received 3 rows.

This returned us the top three rows matching our query - and note that as we have (just a moment ago) said that our results should be ordered by descending XRT exposure, we should have got the three observations with the longest exposures.

q.results

We did!

I said a moment ago that by default all matching rows are returned, and this is true. Some of you may be wondering why, therefore, the output above repeatedly says: "Calling DB look-up for rows 0 -- 1000" (if you are not wondering this, skip to the next section). Was I lying about getting all rows?

No, I wasn't, and in general what I'm about to say won't matter, but it's here to satisfy your curiosity.

There is a limit to the amount of server resources a single query can consume. In practice this means that if you request some enormous query with lots of rows, the query will be terminated because it uses more memory than is permitted and you will get some unhelpful error (probably an HTTP 500 error). To avoid this, the Python module will never ask for more than 1,000 rows at a time. But this doesn't mean that you only get 1,000 rows; it just means that the Python back end will get your results in chunks, requesting the first 1,000 rows and then (if necessary) the next 1,000 etc., giving (non-silent) output such as:

Calling DB look-up for rows 0 -- 1000 Calling DB look-up for rows 1001 -- 2000 Calling DB look-up for rows 2001 -- 3000

The results of these calls will be stitched together for you and this entire process would be completely invisible if we hadn't said silent=False.

---

Retrieving products from a query

Having identified observations using a query, you may wish to download them. You can do this by calling the downloadObsData() function. This is literally just a wrapper the function of the same name in the data module and as already warned, I'm not going to redocument that here. Arguments are passed through to data.downloadObsData() as **kwargs, with just three exceptions. These exceptions apply to almost every data product function provided throughout the query module, so I will refer back to this section a few times on subsequent page. These exceptions are:

• You do not specify silent and verbose; these are properties of your query object, and are set from them.
• You do not supply the identifier of the object(s) you want the products for; the products are retrieved for the objects in your query's results table.
• There is an optional subset argument, which lets you specify a subset of the results table for which you want data products.

This last point we will return to in a moment

First, let's do a simple demonstration of this. I will deliberately execute a query that doesn't get too many rows.

q = uq.ObsQuery(silent=False)
q.addConeSearch(name='GRB 210205A', radius=300)
q.submit()
q.results

Resetting query details
Need to get the metadata to check the query is valid.
Calling DB look-up for rows 0 -- 1000
Resolved 'GRB 210205A' as (347.219333, 56.296528) via SIMBAD
Received 2 rows.

As you can see, this gave us two rows, and we can save the observations simply enough.

q.downloadObsData(destDir='/tmp/APIDemo_download1',
                 instruments=('BAT', 'XRT'),
                 getTDRSS=True)

Making directory /tmp/APIDemo_download1
Downloading 2 datasets
Making directory /tmp/APIDemo_download1/01030629001
Making directory /tmp/APIDemo_download1/01030629001/bat/
Making directory /tmp/APIDemo_download1/01030629001/bat/hk/
Making directory /tmp/APIDemo_download1/01030629001/bat/masktag/
Making directory /tmp/APIDemo_download1/01030629001/bat/rate/
Making directory /tmp/APIDemo_download1/01030629001/bat/survey/
Making directory /tmp/APIDemo_download1/01030629001/xrt/
Making directory /tmp/APIDemo_download1/01030629001/xrt/event/
Making directory /tmp/APIDemo_download1/01030629001/xrt/hk/
Making directory /tmp/APIDemo_download1/01030629001/auxil/

Downloading 01030629001:   0%|          | 0/46 [00:00<?, ?files/s]

Making directory /tmp/APIDemo_download1/01030629000
Making directory /tmp/APIDemo_download1/01030629000/bat/
Making directory /tmp/APIDemo_download1/01030629000/bat/event/
Making directory /tmp/APIDemo_download1/01030629000/bat/hk/
Making directory /tmp/APIDemo_download1/01030629000/bat/masktag/
Making directory /tmp/APIDemo_download1/01030629000/bat/products/
Making directory /tmp/APIDemo_download1/01030629000/bat/rate/
Making directory /tmp/APIDemo_download1/01030629000/bat/survey/
Making directory /tmp/APIDemo_download1/01030629000/xrt/
Making directory /tmp/APIDemo_download1/01030629000/xrt/event/
Making directory /tmp/APIDemo_download1/01030629000/xrt/hk/
Making directory /tmp/APIDemo_download1/01030629000/auxil/
Making directory /tmp/APIDemo_download1/01030629000/tdrss/

Downloading 01030629000:   0%|          | 0/90 [00:00<?, ?files/s]

The arguments I supplied to downloadObsData() are the standard arguments for data.downloadObsData() which are documented here.

One thing that is worth noting is that, since downloadObsData() uses the observation identifier to work out which data you are asking for, if your query did not retrieve the relevant column, this function would fail. If you are manually selecting columns and want to ensure you have the necessary column you can always look at the metadata (described above). If the table in question contains observation identifiers then the metadata will include "IsObsCol", which will be 1 for the relevant column.

Getting products for only some rows

After performing a query you may realise that you don't want to get data for all of the rows, but only some of them. You could repeat the query, of course, with extra filtering, but this would be a bit wasteful. The subset parameter solves this; it exists for all functions for getting products via the query module, and allows you to define a subset of rows for which products should be retrieved.

The argument itself takes a pandas Series of bools, identifying the rows, which sounds complicated, but actually you just need to give it a pandas filter expression. It is beyond the scope of this documentation to describe those in detail, but the examples below should give you a primer.

First, I'm going to do a cone search around GK Per, ordering by exposure (longest first)

q = uq.ObsQuery(silent=False)
q.addConeSearch(name='GK Per', radius=300)
q.sortCol = 'xrt_exposure'
q.sortDir='DESC'
q.submit()
q.results

Resetting query details
Need to get the metadata.
Calling DB look-up for rows 0 -- 1000
Resolved 'GK Per' as (52.80004119305, 43.90429720867) via SIMBAD
Received 145 rows.

Now, looking at this, I want to get only those rows that have more than 6ks of XRT data. That is, those rows where q.results['xrt_exposure']>6000, and that is my subset definition:

q.downloadObsData(destDir='/tmp/APIDemo_download2',
                  subset=q.results['xrt_exposure']>6000,
                  instruments=('XRT',),
                  getAuxil=False)

Making directory /tmp/APIDemo_download2
Downloading 4 datasets
Making directory /tmp/APIDemo_download2/00030842017
Making directory /tmp/APIDemo_download2/00030842017/xrt/
Making directory /tmp/APIDemo_download2/00030842017/xrt/event/
Making directory /tmp/APIDemo_download2/00030842017/xrt/hk/
Making directory /tmp/APIDemo_download2/00030842017/xrt/ssin/

Downloading 00030842017:   0%|          | 0/109 [00:00<?, ?files/s]

Making directory /tmp/APIDemo_download2/00030842015
Making directory /tmp/APIDemo_download2/00030842015/xrt/
Making directory /tmp/APIDemo_download2/00030842015/xrt/event/
Making directory /tmp/APIDemo_download2/00030842015/xrt/hk/
Making directory /tmp/APIDemo_download2/00030842015/xrt/ssin/

Downloading 00030842015:   0%|          | 0/99 [00:00<?, ?files/s]

Making directory /tmp/APIDemo_download2/00030842012
Making directory /tmp/APIDemo_download2/00030842012/xrt/
Making directory /tmp/APIDemo_download2/00030842012/xrt/event/
Making directory /tmp/APIDemo_download2/00030842012/xrt/hk/
Making directory /tmp/APIDemo_download2/00030842012/xrt/ssin/

Downloading 00030842012:   0%|          | 0/99 [00:00<?, ?files/s]

Making directory /tmp/APIDemo_download2/00030842005
Making directory /tmp/APIDemo_download2/00030842005/xrt/
Making directory /tmp/APIDemo_download2/00030842005/xrt/event/
Making directory /tmp/APIDemo_download2/00030842005/xrt/hk/
Making directory /tmp/APIDemo_download2/00030842005/xrt/ssin/

Downloading 00030842005:   0%|          | 0/114 [00:00<?, ?files/s]

If you want to check your filter expression before actually using it, you can use the .loc property of a DataFrame like this:

q.results.loc[q.results['xrt_exposure']>6000]

And you can see that the DataFrame we got back contains only the rows we wanted.

I'll give a few more examples of creating subsets, and rather then fill up your /tmp area by downloading, I'll define them as variables and print them using .loc, but you can just put my examples straight into the subset argument of downloadObsData. I'll include such code but commented out, below.

First, let's show you how to apply multiple filters because this always take me 3 attempts to get right. Let's say we only want results with more than 6ks of XRT exposure, but less than 7ks.

subset = (q.results['xrt_exposure']>6000)&(q.results['xrt_exposure']<7000)
q.results.loc[subset]
# Uncomment the following if you want:

# q.downloadObsData(destDir='/tmp/APIDemo_download3',
#                   subset=(q.results['xrt_exposure']>6000)&(q.results['xrt_exposure']<7000),
#                   instruments=('XRT',),
#                   getAuxil=False)

And lastly, the isin function which is really handy as well. This lets us make a subset by giving some values we want a column to contain.So, for example, imagine I only wanted to download the data in the above for those cases with a "target_id" of 81445, 45767 or 81637. I could do that as follows:

myTargs = (81445, 45767, 81637)
subset=q.results['target_id'].isin(myTargs)
q.results.loc[q.results['target_id'].isin(myTargs)]
# q.downloadObsData(destDir='/tmp/APIDemo_download4',
#                   subset=q.results['target_id'].isin(myTargs),
#                   instruments=('XRT',),
#                   getAuxil=False)

Astute readers will have realised that this effectively takes place of the IN operator in SQL, which is not supported by the addFilter command (largely for security reasons).

UK Swift Science Data Centre
Last updated 2024 May 24
Web page maintained by Vatsalya Maddi
E-mail: swift help
Please read our privacy notice.