Science Archive Documentation
The LCO science archive is the repository for all science data (raw, processed, and ancillary) and data products (source tables, final calibration files) produced by the LCO network. There are two ways to access data stored in the archive: the API (application programming interface) and its web frontend UI (user interface).
The archive API is a web service that provides methods to query the metadata associated with data products stored in the archive. Query results are returned in JSON, so they can be easily parsed by a wide variety of libraries and programs. When the desired data products are found, they can be downloaded using the urls in the metadata. The archive UI is a user-friendly interface to the API. To use the UI, users must login with the same username and password required for the observatory Portal. To use the API, users must submit their Portal username and password to the API's authentication endpoint.
Science Data Products
All image FITS files are Rice-compressed ("fpacked") to reduce file-size and increase transmission speed, and these bear fz file extensions when downloaded. These can be decompressed with CFITSIO funpack, although many common astronomy software applications (e.g. ds9) can interpret the compressed format.
The FITS header keywords and their meanings are given in this table.
Raw frames from science imagers should appear in the archive within a few minutes of being taken (subject to internet latency and occasional outages). Raw frames have e00 (or b00 or d00) suffixes in their basenames. Frames from SBIG cameras (instrument code kbXX) and Spectral cameras (instrument code fsXX) are stored as 2-dimensional (fpacked) FITS frames. Frames from Sinistro cameras (instrument code flXX) are stored as 3-dimensional (fpacked) FITS frames in which the third dimension is used to separate the four separately read quadrants. Raw frames from the FLOYDS spectrograph cameras (enXX) are also stored as 2-dimensional (fpacked) FITS frames.
Quicklook reduced frames are produced upon the arrival of each raw science imager frame and are available from the archive within approximately ten minutes later. These reduction use the best-at-time calibration files (bias/dark/flat). Quicklook reduced frames using the ORAC-DR pipeline have e10 suffixes in their basenames, and quicklook reduced frames using the BANZAI pipeline have e11 suffixes in their basenames. Quicklook frames are deleted after one week.
Final reduced frames are produced at the conclusion of the night at each telescope, when all science and calibration frames have been transferred to LCOGT headquarters. These use the best-for-night calibration files (bias/dark/flat). Final reduced frames using the ORAC-DR pipeline have e90 suffixes in their basenames, and final reduced frames using the BANZAI pipeline have e91 suffixes in their basenames.
In the UI, the calibration frames (bias, dark, flat, etc) used in the processing of a science frame are revealed by clicking on the '+' symbol in the left-most column. Conversely, clicking on the '+' symbol left of a calibration frame will unfurl a list of all other frames to which that calibration frame was applied. In the API, the related calibration frames can be retrieved by using the 'related_frames' key in the API payload.
BANZAI pipeline processed images are multi-extension FITS files. Each file includes three extensions: the reduced image, the catalog of sources (CAT) detected by SEP in a FITS binary table, and the bad pixel mask (BPM) in an array. The catalog lists the pixel positions (X, Y), semi-major and semi-minor axes (A, B), positions angles (THETA), fluxes and errors (FLUX, FLUXERR) of each source.
The BANZAI pipeline began running on 1 April 2016.
Reduced data processed with the ORAC-DR pipeline are available for the period May 1, 2014 through May 31, 2016. These are files with the suffix e90 (final reduced) and e10 (quicklook reduced). Catalogs generated by SExtractor are archived as separate files with suffixes such as _e90_cat.fits_.
The Python/Pyraf FLOYDS pipeline processes data taken with the FLOYDS low-resolution spectrograph. The data product, including intermediate and final reduced files, as well as guider images, comes packaged in a .tar.gz file. This package will be associated with the data products that were used to create it, and vice versa.
Proprietary Data Policy
Data acquired for science projects sponsored by LCOGT Science Collaboration institutions and other contracted institutions is subject to a 12-month proprietary period. During this period, which begins on the date the data are acquired, only PIs and registered Co-Is of a science project have access to that project's data (and associated metadata).
Data acquired for educational projects and public outreach events are made public immediately. In addition, calibration observations (bias frames, dark frames, flat fields, etc.) and observations of photometric standard fields are also available to all users.
Filename elements and search filters
The file name structure is: (site)(tel)-(instr)-YYYYMMDD-(frame)-(type)(red.level).fits . Thus, a representative frame name might be lsc1m005-fl03-20160401-0666-e91.fits.
The ORAC pipeline creates files named ...e90_cat.fits as the (type=) CATALOG list of sources. The FLOYDS pipeline creates a tar file (PROPOSAL_ID....tar) containing all the data products associated with a given FLOYDS science exposure
|Site Code||Observatory Location|
|BPL||Back Parking Lot Goleta, CA|
|COJ||Siding Spring Observatory Coonabarabran, AUSTRALIA|
|CPT||South African Astronomical Observatory Sutherland, SOUTH AFRICA|
|ELP||McDonald Observatory Fort Davis, TX|
|LSC||Cerro Tololo Interamerican Observatory La Serena, CHILE|
|NGQ||Ali Observatory Shiquanhe, Tibet Autonomous Region, CHINA|
|OGG||Halekala Observatory Maui, HI|
|SQA||Sedgwick Reserve Santa Ynez, CA|
|TFN||Teide Observatory Tenerife, SPAIN|
|Telescope Code||Telescope Description|
|0m403||0.4m telescope (0m4a) in 2m enclosure (clma) at COJ|
|0m404||0.4m telescope (0m4c) in 2m enclosure (clma) at OGG|
|0m405||0.4m telescope (0m4b) in 2m enclosure (clma) at COJ|
|0m406||0.4m telescope (0m4b) in 2m enclosure (clma) at OGG|
|0m409||0.4m telescope (0m4b) in Aqawan (aqwa) at LSC|
|0m410||0.4m telescope (0m4b) in Aqawan (aqwa) at TFN|
|0m414||0.4m telescope (0m4a) in Aqawan (aqwa) at TFN|
|0m801||0.8m telescope (0m8a) in dome (doma) at SQA|
|1m002||1.0m telescope (1m0a) in dome (doma) at BPL|
|1m003||1.0m telescope (1m0a) in dome (domb) at COJ|
|1m004||1.0m telescope (1m0a) in dome (domc) at LSC|
|1m005||1.0m telescope (1m0a) in dome (doma) at LSC|
|1m008||1.0m telescope (1m0a) in dome (doma) at ELP|
|1m009||1.0m telescope (1m0a) in dome (domb) at LSC|
|1m010||1.0m telescope (1m0a) in dome (doma) at CPT|
|1m011||1.0m telescope (1m0a) in dome (doma) at COJ|
|1m012||1.0m telescope (1m0a) in dome (domc) at CPT|
|1m013||1.0m telescope (1m0a) in dome (domb) at CPT|
|2m001||2.0m telescope (2m0a) in 2m enclosure (clma) at OGG|
|2m002||2.0m telescope (2m0a) in 2m enclosure (clma) at COJ|
Instrument codes Instrument Detector enXX FLOYDS Andor Newton 540P camera with e2v CCD42-10, 2048 X 512, 13.5 um pix flXX Sinistro Fairchild CCD486 BI, 4096 X 4097, 15 um pix fsXX Spectral Imager Fairchild CCD486 BI, 4096 X 4097, 15 um pix kbXX SBIG STX-6303 3072 X 2048, 9 um pix kbXX SBIT STX-16803 4096 X 4096, 9 um pix (no longer in service)
YYYYMMDD: local date at start of night
Frame number: increases sequentially, per night
Type Code Search Value Type of Observation a ARC Spectrograph wavelength calibration b BIAS Bias level calibration d DARK Dark level calibration e EXPOSE Imager Science exposure x EXPERIMENTAL Experimental Frame - Engineering w LAMPFLAT Spectrograph flatfield calibration f SKYFLAT Imager flat calibration taken on twilight sky e SPECTRUM Spectrograph Science exposure s STANDARD Photometric Standard field exposure
Reduction Level Description 00 Raw frame 10 Quicklook data, processed with ORAC pipeline 11 Quicklook data, processed with BANZAI pipeline 90 Reduced data, processed with ORAC pipeline 91 Reduced data, processed with BANZAI pipeline User Interface
By signing in to the Science Archive UI, you will be able to search for data acquired for projects for which you are the PI or a Co-I. Use the same username and password that you do to access the observatory portal. Public data is available without logging in.
On the left side of the user interface are a set of fields which can be used to narrow down the results in the main table. Starting from the top, these fields are:
This widget specifies the time period from which results will appear. There are a few pre defined date ranges to choose from, including this semester, last semester and the last month. If you need a more specific time range, choose "Custom Range"
This is the proposal from which the frames were taken. If you are logged in, your proposals will appear near the top. Public proposals are listed as well.
The points fields allow you to search for data by position on the sky. Entering an object designation (eg. M42) in the Lookup field will use an online service to search for the object, return the RA/Dec and populate the RA and Dec search fields. You can also manually enter position in decimal or sexagesimal.
This field directly searches the data's OBJECT fits keyword. It does not do any kind of spatial search.
This is the type of observation.
The level of processing the data has gone through.
The site at which the data was taken
The telescope class with which the data was taken
The instrument with which the data was taken (camera)
The filter used to take the observation
The exposure time of the image
On the top right of the table there is a row of buttons.
The refresh button will reload the data in the table, preserving filters.
The columns button will allow you to pick which columns to show in the table. Some columns are hidden by default.
The Export table data button will export the displayed data in the table in a variety of formats, including csv and excel.
To download data, select the row that contains the data you wish do download using the checkbox. Multiple rows can be selected at once. Rows will stay selected if you change the search filters, so you can gather as much data as you'd like.
Once you have selected the date you wish to download, click the download button. You will be prompted to download a .zip file containing your data.
Alternatively, if you wish you can download a wget script instead that can be run as an executable on your machine. To download this script, select "wget script" from the dropdown menu that is activated by pressing the down caret next to the download button.
There is a daily request limit set on the archive. All logged in users, regardless if using the archive UI or API, are limited to 5000 requests a day. Non authenticated users are limited to 3500 requests a day.
Actions that count as a request include: performing a new search, changing pages, and downloading a zip file. While using the API, a single request is any time your code performs a http request against the api.
API The API documentation is located on the LCOGT developers page Changelog LCOGT Archive Changelog Date Change/Action May 1, 2014 Start of LCOGT V1.0 Operations April 1, 2016 Initial release of LCOGT Science Archive April 12, 2016 Inclusion of all guide images in FLOYDS data products June 13, 2016 Removal of ORAC-DR paths from data product crawler Aug 8, 2016 Added ability to export displayed table data as well as view headers directly in the archive interface May 16, 2017 The fluxes of catalog sources in BANZAI-processed images are now being correctly calculated in apertures with diameters in arcseconds. Prior to this change, the fluxes were calculated in apertures with diameters in pixels. The pipeline version number (PIPEVER keyword) was incremented to 0.7.9dev1197.