Archive

Science Archive Documentation

The LCO science archive is the repository for all science data (raw, processed, and ancillary) and data products (source tables, final calibration files) produced by the LCO network. There are two ways to access data stored in the archive: the API (application programming interface) and its web frontend UI (user interface).

The archive API is a web service that provides methods to query the metadata associated with data products stored in the archive. Query results are returned in JSON, so they can be easily parsed by a wide variety of libraries and programs. When the desired data products are found, they can be downloaded using the urls in the metadata. The archive UI is a user-friendly interface to the API. To use the UI, users must login with the same username and password required for the observatory Portal. To use the API, users must submit their Portal username and password to the API's authentication endpoint.

The use of the LCO archive and/or data implies acceptance of LCO's terms of use policy.

Science Data Products

All image FITS files are Rice-compressed ("fpacked") to reduce file-size and increase transmission speed, and these bear fz file extensions when downloaded. These can be decompressed with CFITSIO funpack, although many common astronomy software applications (e.g. ds9) can interpret the compressed format.

The FITS header keywords and their meanings are given in this table.

Raw frames from science imagers should appear in the archive within a few minutes of being taken (subject to internet latency and occasional outages). Raw frames have e00 (or b00 or d00) suffixes in their basenames. Frames from SBIG cameras (instrument code kbXX) and Spectral cameras (instrument code fsXX) are stored as 2-dimensional (fpacked) FITS frames. Frames from Sinistro cameras (instrument code flXX) are stored as 3-dimensional (fpacked) FITS frames in which the third dimension is used to separate the four separately read quadrants. Raw frames from the FLOYDS spectrograph cameras (enXX) are also stored as 2-dimensional (fpacked) FITS frames.

Quicklook reduced frames are produced upon the arrival of each raw science imager frame and are available from the archive within approximately ten minutes later. These reduction use the best-at-time calibration files (bias/dark/flat). Quicklook reduced frames using the ORAC-DR pipeline have e10 suffixes in their basenames, and quicklook reduced frames using the BANZAI pipeline have e11 suffixes in their basenames. Quicklook frames are deleted after one week.

Final reduced frames are produced at the conclusion of the night at each telescope, when all science and calibration frames have been transferred to LCOGT headquarters. These use the best-for-night calibration files (bias/dark/flat). Final reduced frames using the ORAC-DR pipeline have e90 suffixes in their basenames, and final reduced frames using the BANZAI pipeline have e91 suffixes in their basenames.

In the UI, the calibration frames (bias, dark, flat, etc) used in the processing of a science frame are revealed by clicking on the '+' symbol in the left-most column. Conversely, clicking on the '+' symbol left of a calibration frame will unfurl a list of all other frames to which that calibration frame was applied. In the API, the related calibration frames can be retrieved by using the 'related_frames' key in the API payload.

BANZAI pipeline

BANZAI pipeline processed images are multi-extension FITS files. Each file includes three extensions: the reduced image, the catalog of sources (CAT) detected by SEP in a FITS binary table, and the bad pixel mask (BPM) in an array. The catalog lists the pixel positions (X, Y), semi-major and semi-minor axes (A, B), positions angles (THETA), fluxes and errors (FLUX, FLUXERR) of each source.

The BANZAI pipeline began running on 1 April 2016; all frames going back to the start of LCOGT operations (1 May 2014) will be re-processed by this pipeline by the end of 2016.

ORAC-DR pipeline

Reduced data processed with the ORAC-DR pipeline are available for the period May 1, 2014 through May 31, 2016. These are files with the suffix e90 (final reduced) and e10 (quicklook reduced). Catalogs generated by SExtractor are archived as separate files with suffixes such as _e90_cat.fits_.

FLOYDS pipeline

The Python/Pyraf FLOYDS pipeline processes data taken with the FLOYDS low-resolution spectrograph. The data product, including intermediate and final reduced files, as well as guider images, comes packaged in a .tar.gz file. This package will be associated with the data products that were used to create it, and vice versa.

Proprietary Data Policy

Data acquired for science projects sponsored by LCOGT Science Collaboration institutions and other contracted institutions is subject to a 12-month proprietary period. During this period, which begins on the date the data are acquired, only PIs and registered Co-Is of a science project have access to that project's data (and associated metadata).

Data acquired for educational projects and public outreach events are made public immediately. In addition, calibration observations (bias frames, dark frames, flat fields, etc.) and observations of photometric standard fields are also available to all users.

Filename elements and search filters

The file name structure is: (site)(tel)-(instr)-YYYYMMDD-(frame)-(type)(red.level).fits . Thus, a representative frame name might be lsc1m005-fl03-20160401-0666-e91.fits.

Exceptions

The ORAC pipeline creates files named ...e90_cat.fits as the (type=) CATALOG list of sources. The FLOYDS pipeline creates a tar file (PROPOSAL_ID....tar) containing all the data products associated with a given FLOYDS science exposure

Site

Site code Observatory Location
BPL Back Parking Lot Goleta, CA
COJ Siding Spring Observatory Coonabarabran, AUSTRALIA
CPT South African Astronomical Observatory Sutherland, SOUTH AFRICA
ELP McDonald Observatory Fort Davis, TX
LSC Cerro Tololo Interamerican Observatory La Serena, CHILE
NGQ Ali Observatory Shiquanhe, Tibet Autonomous Region, CHINA
OGG Halekala Observatory Maui, HI
SQA Sedgwick Reserve Santa Ynez, CA
TFN Teide Observatory Tenerife, SPAIN

Telescope

Telescope Code Telescope Description
0m403 0.4m telescope (0m4a) in 2m enclosure (clma) at COJ
0m404 0.4m telescope (0m4c) in 2m enclosure (clma) at OGG
0m405 0.4m telescope (0m4b) in 2m enclosure (clma) at COJ
0m406 0.4m telescope (0m4b) in 2m enclosure (clma) at OGG
0m409 0.4m telescope (0m4b) in Aqawan (aqwa) at LSC
0m410 0.4m telescope (0m4b) in Aqawan (aqwa) at TFN
0m414 0.4m telescope (0m4a) in Aqawan (aqwa) at TFN
0m801 0.8m telescope (0m8a) in dome (doma) at SQA
1m002 1.0m telescope (1m0a) in dome (doma) at BPL
1m003 1.0m telescope (1m0a) in dome (domb) at COJ
1m004 1.0m telescope (1m0a) in dome (domc) at LSC
1m005 1.0m telescope (1m0a) in dome (doma) at LSC
1m008 1.0m telescope (1m0a) in dome (doma) at ELP
1m009 1.0m telescope (1m0a) in dome (domb) at LSC
1m010 1.0m telescope (1m0a) in dome (doma) at CPT
1m011 1.0m telescope (1m0a) in dome (doma) at COJ
1m012 1.0m telescope (1m0a) in dome (domc) at CPT
1m013 1.0m telescope (1m0a) in dome (domb) at CPT
2m001 2.0m telescope (2m0a) in 2m enclosure (clma) at OGG
2m002 2.0m telescope (2m0a) in 2m enclosure (clma) at COJ

Instrument

Instrument codes Instrument Detector
enXX FLOYDS Andor Newton 540P camera with e2v CCD42-10, 2048 X 512, 13.5 um pix
flXX Sinistro Fairchild CCD486 BI, 4096 X 4097, 15 um pix
fsXX Spectral Imager Fairchild CCD486 BI, 4096 X 4097, 15 um pix
kbXX SBIG STX-6303 3072 X 2048, 9 um pix
kbXX SBIT STX-16803 4096 X 4096, 9 um pix (no longer in service)

Date

YYYYMMDD: local date at start of night

Frame Number

Frame number: increases sequentially, per night

Observation Type

Type Code Search Value Type of Observation
a ARC Spectrograph wavelength calibration
b BIAS Bias level calibration
d DARK Dark level calibration
e EXPOSE Imager Science exposure
x EXPERIMENTAL Experimental Frame - Engineering
w LAMPFLAT Spectrograph flatfield calibration
f SKYFLAT Imager flat calibration taken on twilight sky
e SPECTRUM Spectrograph Science exposure
s STANDARD Photometric Standard field exposure

Reduction Level

Reduction Level Description
00 Raw frame
10 Quicklook data, processed with ORAC pipeline
11 Quicklook data, processed with BANZAI pipeline
90 Reduced data, processed with ORAC pipeline
91 Reduced data, processed with BANZAI pipeline

User Interface

By signing in to the Science Archive UI, you will be able to search for data acquired for projects for which you are the PI or a Co-I. Use the same username and password that you do to access the observatory portal. Public data is available without logging in.

Search filters

On the left side of the user interface are a set of fields which can be used to narrow down the results in the main table. Starting from the top, these fields are:

Date (calendar)

This widget specifies the time period from which results will appear. There are a few pre defined date ranges to choose from, including this semester, last semester and the last month. If you need a more specific time range, choose "Custom Range"

Proposal

This is the proposal from which the frames were taken. If you are logged in, your proposals will appear near the top. Public proposals are listed as well.

Point

The points fields allow you to search for data by position on the sky. Entering an object designation (eg. M42) in the Lookup field will use an online service to search for the object, return the RA/Dec and populate the RA and Dec search fields. You can also manually enter position in decimal or sexagesimal.

Object

This field directly searches the data's OBJECT fits keyword. It does not do any kind of spatial search.

Obstype

This is the type of observation.

  • ARC Spectrograph calibration
  • BIAS A Bias frame
  • CATALOG A source extracted catalog
  • DARK A Dark frame
  • EXPERIMENTAL An experimental frame, usually not for science
  • EXPOSE Image data, used for science
  • LAMPFLAT Spectrograph calibration
  • SKYFLAT Calibration frame
  • SPECTRUM Spectrograph data
  • STANDARD Data taken from a standard target
Reduction level

The level of processing the data has gone through.

  • 00 Raw data, no processing
  • 10 Quicklook data, processed by the ORAC pipeline with the available calibration frames at the time
  • 11 Quicklook data, processed by the BANZAI pipeline with the available calibration frames at the time
  • 90 Reduced data, processed by the ORAC pipeline with the best calibration frames
  • 91 Reduced data, processed by the BANZAI pipeline with the best calibration frames
Site

The site at which the data was taken

Telescope

The telescope class with which the data was taken

Instrument

The instrument with which the data was taken (camera)

Filter

The filter used to take the observation

Exposure Time

The exposure time of the image

Other User Interface Elements

On the top right of the table there is a row of buttons.

The refresh button will reload the data in the table, preserving filters.

The columns button will allow you to pick which columns to show in the table. Some columns are hidden by default.

The Export table data button will export the displayed data in the table in a variety of formats, including csv and excel.

Search Examples

Search for all reduced frames acquired for a particular project in the last 30 days.

  1. In the upper left corner of the UI, click on the calendar box.
  2. In the drop-down menu that appears, click on "Last 30 days".
  3. Just below that calendar box, click on the "Proposal" field (where it says "All").
  4. From the drop-down menu, select your proposal ID. For example, CON2016A-007.
  5. Further down in the column of search filters, click on the "Reduction Level" field (where it says "All").
  6. From the drop-down menu, select "Reduced (experimental)" to select frames processed through the BANZAI data reduction pipeline.
  7. To select all of the frames displayed in the table, click on the check box at the top of the column of check boxes (left of the "Basename" column).
  8. To download the selected frames, click on the blue "Download ##" button (just above the column of checkboxes), where "##" will be number of frames you selected.

Search for all frames in the vicinity of a particular target in the last year.

Downloading data

To download data, select the row that contains the data you wish do download using the checkbox. Multiple rows can be selected at once. Rows will stay selected if you change the search filters, so you can gather as much data as you'd like.

Once you have selected the date you wish to download, click the download button. You will be prompted to download a .zip file containing your data.

Alternatively, if you wish you can download a wget script instead that can be run as an executable on your machine. To download this script, select "wget script" from the dropdown menu that is activated by pressing the down caret next to the download button.

Request Limits

There is a daily request limit set on the archive. All logged in users, regardless if using the archive UI or API, are limited to 5000 requests a day. Non authenticated users are limited to 3500 requests a day.

Actions that count as a request include: performing a new search, changing pages, and downloading a zip file. While using the API, a single request is any time your code performs a http request against the api.

API

Changelog

LCOGT Archive Changelog

Date Change/Action
May 1, 2014 Start of LCOGT V1.0 Operations
April 1, 2016 Initial release of LCOGT Science Archive
April 12, 2016 Inclusion of all guide images in FLOYDS data products
June 13, 2016 Removal of ORAC-DR paths from data product crawler
Aug 8, 2016 Added ability to export displayed table data as well as view headers directly in the archive interface
May 16, 2017 The fluxes of catalog sources in BANZAI-processed images are now being correctly calculated in apertures with diameters in arcseconds. Prior to this change, the fluxes were calculated in apertures with diameters in pixels. The pipeline version number (PIPEVER keyword) was incremented to 0.7.9dev1197.