The LCO science archive is the repository for all science data (raw, processed, and ancillary) and data products (source tables, final calibration files) produced by the LCO network. There are two ways to access data stored in the archive: the API (application programming interface) and its web frontend UI (user interface).
The archive API is a web service that provides methods to query the metadata associated with data products stored in the archive. Query results are returned in JSON, so they can be easily parsed by a wide variety of libraries and programs. When the desired data products are found, they can be downloaded using the urls in the metadata. The archive UI is a user-friendly interface to the API. To use the UI, users must login with the same username and password required for the observatory Portal. To use the API, users must submit their Portal username and password to the API's authentication endpoint.
Science Data Products
All image FITS files are Rice-compressed ("fpacked") to reduce file-size and increase transmission speed, and these bear fz file extensions when downloaded. These can be decompressed with CFITSIO funpack, although many common astronomy software applications (e.g. ds9) can interpret the compressed format.
The FITS header keywords and their meanings are given in this table.
Raw frames from science imagers should appear in the archive within a few minutes of being taken (subject to internet latency and occasional outages). Raw frames have e00 (or b00 or d00) suffixes in their basenames. Frames from SBIG cameras (instrument code kbXX) and Spectral cameras (instrument code fsXX) are stored as 2-dimensional (fpacked) FITS frames. Frames from Sinistro cameras (instrument code flXX) are stored as 3-dimensional (fpacked) FITS frames in which the third dimension is used to separate the four separately read quadrants. Raw frames from the FLOYDS spectrograph cameras (enXX) are also stored as 2-dimensional (fpacked) FITS frames.
Quicklook reduced frames are produced upon the arrival of each raw science imager frame and are available from the archive within approximately ten minutes later. These reduction use the best-at-time calibration files (bias/dark/flat). Quicklook reduced frames using the ORAC-DR pipeline have e10 suffixes in their basenames, and quicklook reduced frames using the BANZAI pipeline have e11 suffixes in their basenames. Quicklook frames are deleted after one week.
Final reduced frames are produced at the conclusion of the night at each telescope, when all science and calibration frames have been transferred to LCOGT headquarters. These use the best-for-night calibration files (bias/dark/flat). Final reduced frames using the ORAC-DR pipeline have e90 suffixes in their basenames, and final reduced frames using the BANZAI pipeline have e91 suffixes in their basenames.
In the UI, the calibration frames (bias, dark, flat, etc) used in the processing of a science frame are revealed by clicking on the '+' symbol in the left-most column. Conversely, clicking on the '+' symbol left of a calibration frame will unfurl a list of all other frames to which that calibration frame was applied. In the API, the related calibration frames can be retrieved by using the 'related_frames' key in the API payload.
BANZAI pipeline processed images are multi-extension FITS files. Each file includes three extensions: the reduced image, the catalog of sources (CAT) detected by SEP in a FITS binary table, and the bad pixel mask (BPM) in an array. The catalog lists the pixel positions (X, Y), semi-major and semi-minor axes (A, B), positions angles (THETA), fluxes and errors (FLUX, FLUXERR) of each source.
The BANZAI pipeline began running on 1 April 2016; all frames going back to the start of LCOGT operations (1 May 2014) will be re-processed by this pipeline by the end of 2016.
Reduced data processed with the ORAC-DR pipeline are available for the period May 1, 2014 through May 31, 2016. These are files with the suffix e90 (final reduced) and e10 (quicklook reduced). Catalogs generated by SExtractor are archived as separate files with suffixes such as _e90_cat.fits_.
The Python/Pyraf FLOYDS pipeline processes data taken with the FLOYDS low-resolution spectrograph. The data product, including intermediate and final reduced files, as well as guider images, comes packaged in a .tar.gz file. This package will be associated with the data products that were used to create it, and vice versa.
Proprietary Data Policy
Data acquired for science projects sponsored by LCOGT Science Collaboration institutions and other contracted institutions is subject to a 12-month proprietary period. During this period, which begins on the date the data are acquired, only PIs and registered Co-Is of a science project have access to that project's data (and associated metadata).
Data acquired for educational projects and public outreach events are made public immediately. In addition, calibration observations (bias frames, dark frames, flat fields, etc.) and observations of photometric standard fields are also available to all users.
Filename elements and search filters
The file name structure is: (site)(tel)-(instr)-YYYYMMDD-(frame)-(type)(red.level).fits . Thus, a representative frame name might be lsc1m005-fl03-20160401-0666-e91.fits.
The ORAC pipeline creates files named ...e90_cat.fits as the (type=) CATALOG list of sources. The FLOYDS pipeline creates a tar file (PROPOSAL_ID....tar) containing all the data products associated with a given FLOYDS science exposure
||Back Parking Lot
||Siding Spring Observatory
||South African Astronomical Observatory
||Sutherland, SOUTH AFRICA
||Fort Davis, TX
||Cerro Tololo Interamerican Observatory
||La Serena, CHILE
||Shiquanhe, Tibet Autonomous Region, CHINA
||Santa Ynez, CA
||0.4m telescope (0m4a) in 2m enclosure (clma) at COJ
||0.4m telescope (0m4c) in 2m enclosure (clma) at OGG
||0.4m telescope (0m4b) in 2m enclosure (clma) at COJ
||0.4m telescope (0m4b) in 2m enclosure (clma) at OGG
||0.4m telescope (0m4b) in Aqawan (aqwa) at LSC
||0.4m telescope (0m4b) in Aqawan (aqwa) at TFN
||0.4m telescope (0m4a) in Aqawan (aqwa) at TFN
||0.8m telescope (0m8a) in dome (doma) at SQA
||1.0m telescope (1m0a) in dome (doma) at BPL
||1.0m telescope (1m0a) in dome (domb) at COJ
||1.0m telescope (1m0a) in dome (domc) at LSC
||1.0m telescope (1m0a) in dome (doma) at LSC
||1.0m telescope (1m0a) in dome (doma) at ELP
||1.0m telescope (1m0a) in dome (domb) at LSC
||1.0m telescope (1m0a) in dome (doma) at CPT
||1.0m telescope (1m0a) in dome (doma) at COJ
||1.0m telescope (1m0a) in dome (domc) at CPT
||1.0m telescope (1m0a) in dome (domb) at CPT
||2.0m telescope (2m0a) in 2m enclosure (clma) at OGG
||2.0m telescope (2m0a) in 2m enclosure (clma) at COJ
||Andor Newton 540P camera with e2v CCD42-10, 2048 X 512, 13.5 um pix
||Fairchild CCD486 BI, 4096 X 4097, 15 um pix
||Fairchild CCD486 BI, 4096 X 4097, 15 um pix
||3072 X 2048, 9 um pix
||4096 X 4096, 9 um pix (no longer in service)
YYYYMMDD: local date at start of night
Frame number: increases sequentially, per night
||Type of Observation
||Spectrograph wavelength calibration
||Bias level calibration
||Dark level calibration
||Imager Science exposure
||Experimental Frame - Engineering
||Spectrograph flatfield calibration
||Imager flat calibration taken on twilight sky
||Spectrograph Science exposure
||Photometric Standard field exposure
||Quicklook data, processed with ORAC pipeline
||Quicklook data, processed with BANZAI pipeline
||Reduced data, processed with ORAC pipeline
||Reduced data, processed with BANZAI pipeline