Raw data are transferred from the spectrographs to LCO headquarters in near real time. Each new data file arrival triggers the pipeline. New calibration data are copied to local disk storage and relevant metadata are saved to a searchable database for later use. New science spectra are processed immediately, using the best-available calibration data, based on a suitable database search. Some intermediate data products are saved to local disk, and their metadata also go into the database. At the end of processing, the pipeline bundles the raw data, selected extraction-level data products, and summary data products into a compressed tar file, which is sent to the LCO Archive.
Separate processes run on a daily basis to construct the various master-calibration files and other derived calibration data. The inputs for these processes are extracted from the database, and their products are sent to the Archive in case they are needed by users. There is a facility to reprocess "old" data through the pipeline, if it is necessary.
To better understand the pipeline processing, it may be beneficial to read the webpage describing the NRES instrument architecture, including the calibration system.
The NRES spectrograph and its associated calibration system and site software produce four kinds of calibration images and one kind of science image. The calibration images are (1) BIAS images, (2) DARK images, (3) images, called LAMPFLATs, in which two input fibers (the reference fiber and one star fiber) are illuminated by the tungsten-halogen (TH) lamp, and (4) images, called DOUBLEs, in which two input fibers (the reference fiber and one star fiber) are illuminated by a ThAr lamp. In the science images, the reference fiber is illuminated by a ThAr lamp (either slave or master) and one of the star fibers by starlight from a telescope. The science data are more complicated than the calibration images because they contain additional data from telescopes, autoguiders, and the exposure meter.
Calibration data files are normally acquired in groups of the same type, during daylight hours. These files are processed (external to the pipeline) on a daily basis and averaged to create "Supercalibration" files: SuperBIASes, SuperDARKs, and SuperFLATs. The SuperFLATs, which include valid flats for all fibers, are used to construct files (called TRACE files) that describe the positions and cross-dispersion shapes of the spectrum orders on the CCD. Multiple DOUBLE files are used to construct files (called TRIPLEs) that describe the fiber-to-fiber offsets between ThAr spectra along the dispersion axes. The set of calibration files is completed by standard stellar spectra (called ZERO files) that are compared to observed spectra to estimate radial velocities of stars. The ZERO spectra are made from averages of spectra of an observed star, which is chosen because it's of similar spectral type to the target star.
For each science image, the pipeline selects the "best" calibration files from a database search, according to simple rules. The calibration steps that follow are:
The procedure for determining radial velocities continues to be improved as the pipeline evolves. It is possible to run the RV estimation code independently, after the rest of the pipeline has executed.
Stellar radial velocities are estimated by comparing the BLAZ extracted spectrum with a ZERO file (a standard stellar spectrum). To insure that a spectrum from a particular science target is always compared with the same ZERO spectrum, each target star is linked with a particular ZERO file in the database. First, the pipeline determines an approximate redshift by cross-correlating the BLAZ and ZERO spectra, but only for the echelle order containing the Mg b lines (roughly 516 nm). Based on this preliminary estimate, the pipeline then interpolates the entire ZERO spectrum to the provisional redshifted wavelength scale and breaks each order into a number of "blocks", i.e. contiguous wavelength segments. The pipeline then performs a fit to estimate the residual redshift of each block and formal errors. Last, the pipeline constructs several estimates of the "mean" redshift, taking differently-weighted averages or medians of the individual block redshifts. Outputs of the radial velocity analysis are written to a FITS extension file with an empty main data segment. The first extension table contains the cross-correlation function and various cross-correlation-related statistics. The second extension table contains the computed residual redshifts per order and block, and useful statistics related to them.
Stellar spectrum output data files produced by the NRES reduction pipeline will soon have a new format. We hope this will be both more accurate, more complete, and easier to use. Data from the archive arrive in a form that depends on how you acquire it. Please refer to the archive documentation to understand differences between downloading your data via a web browser, or with a wget script, or one file at a time with a Python script, or perhaps other methods. In any case the underlying data items are gzipped tar files, each corresponding to an individual spectrum exposure. These have names like lscnrs01-fl09-20180307-0036-e91.tar.gz lscnrs01-fl09-20180307-0037-e91.tar.gz where lsc = the observing site: lsc=CTIO, elp=McDonald, cpt=SAAO, tlv=Wise nrs0x = which spectrograph; NRES-0 to NRES-4 20180307 = The OBS-DATE date on which the observation was started (This date rolls over at a different UT time for each site, such that OBS-DATE is fixed during each night of observation.) 0036 = The nightly sequence number e91 = A code for the reduction level. e91 = reduced with Banzai pipeline Unpacking (tar -xzf file.tar.gz) each of these tar files yields a directory named after the observation, eg lscnrs01-fl09-20180307-0031-e91. Each such directory will henceforth contain only 2 files: the fpacked main data file (eg lscnrs01-fl09-20180307-0031-e91.fits.fz) and the diagnostic plots pdf file (eg lscnrs01-fl09-20180307-0031-e91.pdf) Running funpack on the main data file yields the main data file (eg lscnrs01-fl09-20180307-0031-e91.fits) The main data file is a multi-extension fits file containing 10 extensions. The first of these is essentially only a header; the last is a binary table extension. All the rest are IMAGE extensions. Their contents are as follows: Extn #,name Data contents Header keywords 0: -- None Everything relating to the observation as a whole: instrument config, dates, scheduling info, names of calib frames used, weather info, instrument setup info, spectrograph environment readings. Data unique to this star spectrum: Telescope RA, Dec, Object name, total fluxes of star and ThAr, telescope lat, long. NORD, per-order scaling factor AMPFLxxx applied to the flat field calibration before sub- tracting from the raw extracted spectrum. Wavelength solution parameters (SINALP, FL, Y0, Z0), polynomial correction coeffs C0 to C14, between-fiber correctopm coeffs FIBC0 to FIBC9, name of ThAr line catalog file. Cross-correl peak width (km/s) and height, RCC = redshift of peak relative to template RVCC = redshift (km/s) rel to barycenter Robust average, median, formal errors for per-block redshifts. 1: SPECRAW Raw extracted star Nothing of interest. spectrum (NX,NORD) (float) 2: SPECFLAT Flat-fielded Nothing of interest. extracted star spectrum (NX,NORD) (float). This spectrum is the most nearly free of instrumental signatures, but it is noisy near its boundaries. 3: SPECBLAZE Blaze-subtracted Nothing of interest. extracted star spectrum (NX,NORD) (float). This spectrum has simple noise properties, and is the one used in radial velocity estimation. 4: THARRAW Raw extracted ThAr Nothing of interest. spectrum (NX,NORD) from calibration fiber (float) 5: THARFLAT Flat-fielded Nothing of interest. extracted ThAr spectrum (NX,NORD) (float) 6: WAVESPEC Wavelength solu- Nothing of interest. tion [nm] vs (NX,NORD) for the starlight- carrying fiber (double) 7: WAVETHAR Wavelength solu- Nothing of interest. tion [nm] vs (NX,NORD) for the standard ThAr fiber (fiber 1) (double) 8: SPECXCOR Correlation fn lag (km/s) vs pixel index, in fits-standard with template spec CRVAL1, CDELT1, CTYPE1, CRPIX1 format. vs lag (km/s) (float) 9: RVBLOCKFIT Redshift and descriptive info about table columns. related parameters per (order,block) from least-squares fit, relative to RCC redshift (double, mostly) Data columns are: ZBLOCK Array of (NORDER,NBLOCK) values of measured redshift relative to cross-correlation redshift. Units are dimensionless (velocity/c) ERRZBLOCK Formal error on ZBLOCK SCALE Estimated scale factor connecting intensity in current spectrum to that in template spectrum, by order and block. (dimensionless) ERRSCALE Formal error on SCALE LX1COEF Estimated zero-point shift between intensity in current spectrum and in template spectrum, by order and block. Units are ADU. ERRLX1 Formal error on LX1COEF PLDP Estimated Photon-Limited Doppler Precision by order and block (km/s). BLKINDX Block index. Range [0 - NBLOCK-1] ORDINDX Order index. Range [0 - NORD-1]
At the end of the reduction procedure, the pipeline bundles various output data into a gzipped tar file, and writes them to the LCO science archive. An example tar file name is:
lscnrs01-fl09-20170830-0044-e91.tar.gz
The naming convention for the tar files is {Site ID}{NRES ID}-{Camera ID}-{DAY-OBS}-{Image number}-e91.tar.gz, where the "e91" indicates that processing is complete. An example of the data products contained within the tar files is:
README (A list of the files in the tarball) arc_lsc_nres01_fl09_20170716.fits.fz (The ThAr arc spectrum (i.e. TRIPLE file) used in the reduction) flat_lsc_nres01_fl09_20170716.fits.fz (The flat field used in the reduction) lscnrs01-fl09-20170717-0047-e91-blaze.fits.fz (The extracted spectrum with the blaze function subtracted) lscnrs01-fl09-20170717-0047-e91-noflat.fits.fz (The extracted spectrum with no flat field applied) lscnrs01-fl09-20170717-0047-e91-rv.fits.fz (The radial velocity solution) lscnrs01-fl09-20170717-0047-e91-wave.fits.fz (The wavelengths for each pixel in the extracted spectra) lscnrs01-fl09-20170717-0047-e91.fits.fz (The reduced spectrum) lscnrs01-fl09-20170717-0047-e91.pdf (A set of quality control plots. See below.) trace_lsc_nres01_fl09_20170716.fits.fz (The extraction region used in the reduction (i.e. TRACE file))
For every science spectrum that's processed, the pipeline creates a set of diagnostic plots. Some aim to show aspects of the target star spectrum; others contain diagnostics of the accuracy of the TRACE file used for extracting 1-dimensional spectra, and of the wavelength solution. Consult the NRES diagnostic plots page for a thorough description.
Version 0.8 May 10th 2018
Update of data output format. Newly observed data will be processed with new pipeline. Old data will be reprocessed at a later time.