NRES pipeline

Data Flow

Raw data are transferred from the spectrographs to LCO headquarters in near real time. Each new data file arrival triggers the pipeline. New calibration data are copied to local disk storage and relevant metadata are saved to a searchable database for later use. New science spectra are processed immediately, using the best-available calibration data, based on a suitable database search. Some intermediate data products are saved to local disk, and their metadata also go into the database. At the end of processing, the pipeline bundles the raw data, selected extraction-level data products, and summary data products into a compressed tar file, which is sent to the LCO Archive.

Separate processes run on a daily basis to construct the various master-calibration files and other derived calibration data. The inputs for these processes are extracted from the database, and their products are sent to the Archive in case they are needed by users. There is a facility to reprocess "old" data through the pipeline, if it is necessary.

To better understand the pipeline processing, it may be beneficial to read the webpage describing the NRES instrument architecture, including the calibration system.

Data Types

The NRES spectrograph and its associated calibration system and site software produce four kinds of calibration images and one kind of science image. The calibration images are (1) BIAS images, (2) DARK images, (3) images, called LAMPFLATs, in which two input fibers (the reference fiber and one star fiber) are illuminated by the tungsten-halogen (TH) lamp, and (4) images, called DOUBLEs, in which two input fibers (the reference fiber and one star fiber) are illuminated by a ThAr lamp. In the science images, the reference fiber is illuminated by a ThAr lamp (either slave or master) and one of the star fibers by starlight from a telescope. The science data are more complicated than the calibration images because they contain additional data from telescopes, autoguiders, and the exposure meter.

Calibration data files are normally acquired in groups of the same type, during daylight hours. These files are processed (external to the pipeline) on a daily basis and averaged to create "Supercalibration" files: SuperBIASes, SuperDARKs, and SuperFLATs. The SuperFLATs, which include valid flats for all fibers, are used to construct files (called TRACE files) that describe the positions and cross-dispersion shapes of the spectrum orders on the CCD. Multiple DOUBLE files are used to construct files (called TRIPLEs) that describe the fiber-to-fiber offsets between ThAr spectra along the dispersion axes. The set of calibration files is completed by standard stellar spectra (called ZERO files) that are compared to observed spectra to estimate radial velocities of stars. The ZERO spectra are made from averages of spectra of an observed star, which is chosen because it's of similar spectral type to the target star.

Extraction and Wavelength Calibration

For each science image, the pipeline selects the "best" calibration files from a database search, according to simple rules. The calibration steps that follow are:

  • Bias and dark subtraction;
  • Determining the positions of the orders, using the TRACE data, and then fitting and subtracting a model of the between-order background light;
  • Performing an iterative preliminary extraction and order cross-dispersion centroid computation, using the TRACE data to define "extraction boxes" that are nominally centered on the order positions. If the initial computed centroid displacements are too large, a parametric adjustment of the TRACE data is applied to move the boxes.
  • Computing an optimally-extracted spectrum using cross-dispersion profiles from the TRACE file and a noise model.
  • Examining the residuals around the extracted spectrum to identify evidence of radiation events. Where evidence is found, the fitting weights for nearby data points are set to zero, and the fit is re-computed.
  • Saving three versions of the resulting extracted spectrum to local disk. The versions are:
    • a raw extracted ("EXTR") spectrum. Each order looks more or less like the blaze function of the spectrograph.
    • a raw extracted spectrum with a constant multiple of the SuperFLAT spectrum subtracted from it. This ("BLAZ") spectrum should have a near-zero mean value, and goes to zero at the edges of the blaze function. It has desirable noise properties for use in radial velocity estimation.
    • a ("SPEC") spectrum that is the ratio of raw extracted spectrum and the SuperFLAT, i.e. something like the true stellar spectrum with the instrumental response removed. It is noisy at the edges, and prone to systematics arising from differences between star and flat illumination of the spectrograph optics.
  • Adjusting parameters in a model of vacuum wavelength vs (x-coordinate, order index, fiber index) to give an optimum match between the positions of emission lines observed in the reference spectrum and those implied by the wavelength model and the ThAr line catalog by Redman (2013). The pipeline saves the entire wavelength solution, along with all of the model parameters, to local disk.

Radial Velocity Estimation

The procedure for determining radial velocities continues to be improved as the pipeline evolves. It is possible to run the RV estimation code independently, after the rest of the pipeline has executed.

Stellar radial velocities are estimated by comparing the BLAZ extracted spectrum with a ZERO file (a standard stellar spectrum). To insure that a spectrum from a particular science target is always compared with the same ZERO spectrum, each target star is linked with a particular ZERO file in the database. First, the pipeline determines an approximate redshift by cross-correlating the BLAZ and ZERO spectra, but only for the echelle order containing the Mg b lines (roughly 516 nm). Based on this preliminary estimate, the pipeline then interpolates the entire ZERO spectrum to the provisional redshifted wavelength scale and breaks each order into a number of "blocks", i.e. contiguous wavelength segments. The pipeline then performs a fit to estimate the residual redshift of each block and formal errors. Last, the pipeline constructs several estimates of the "mean" redshift, taking differently-weighted averages or medians of the individual block redshifts. Outputs of the radial velocity analysis are written to a FITS extension file with an empty main data segment. The first extension table contains the cross-correlation function and various cross-correlation-related statistics. The second extension table contains the computed residual redshifts per order and block, and useful statistics related to them.

Data Products

At the end of the reduction procedure, the pipeline bundles various output data into a gzipped tar file, and writes them to the LCO science archive. An example tar file name is:


The naming convention for the tar files is {Site ID}{NRES ID}-{Camera ID}-{DAY-OBS}-{Image number}-e91.tar.gz, where the "e91" indicates that processing is complete. An example of the data products contained within the tar files is:

README                                             (A list of the files in the tarball)
arc_lsc_nres01_fl09_20170716.fits.fz (The ThAr arc spectrum (i.e. TRIPLE file) used in the reduction)
flat_lsc_nres01_fl09_20170716.fits.fz (The flat field used in the reduction) lscnrs01-fl09-20170717-0047-e91-blaze.fits.fz (The extracted spectrum with the blaze function subtracted) lscnrs01-fl09-20170717-0047-e91-noflat.fits.fz (The extracted spectrum with no flat field applied) lscnrs01-fl09-20170717-0047-e91-rv.fits.fz (The radial velocity solution) lscnrs01-fl09-20170717-0047-e91-wave.fits.fz (The wavelengths for each pixel in the extracted spectra) lscnrs01-fl09-20170717-0047-e91.fits.fz (The reduced spectrum) lscnrs01-fl09-20170717-0047-e91.pdf (A set of quality control plots. See below.) trace_lsc_nres01_fl09_20170716.fits.fz (The extraction region used in the reduction (i.e. TRACE file))

For every science spectrum that's processed, the pipeline creates a set of diagnostic plots. Some aim to show aspects of the target star spectrum; others contain diagnostics of the accuracy of the TRACE file used for extracting 1-dimensional spectra, and of the wavelength solution. Consult the NRES diagnostic plots page for a thorough description.