TRES Reduction Tasks
Jessica Mink, 2013-Aug-23

[Observation] [Quicklook)] [IDL Pipeline)] [IRAF Pipeline (Software) [Archive]

Checking Data

tres.trsdate yyyy.mmdd or now
This task is run before any other TRES tasks because it sets the working directory based on the date. First it transfers to quickdiryyyy.dddd, the processing directory. Then it sets that processing directory and the directory from which to read raw data for all of the top and second level processing taslks (trsproc, trslist, trsfiles, trsgroup, qtres, ctres, ttres, and ftres), so they all can be used with sequence numbers instead of full file names. If the first argument is "now", today's observing date (local date at noon of the observing day) is used.
tres.trsgroup runs tres.trslist and sorts the data files for the night specified by trsdate into files named by object, configuration, and exposure time: [obj][s m l][b if binned][fiber(s)]_[exposure in seconds].list. The string [s m l][b if binned][fiber(s)] is used in many places and referred to as apfib.
tres.trslist 1-300 fibkey=xxx fibsize=x.x apfib=x object=xxxx
This task lists all of the TRES files meeting specific characteristics. By default it looks in the current working directory, but if trsdate has been run, it looks for files in the raw directory, datedir in a specified instdir, such as /tres on Mt. Hopkins. If outlist is specified, names of files meeting the selection criteria are written to the specified file, one per line, which can be used for many of the tasks below. If verbose=yes, that file contains a tab-separated table of observation parameters for each file.

Quick Look

There are two programs set up to run an approximate reduction of TRES data to see how the spectra look.

tres.qtres reduces a raw spectrum image to one or several multi-order echelle spectra. It uses tres.tproc to all of the process, then splot to plot the resulting date
tres.ctres reduces a list of raw spectrum images taken of the same object with the same fiber size and binning to dispersion-corrected multi-order echelle spectra with cosmic rays removed by a comparison method. Dispersion shifts are computed using the shift from the closest ThAr spectra or interpolating between the closest ThAr spectra before and after each exposure. The final products are one echelle spectrum file per fiber with a non-linear dispersion function in each header.
tres.ttres reduces a list of raw ThAr spectrum images taken with the same fiber size and binning to dispersion-corrected multi-order echelle spectra with cosmic rays removed by a comparison method. The dispersion function is computed by cross-correlating to a reference spectrum in pixel space and adding the resulting shift to the dispersion function of the reference spectrum. The final products are one echelle spectrum file per fiber with a non-linear dispersion function in each header. tres.tharplot can be used to examine to wavelength fit because it labels the positions of the brightest ThAr emission lines.
tres.dtres reduces a list of dark images taken with the same binning by running CCDPROC, merging amplifiers, and removing cosmic rays with a comparison method. The final products are unextracted 2-D images.

Pipeline

There are two TRES pipelines, one in IRAF developed by Jessica Mink and one in IDL developed by Lars Buchhave.

IDL Pipeline

Lars' pipeline is continuously being improved, and a significant number of shell scripts and a few C programs are used to handle the significan amount of data manipulation which must occur both before and after the pipeline turns 2-D images into searchable 51-order spectra with wavelength solutions. This page describes the entire process from the transfer of raw data from the 1.5-meter telescope on Mt. Hopkins in Arizona to a searchable database of results, spectra, and graphs.

IRAF Pipeline

The TRES pipeline adds automatic sorting of files by fiber configuration and exposure time (which matters for our cosmic ray removal algorithm).

tres.trsproc calls trsgroup, reads the list files of list files it produces, and processes them using the appropriate "quick look" task from the list above. trsgroup is only called to create lists of flat, bias, dark, ThAr, or object files if a list for that type of file does not already exist. The type or types of spectra to reduce is selected by a command line argument which is by default a null string:
b Bias images (BIAS is object name) d Dark images (DARK is object name) f Flat field spectra (FLAT is object name)
c Thorium Argon spectra (COMP is object name) o Object spectra (all of the rest)

Bias and Dark Files

tres.trsproc
runs tres.ftres to process files through cosmic ray removal:
mscred.ccdproc trim+ overscan+ fixpix+ to bias-subtract, trim, and remove bad pixels from both amplifiers of all files.
tres.gaincorr corrects for gain variation between amplifiers. The raw files are moved to the Raw/ subdirectory, and the output files have the same names as the originals.
tres.tampmerge is run to combine the two amps to single image FITS files.
tres.tcosmic is run to remove cosmic rays.
Check bias files by running tres.txstat on them.
mscred.combine all of the bias files except those for which any amplifier stands out to an output Zero.fits file. implot it to check for missed bad columns. We are not subtracting bias images, and are relying instead on overscan removal to take care of any bias signal without adding noise from individual pixels.
tres.txstat on dark files to check for light leaks.
mscred.combine them to an output Dark.fits file.
Check values using plot.implot.
mscred.ccdproc dark+ on all data files if there is significant dark flux.

Flat Field Files

tres.ftres reduces a list of raw flat spectrum images taken with the same fiber size and binning by running CCDPROC, merging amplifiers, and removing cosmic rays with a comparison method. The final products are unextracted 2-D images, images for pixel-to-pixel flattening and masking out spectra for scattered light removal, database/ap files for spectrum extraction, and extracted spectra for blaze function removal. Fiber to fiber normalization spectra can be created from the extracted spectra using tres.tpmake.

mscred.ccdproc trim+ overscan+ fixpix+ to bias-subtract, trim, and remove bad pixels from all amplifiers of all files.
tres.gaincorr corrects for gain variation between amplifiers. The raw files are moved to the Raw/ subdirectory, and the output files have the same names as the originals.
tres.tampmerge flat.*.fits combines the two amps to single image FITS files.
tres.tcosmic @flatxx.fits compares images with the same characteristics and removes cosmic rays from each exposure. Temporary median and limit files are created and my be deleted. Nelson Caldwell developed the algorithm which compares two files at a time, using statistics from the median file to figure out when to reject high pixels.

Since TRES is so stable, we wavelength calibrate by shifts in the dispersion direction from a reference spectrum and flatten and extract based on the same flat field which was used for that reference spectrum, all of which is in the tresdata$ directory. Here is the process used to produce new flattening images and extraction masks:

New aperture mask, flattening and scattered light masking files are made by running tres.ftres with makeref=yes, or outside of the pipeline by running running tres.tmakeref on the file resulting from running tres.trsmed to combine multiple flat field exposures of the same type into a single image file.
tres.tmakeref flatxx12.fits or tres.tmakeref flatxx3.fits Example run aptrace on summed dome flat field file using a specified standard template, and then apflatten to create a normalization file which is used by tres.tflat to remove IR fringing and other pixel to pixel variation, more or less. The new apflattening file is renamed flat[lms][b][12|3].flat.fits and is then used by tres.tflat on the median dome flat field file, after which aptrace is run again to create extraction templates for all of the apertures in the file. If the input file includes fibers 1 and 2, the database/apflat* file is then split to create database/apflat*1 and database/apflat*2.
tres.textract flatxx.fits extracts spectra for each fiber present in ech image. tres.tarith flatxx_f2.ec.fits / flatxx_f1.ec.fits flatxxf2df1.ec.fits makes a normalization file which can be used to scale the sky fiber to the object fiber for sky removal.

Calibration Lamp (ThAr) Files Use tres.trsgroup to group all of the COMP files by apfib and exposure, so cosmic rays can be removed. tres.ttres @compxx.list flatten+ cosmic+ extract+ disperse+ calls tres.tproc on each image to trim, remove overscan, merge, and flatten all COMP image files, tres.tcosmic on the list of images to remove particle hits by comparing similar images region by region, tres.textract1 on each image to extract the spectra based on the FLAT template set above, and tres.tcal1 on each extracted spectrum to add the correct dispersion function by fitting or cross-correlating as below: Run ttres on each group of COMP files. For each configuration, there should be a reference for the night. If three or more files have been taken sequentially as a night reference, run tres.ttres seq1-seqn compstd="sum" compid+ on the appropriate sequence numbers. After shifting by an image mean or median in pixel space along the dispersion direction, the ThAr emission line fit will be run using echelle.ecreidentify in interactive mode. If the reference spectrum's dispersion is close to the summed spectrum's dispersion, you will just have to cross-correlate using xm refit using f, type q twice to quit the fit and the program, then y or carriage return to save the fit on the way out. If two fibers exist in these files, the program will run the fit on each in order and you will have to type the above commands twice. Otherwise, you should use the first file of the list with the longest exposures to get the best signal-to-noise thusly: tres.ttres COMPfffxnnn.list compstd="first" compid+ on the appropriate list file. The ThAr emission line fit will be run in interactive mode, but if the reference spectrum's dispersion is close to the summed spectrum's dispersion, you will just have to cross-correlate using xm refit using f, type q twice to quit the fit and the program, then y or carriage return to save the fit on the way out. If two fibers exist in these files, the program will run the fit on each in order and you will have to type the above commands twice. Run tres.ttres COMPfffxnnn.list compstd="" compid- on each of the lists for which you have set up a reference spectrum interactively. Each spectrum will be cross-correlated against the night's reference for that configuration, and the resulting shift will be applied to the reference database/ec* dispersion function, which will be refit and added to the spectrum header for each fiber. The shifts will be tabulation with the times of exposures in files named compfff.shift where fff is the configuration (APFIB) code. Shifts for object spectra will be interpolated from the shifts in these files. In more detail, for each image file: mscred.ccdproc trim+ overscan+ fixpix+ to bias-subtract, trim, and remove bad pixels from all amplifiers of all files. tres.gaincorr corrects for gain variation between amplifiers. tres.tampmerge is run to merge the two amplifiers of each comparison file to single image FITS files. The raw files are optionally moved to the Raw/ subdirectory, and the output files have the same names as the originals. tres.tflat @compxx.list removes fringing and pixel-to-pixel variations from comparison lamp files. The original merged, unnormalized file is saved in the Unflat/ directory. For all files with same observing parameters: tres.tcosmic @compxx.fits compares images with the same characteristics and removes cosmic rays from each exposure. Temporary median and limit files are created and my be deleted. Nelson Caldwell developed the algorithm which compares two files at a time, using statistics from the median file to figure out when to reject high pixels. For each image file: tres.textract1 compfile extracts the calibration spectra, using flatxx.ec.fits as apref. [Turn off *all* processing except extraction to produce comp.ec.fits.] tres.tcal1 compfile runs rvsao.pxcsao against an appropriate reference comparison lamp file to get a single dispersion-direction pixel shift which is applied to the wavelength solution with which the file is then dispersed. If interactive=yes, the dispersion function can be refined using ecidentify. The cursor "x" command cross-correlates to get a better match between features. "f" fits to the new matches and brings up a display of residuals in which outliers can be deleted with "d" and the dispersion refit with "f" until the graph looks satisfactory, with residuals within 0.015 or so. "q" exits the residuals and another "q" followed by a "y" response exits and writes the revised id file to the database directory. It is then retrieved to add the dispersion function to the spectrum header. Object Files tres.ctres objects.fits or tres.trsproc o to process all objects. For each image file: First mscred.ccdproc trim+ overscan+ fixpix+ bias-subtracts, trims, and removes bad pixels from all amplifiers of all files. tres.gaincorr corrects for gain variation between amplifiers. tres.tampmerge is run to merge the two amplifiers of each object file to single image FITS files. The raw files are moved to the Raw/ subdirectory, and the output files have the same names as the originals. tres.tflat divides by the normalization image made by tmakeflat, removing pixel to pixel variation and fringing. The original merged, unnormalized file is saved in the Unflat/ directory. For all files with same observing parameters: tres.tcosmic @objectxx.list compares images from the same object, FIBSIZE, binning, and FIBKEY and removes cosmic rays from each exposure. Temporary median and limit files are created and may be deleted. Nelson Caldwell developed the algorith which compares two files at a time, using statistics from the median file to figure out when to reject high pixels. If ctres.sumspec=yes, tres.trssum sums all exposures of a single pointing into a single set of spectra, assigning the sum a new observation sequence number and name. This file is processed just like its components through the following steps. For each image file: tres.textract1 object.fits extracts the object spectra, using the appropriate dome flat, flat[size][binning][fiber].flat.fits as the aperture reference. tres.tdisp1 object_f*.ec.fits adds dispersion functions for each order to each spectrum refspec file.ec.fits references="comp.ec", dispcor file.ec.fits filed.ec.fits linearize- The original .ec file is moved to Nodisp/. rvsao.xcsao *.fits can be used to find the radial velocities of all of the object spectra, but there are not any good templates yet. [Definitions] [Preparation] [Observation] [Reduction (Software)] [Distribution] [Archiving] [Instruments] [TRES]

`b`	Bias images (BIAS is object name)	`d`	Dark images (DARK is object name)	`f`	Flat field spectra (FLAT is object name)
`c`	Thorium Argon spectra (COMP is object name)	`o`	Object spectra (all of the rest)

TRES Reduction Tasks Jessica Mink, 2013-Aug-23