Automating Reduction of Multifiber Spectra
from the MMT's Hectospec and Hectochelle
Douglas J. Mink, William F. Wyatt, Nelson Caldwell, Maureen A. Conroy,
Gabor Furesz, Susan P. Tokarz
Smithsonian Astrophysical Observatory, Cambridge, MA 02138
Hectospec, the MMT's low resolution, 300 fiber spectrograph, posed some
interesting and unique problems for reduction software development. To
address these issues as well as the usual ones encountered in a multifiber
spectrograph, we built a semi-automatic reduction pipeline, SPECROAD, using
host IRAF CL scripting. The pipeline is based on existing IRAF tasks,
modified when necessary, with some locally-developed procedures added.
For quality control purposes, some human interaction was kept in the
process. The pipeline takes raw data and produces one-dimensional,
wavelength calibrated, sky subtracted, and velocity correlated spectra
that are automatically distributed to PIs and added to a database. We
are currently developing a pipeline for the 240-fiber Hectochelle
high-dispersion spectrograph and we will discuss the additional issues
which have complicated its data reduction.
Hectospec and Hectochelle are multi-fiber spectrographs with robot-controlled
fiber placement. They are mounted on the MMT 6.5-meter telescope and used in
its f/5 mode, with a field of view one degree in diameter. Hectospec
(Fabricant et al. 2005) is low-resolution, with 300 fibers covering 3700-9200
angstroms at 1.2 angstrom/pixel using its 270 line/millimeter grating.
Hectochelle (Szentgyorgyi et al. 1998) is high-resolution, with 240 fibers
at 0.03 angstrom/pixel. Both have dual-chip/dual amplifier detectors
giving four readouts per exposure. An earlier paper (Mink et al 2004)
described the entire pre-observation to archive process; here we will
describe how raw data from the telescope is put into a scientifically
IRAF in a Modern Pipeline
We used the IRAF CL scripting language as the basis for this pipeline
because of our experience using it for the FAST Spectrograph and other
data reduction problems over the years. IRAF's pre-existing tasks for
extracting spectra from 2-dimensional images (Valdes 1991) were easily
modified to do what we needed. By remaining in IRAF, our own existing
IRAF SVDFIT package (Mink and Kurtz 2001) for background removal and
XCSAO package (Kurtz and Mink 1998) for redshift cross-correlation could
be reused. IRAF's cross-platform compatibilities have meant that processing
could be moved between Solaris and Linux fairly easily.
The SPECROAD master script first sorts the raw data files. Calibration
files, including bias and dark frames, comparison lamp exposures, and
sky and dome flat field images are processed using IRAF mscred.ccdproc,
removing overscan, dark counts if necessary, and trimming. After merging
the four amplifiers of each multiextension file into single images, each
set of dome, sky flat and comparison files is combined into a single image,
removing cosmic rays at the same time. Bias and dark images are examined
but usually not used. Image overscan regions provide a better
measure of their bias level, and dark current is very low and stable.
Each task adds information to the image header keeping an image from
being processed twice and allowing processing to be resumed at any point.
The spectrograph is so stable that only one set of contiguous calibration
lamp exposures are taken per night. Small adjustments in wavelength
calibration are made for each object by calculating shifts in the night
sky lines and applying that shift when the extracted spectra are linearized.
Comparison wavelength solutions from previous runs can be used as reference
spectra to get an automatic wavelength solution for comparison spectra on
a given night.
SPECROADCAL creates a flat field using the IRAF apextract.apflatten task
to flatten the combined calibration lamp image, and 300
flat field and calibration spectra are extracted from the images. From
this point on all operations are on 300 separate spectra, though they
reside in a single IRAF multispec file. Dispersion functions are fit to
all of the fibers, usually automatically based on earlier fits. Linear
spectra of a flat field pointing, usually of the twilight sky, but
sometimes of a reflective panel on the dome, are linearized using the
dispersion functions fit above. Throughput correction spectra are made
by ratioing the spectrum from each fiber to the mean of several fibers
so that the color and response of fibers can be compared.
SPECROADOBJ processes object spectrum images in the same way that calibration
images are processed through the wavelength linearization step, except that
multiple exposures of the same pointing are not combined until after they
have been linearized in wavelength. The combined spectra are then
throughput corrected, sky absorption at 6800 and 7600 Angstroms is divided
out, and extra flux at the red end of the spectrum due to an LED source
in the fiber positioner in the instrument is removed. Spectra
are then split into separate files, and the SKYPROC program removes the
sky background from each object spectrum individually.
Hectospec gets good response beyond 9200 Angstroms, but to use information
from object spectra redward of 7000 Angstroms, OH emission from the night
sky needs to be removed. A two-pronged approach has been used in the
SKYPROC script. First, we select the closest object-free sky spectra on
the same CCD chip as each object, and combine them. The point spread
functions differ enough between the two detectors that the skylines are
inaccurately removed if spectra from the two detectors are mixed. The
results are then fit to the object+sky spectrum and removed by treating
the sum as a single eigenvector and using our SVDFIT package (Mink and Kurtz
2001) which was originally developed to fit and remove multiple components
of a sky spectrum. This method has proven to be better at removing the
majority of the troublesome emission lines than subtraction scaled by
matching either single or multiple night sky spectral lines. Figure 1
shows a typical galaxy spectrum before and after sky removal. Note that
the bright mercury and oxygen lines are first removed entirely.
The pipeline suite of scripts uses a simple, light-weight method to manage
multiprocessing. An array stores the process ID of each job when forked, up
to a user-settable number. When the maximum number of jobs to be run in
parallel is reached, the scripts wait on the first job, check its exit status
and copy its output to standard output and error. If another job is to be
run, the cycle is repeated until done.
A complication is that IRAF processes require write access to their parameter
files, so they must be copied to one temporary directory per job run in
parallel. The uparm variable is pointed to the appropriate directory just
before the job is forked.
Hectospec and Hectochelle pipelines do enough I/O that two or three jobs
can be run for each processor in a system if the I/O is not a bottleneck.
The single-threaded parts of the pipeline are observed to limit the
real-time gain to a factor of about 2.5 with our existing CPUs and Gigabit
Redshifts and Quality Control
After the sky is removed, each spectrum is cross-correlated against
a set of templates using our IRAF rvsao.xcsao task. The RVSAO package includes
a quality checking task, rvsao.qplot which labels emission and absorption
lines on a graphical display, allowing the resulting radial velocities to
be rapidly checked interactively. So far, we have been able to keep up
with the data stream and check every spectrum we reduce.
Hectochelle data is processed in a similar manner, using the appropriate
parameters to define the selected echelle order and chip binning.
Unlike Hectospec, each pointing (multiple exposures on a field with the
same fiber positions) has its own associated calibration lamp and dome
flatfield spectra. So far we have only automated a few binning-filter
combinations; calibrating the remaining configurations will keep us
occupied for a while.
Now that the Hectospec pipeline for the low resolution grating is stable,
upgrades are being added to process spectra taken with the 600 line/mm
grating which gives 0.56 Angstroms per pixel. Several grating tilt
positions are allowed, providing spectra centered on different wavelengths.
Since the tilt is software controlled, multiple tilt positions can be used
per night but each position requires its own set of calibration spectra.
Fabricant, D., Fata, R., Roll, J., Hertz, E., Caldwell, N.,
Gauron, T., Geary, J., McLeod, B., Szentgyorgyi, A., Zajac, J., Kurtz, M.,
Barberis, J., Bergner, H., Brown, W., Conroy, M., Eng, R., Geller, M.,
Goddard, R., Honsa, M., Mueller, M., Mink, D., Ordway, M., Tokarz, S.,
Woods, D., Wyatt, W., Epps, H., Dell'Antonio, I. 2005, PASP, 117, 1411
Kurtz, Michael J. and Mink, Douglas J., 1998, PASP, 110, 934
Mink, D. J., Wyatt, W. F., Roll, J. B., Tokarz, S. P.,
Conroy, M. A., Caldwell, N., Kurtz, M. J., Geller, M. J. 2005, ADASS XIV, 228
Mink, Douglas J. and Kurtz, Michael J. 2001, ADASS X, 491
Szentgyorgyi, Andrew H., Cheimets, Peter, Eng, Roger,
Fabricant, Daniel G., Geary, John C., Hartmann, Lee, Pieri, Mario R.,
Roll, John B. 1998, SPIE, 3355, 242
Valdes, F. 1991, ADASS I, 417