Benjamin Weiner
March 26, 2007
This guide outlines steps to process IMACS mosaic CCD imaging data, including overscan, bias, and flatfielding. These steps were designed for Maryland-Magellan Tunable Filter observations, but can be applied to generic IMACS imaging as well as MMTF data. Wavelength calibration for MMTF will be covered separately. This page looks long because it goes into detail, but the actual steps to flatfield your data should not be too complex.
IMACS CCD data for the 8 chip mosaic are written out as 8 fits files: ccd0001c1.fits, ccd0001c2.fits, .... Each chip has overscan in both x and y directions. These scripts attempt to process the data and deal with the 8 chip files properly. The scripts assume that your images have all 8 subimage files for each image and that they are 2048x4096 with at least 64 columns overscan (the defaults). There is a version of the overscan subtraction script that can be modified to work on non-standard sized images (see below).
The major issues in the data reduction are (1) bookkeeping for the 8 subimage files for each image basename; (2) dealing with the two directions of overscan properly. Following advice from Ian Thompson, these scripts subtract the median of the row overscan from each row (this compensates for temporal variations during the parallel transfer), and then do a conventional ccdproc-style overscan subtraction using a spline fit to the column overscan, which removes the gradient in the x-direction that is due to charge buildup (?), inherent to the way the IMACS CCD controllers work.
The basic idea in this data reduction is that there are a series of shell scripts, e.g. "make_overscan". Each shell script is run with some arguments, which are frequently image basenames or files with lists of basenames, where a basename is "ccd0001" to refer to ccd0001c1.fits, ccd0001c2.fits, etc. Running the shell script at the UNIX prompt writes an output file, e.g. "do_overscan.cl". This is a series of IRAF commands, and you run it from inside IRAF by typing at the IRAF prompt: "cl < do_overscan.cl".
Download a tar file of the MMTF flat/reduction scripts.
If you find these scripts useful or want to suggest changes, please drop me an email to bjw -at- as.arizona.edu. If you use these to reduce data for publication, please acknowledge their use in the text. Thanks!
1. List image headers. In IRAF, cd into each image directory and do:
# in iraf hselect ccd*c2.fits $I,naxis[12],exptime,filter,mmtf-z,title yes > Imheaders.n1(change the output filename as needed). This file should help you to identify flats, objects, etc.
2. Copy images that you want to reduce into a working directory so you don't change the originals.
3. Set script and work directories. In the shell (not in IRAF):
set reducdir = /home/bjw/data1/workdir (or wherever your data are) set scriptdir = /home/bjw/programs/mmtf/scripts (or wherever you put the MMTF scripts)
4. Make lists of "basenames." A basename is e.g. "ccd0001" to refer to ccd0001c1.fits, ccd0001c2.fits, etc. In your working directory, assuming that each file that you want to reduce actually has all 8 subimages:
ls *c2.fits | sed s/c2\.fits// > list.base1.all
Now edit the list of basenames to create sub-lists for each category. e.g.
list.base1.standard list.base1.for_proc (bias, obj, flat only, may omit std stars if desired) list.base1.bias list.base1.flat list.base1.obj
You will later need a file that matches each object basename with the appropriate flatfield basename, e.g. if ccd0081c*.fits is an R-band image and Skyflat_R_c*.fits is a combined R-band flat the file will have this line:
ccd0081 Skyflat_R_See Step 12 below where this file is used, called "list.base2.obj_flat".
5. (If desired) move each category into its own subdirectory
mkdir Biases Flats Objects Standards
sh
for i in `cat list.base1.standard` ; do mv ${i}c*.fits Standards ; done
for i in `cat list.base1.bias` ; do mv ${i}c*.fits Biases ; done
for i in `cat list.base1.flat` ; do mv ${i}c*.fits Flats ; done
for i in `cat list.base1.obj` ; do mv ${i}c*.fits Objects ; done
exit
(If needed) update the lists:
ls Standards/ccd*c2.fits | sed s/c2.fits// > list.base2.standard ls Biases/ccd*c2.fits | sed s/c2.fits// > list.base2.bias ls Flats/ccd*c2.fits | sed s/c2.fits// > list.base2.flat ls Objects/ccd*c2.fits | sed s/c2.fits// > list.base2.obj cat list.base2.bias list.base2.flat list.base2.obj > list.base2.for_proc
6. Generate the IRAF cl scripts. Here, you run a set of shell scripts: each shell script doesn't do anything to the data, but writes a series of IRAF cl commands which you then pipe into IRAF. You can and should read the do_*.cl scripts to see what they do. Replace the list.base2.* names if you named the files differently. These scripts may emit a warning like mv: cannot stat `do_overscan.cl'. This is harmless (the script tries to back up any pre-existing output files so that you don't overwrite them, and this warning happens if the output file did not previously exist).
sh $scriptdir/make_imtype list.base2.bias list.base2.flat list.base2.obj do_imtype.cl sh $scriptdir/make_overscan list.base2.for_proc do_overscan.cl sh $scriptdir/make_biascomb list.base2.bias Biases/Bias_comb_ do_biascomb.cl # make_biascomb issues a warning about running the script # before deleting the intermediate list files it makes sh $scriptdir/make_subbias list.base2.flat Biases/Bias_comb_ do_subbias1.cl sh $scriptdir/make_subbias list.base2.obj Biases/Bias_comb_ do_subbias2.cl # make_subbias issues a warning about editing CCDMEAN before # flatfielding, see below
Note: if you happen to have IMACS images that are not the standard 2048x4096 size, for example if they are binned or subrastered, you may be able to reduce them by editing the make_overscan_size script to match the sizes and running that to create do_overscan.cl. The rest of the scripts should work independent of image size.
7. Now run the scripts inside IRAF. Start iraf (cd to your iraf directory, type cl), and cd to your working directory. Then in IRAF:
# in iraf cl < do_imtype.cl # do_overscan.cl loads ccdred and sets some params but it's best to # epar ccdproc first just to make sure all the params are sensible. noao imred ccdred epar ccdproc ccdtype="", fixpix=no, oversca=yes, trim=yes, zerocor=no, darkcor=no, flatcor=no, illumcor=no, fringecor=no, readcor=no, readaxi=column cl < do_overscan.cl # this can take a while epar imcombine # to check settings: combine=average, reject=avsigclip, scale=none, zero=none cl < do_biascomb.cl cl < do_subbias1.cl cl < do_subbias2.cl
8. Check to make sure everything got done:
hselect Biases/ccd*c1.fits $I,imagetyp,trim yes hselect Flats/ccd*c1.fits $I,imagetyp,trim yes hselect Objects/ccd*c1.fits $I,imagetyp,trim yes hselect Flats/ccd*c1.fits $I,imagetyp,zerocor yes hselect Objects/ccd*c1.fits $I,imagetyp,zerocor yesor you may want to do
ccdlist Biases/ccd*c1.fits,Flats/ccd*c1.fits,Objects/ccd*c1.fits
9. Before flat-fielding, there is a subtlety. IRAF flatfields by dividing by the flat and multiplying back by a header keyword, usually CCDMEAN (= the average value of the flat), so that 1 DN in the input is about 1 DN in the output. There are two issues here:
10. Figure out a good value for CCDMEAN by doing an imstat near the center of the field (e.g. pixels [2000:2040,4040:4080] on chip2).
in shell:
awk '{print $1"c2.fits"}' list.base2.flat > tmp1
awk '{print $1"c2.fits[2000:2040,4040:4080]"}' list.base2.flat > tmp2
in IRAF:
hselect @tmp1 $I,filter,mmtf-z,exptime,title yes
imstat @tmp2
Compare pixel values between similar flats,
and don't forget to take into account
that flats may have differing exposure times. You want to set
CCDMEAN to a common value for each matching set of flats,
but have different CCDMEAN for e.g. flats in different filters.
If some flats
are twilight flats, then you will want to combine them with
imcombine scale=median and adjust CCDMEAN on the
combined flat.
11. Adjust CCDMEAN. Suppose that we have two sets of flats in the 6600 filter that need different CCDMEAN of 31500 and 18900 (5 sec and 3 sec dome flats), and another set in an R filter that need to be combined first. First, combine the R flats and set CCDMEAN:
put names of Rflats in list.base2.flat_R put "Flats/Rflat_comb_" in list.base2.Rflatcomb sh $scriptdir/make_flatcomb list.base2.flat_R Flats/Rflat_comb_ do_Rflat_comb.cl # in iraf cl < do_Rflat_comb.cl imstat Flats/Rflat_comb_c2[2000:2040,4000:4040] # to get value to set CCDMEAN to, here 18800 sh $scriptdir/make_ccdmean list.base2.Rflatcomb 18800 do_ccdmean_R.cl # in iraf cl < do_ccdmean_R.cl
Now look at the narrow band flats (here some MMTF flats in the 6600 filter), sort them into lists of ones that need a matching CCDMEAN (here we had some with 5 sec exposure and some with 3 sec, which need different CCDMEAN):
edit list.base2.flat to make list.base2.flat.6600a, list.base2.flat.6600b sh $scriptdir/make_ccdmean list.base2.flat.6600a 31500 do_ccdmean_1.cl sh $scriptdir/make_ccdmean list.base2.flat.6600b 18900 do_ccdmean_2.cl # in iraf cl < do_ccdmean_1.cl cl < do_ccdmean_2.cl # check that CCDMEAN was set correctly # in iraf hselect @list.base2.flat.sub $I,title,ccdmean yes
12. Now that CCDMEAN is set correctly to prevent any unpleasant rescaling surprises, actually do the flat fielding. Here, list.base2.obj_flat is a file with two columns. The first column is a list of object basenames to be flat-fielded and the second column is the corresponding flat basename, for example:
Objects/ccd0877 Flats/ccd0916 Objects/ccd0878 Flats/ccd0917 Objects/ccd0879 Flats/ccd0917 Objects/ccd0880 Flats/ccd0918
Run the divflat script on this file:
sh $scriptdir/make_divflat list.base2.obj_flat do_divflat.cl # in iraf cl < do_divflat.cl
13. Check the processed files. In IRAF:
ccdlist Objects/*c2.fits to show that they are OTZF'ed (overscan, trim, zerocor, flatcor) as appropriate hselect Objects/*c2.fits $I,title,flatcor yes to check that the right flat and scale were used.
14. Clean up some intermediate files:
rm biascomb.list.c? flatcomb.list.c? *.bak
15. If desired, make mosaics of the images. Note that these mosaics are not precisely corrected for the inter-chip gaps, and not corrected at all for the short camera scale distortions and the small chip rotations. makemosaic2.cl (using imtile) will mosaic the images with 50 pixel gaps between chips. Unfortunately it doesn't copy over the image header fields. Although this is an IRAF script, it is written to be run at a shell prompt; chmod +x the file to make it executable. When it runs, IRAF emits a warning about no login.cl found, which you can ignore. There are also scripts named splitmosaic.cl and splitmosaic2.cl which will divide a mosaic back into individual images. Example uses:
mkdir Mosaics $scriptdir/makemosaic2.cl Objects/ccd0877 Mosaics/ccd0877mos $scriptdir/makemosaic2.cl Objects/ccd0878 Mosaics/ccd0878mos $scriptdir/makemosaic2.cl Objects/ccd0879 Mosaics/ccd0879mos
16. For multiple dithered exposures of a single field, it will probably be more accurate to align (shift, register) the individual CCD sub-images and combine them before mosaicing. In the future we will need a routine to correct for camera distortions in each sub image before align/shift/combine.
Note that dithering and combining mutiple exposures at the same wavelength setting is OK at the center of the field, but that near the edge of the field, the wavelength passed by the MMTF can change rapidly enough with radius that a small dither will move an object a non-negligible amount in wavelength.
To give an example of this, if you are observing with the central wavelength (R=0) at 6600 A, the wavelength at R=3900 pixels is 6511.7 A and at R=4000 pixels is 6507.2 A, so dithering by 100 pixels (20 arcsec) can move an object by 4.5 A. (These numbers can be checked by using the MMTF Fabry-Perot calculator.) At the highest resolution settings of the MMTF, FWHM < 10 A or so, this is an effect you need to take into account. If you want accurate data on objects near the edge of the field, it will be better to photometer them in individual images rather than trying to work on the combined image.
Contact:
Benjamin Weiner, bjw at as.arizona.edu, 26 Mar 2007, revised 4 April 2007