Version 0.1, 26 Mar 2007
Version 0.2, XX Yyy 2007

The major issues in this step are (1) bookkeeping for the 8 subimage files for each image basename and (2) dealing with the two directions of overscan properly. Following advice from Ian Thompson, these scripts subtract the median of the row overscan from each row (this compensates for temporal variations during the parallel transfer), and then do a conventional ccdproc-style overscan subtraction using a spline fit to the column overscan, which removes the gradient in the x-direction that is due to charge buildup (?), inherent to the way the IMACS CCD controllers work. 1. List image headers. In IRAF, cd into each image directory and do:

# in iraf
hselect ccd*c2.fits $I,naxis[12],exptime,filter,mmtf-z,title yes > Imheaders.n1

(change the output filename as needed). This file should help you to identify flats, objects, etc.

2. Copy images that you want to reduce into a working directory so you don't change the originals.

3. Set script and work directories. In the shell (not in IRAF):

set reducdir = /home/bjw/data1/workdir (or wherever your data are)
set scriptdir = /home/bjw/programs/mmtf/scripts 
  (or wherever you put the MMTF scripts)

4. Make lists of "basenames." A basename is e.g. "ccd0001" to refer to ccd0001c1.fits, ccd0001c2.fits, etc. In your working directory, assuming that each file that you want to reduce actually has all 8 subimages:

ls *c2.fits | sed s/c2\.fits// > list.base1.all

Now edit the list of basenames to create sub-lists for each category. e.g.

list.base1.standard
list.base1.for_proc  (bias, obj, flat only, may omit std stars if desired)
list.base1.bias
list.base1.flat
list.base1.obj

You will later need a file that matches each object basename with the appropriate flatfield basename, e.g. if ccd0081c*.fits is an R-band image and Skyflat_R_c*.fits is a combined R-band flat the file will have this line:

ccd0081  Skyflat_R_

See Step 12 below where this file is used, called "list.base2.obj_flat".

5. (If desired) move each category into its own subdirectory

mkdir Biases Flats Objects Standards
sh
for i in `cat list.base1.standard` ; do mv ${i}c*.fits Standards ; done
for i in `cat list.base1.bias` ; do mv ${i}c*.fits Biases ; done
for i in `cat list.base1.flat` ; do mv ${i}c*.fits Flats ; done
for i in `cat list.base1.obj` ; do mv ${i}c*.fits Objects ; done
exit

(If needed) update the lists:

ls Standards/ccd*c2.fits | sed s/c2.fits// > list.base2.standard
ls Biases/ccd*c2.fits | sed s/c2.fits// > list.base2.bias
ls Flats/ccd*c2.fits | sed s/c2.fits// > list.base2.flat
ls Objects/ccd*c2.fits | sed s/c2.fits// > list.base2.obj
cat list.base2.bias list.base2.flat list.base2.obj > list.base2.for_proc

6. Generate the IRAF cl scripts. Here, you run a set of shell scripts: each shell script doesn't do anything to the data, but writes a series of IRAF cl commands which you then pipe into IRAF. You can and should read the do_*.cl scripts to see what they do. Replace the list.base2.* names if you named the files differently. These scripts may emit a warning like mv: cannot stat `do_overscan.cl'. This is harmless (the script tries to back up any pre-existing output files so that you don't overwrite them, and this warning happens if the output file did not previously exist).

sh $scriptdir/make_imtype list.base2.bias list.base2.flat list.base2.obj do_imtype.cl
sh $scriptdir/make_overscan list.base2.for_proc do_overscan.cl
sh $scriptdir/make_biascomb list.base2.bias Biases/Bias_comb_ do_biascomb.cl

# make_biascomb issues a warning about running the script 
# before deleting the intermediate list files it makes

sh $scriptdir/make_subbias list.base2.flat Biases/Bias_comb_ do_subbias1.cl
sh $scriptdir/make_subbias list.base2.obj Biases/Bias_comb_ do_subbias2.cl

# make_subbias issues a warning about editing CCDMEAN before
# flatfielding, see below

Note: if you happen to have IMACS images that are not the standard 2048x4096 size, for example if they are binned or subrastered, you may be able to reduce them by editing the make_overscan_size script to match the sizes and running that to create do_overscan.cl. The rest of the scripts should work independent of image size.

7. Now run the scripts inside IRAF. Start iraf (cd to your iraf directory, type cl), and cd to your working directory. Then in IRAF:

# in iraf
cl < do_imtype.cl

# do_overscan.cl loads ccdred and sets some params but it's best to 
# epar ccdproc first just to make sure all the params are sensible.
noao
imred
ccdred
epar ccdproc
  ccdtype="", fixpix=no, oversca=yes, trim=yes, zerocor=no, darkcor=no,
  flatcor=no, illumcor=no, fringecor=no, readcor=no, readaxi=column

cl < do_overscan.cl
# this can take a while

epar imcombine   
# to check settings: combine=average, reject=avsigclip, scale=none, zero=none

cl < do_biascomb.cl

cl < do_subbias1.cl
cl < do_subbias2.cl

8. Check to make sure everything got done:

hselect Biases/ccd*c1.fits $I,imagetyp,trim yes
hselect Flats/ccd*c1.fits $I,imagetyp,trim yes
hselect Objects/ccd*c1.fits $I,imagetyp,trim yes
hselect Flats/ccd*c1.fits $I,imagetyp,zerocor yes
hselect Objects/ccd*c1.fits $I,imagetyp,zerocor yes

or you may want to do

ccdlist Biases/ccd*c1.fits,Flats/ccd*c1.fits,Objects/ccd*c1.fits

9. Before flat-fielding, there is a subtlety. IRAF flatfields by dividing by the flat and multiplying back by a header keyword, usually CCDMEAN (= the average value of the flat), so that 1 DN in the input is about 1 DN in the output. There are two issues here:

1. For the 8-chip mosaic, we are going to divide e.g. object_c1.fits by flat_R_c1.fits, and the same for chips 2-8. This means that we need to set CCDMEAN to be the same for each of chips 1-8 of the flat field image. (In essence, this is as if we had mosaiced the object and flat images together before dividing; we clearly have to multiply back by a single common scale.)

2. For an MMTF scan where the Fabry-Perot is set to different wavelength settings and we have a different flat for each one, the flats can have different CCDMEAN if they are at different places on the transmission curve of the blocking filter. We may want the entire scan to be multiplied back by the same CCDMEAN, rather than compensating for the curve.

10. Figure out a good value for CCDMEAN by doing an imstat near the center of the field (e.g. pixels [2000:2040,4040:4080] on chip2).

in shell:
awk '{print $1"c2.fits"}' list.base2.flat > tmp1
awk '{print $1"c2.fits[2000:2040,4040:4080]"}' list.base2.flat > tmp2
in IRAF:
hselect @tmp1 $I,filter,mmtf-z,exptime,title yes
imstat @tmp2

Compare pixel values between similar flats, and don't forget to take into account that flats may have differing exposure times. You want to set CCDMEAN to a common value for each matching set of flats, but have different CCDMEAN for e.g. flats in different filters. If some flats are twilight flats, then you will want to combine them with imcombine scale=median and adjust CCDMEAN on the combined flat.

11. Adjust CCDMEAN. Suppose that we have two sets of flats in the 6600 filter that need different CCDMEAN of 31500 and 18900 (5 sec and 3 sec dome flats), and another set in an R filter that need to be combined first. First, combine the R flats and set CCDMEAN:

put names of Rflats in list.base2.flat_R
put "Flats/Rflat_comb_" in list.base2.Rflatcomb
sh $scriptdir/make_flatcomb list.base2.flat_R Flats/Rflat_comb_ do_Rflat_comb.cl
# in iraf
cl < do_Rflat_comb.cl
imstat Flats/Rflat_comb_c2[2000:2040,4000:4040]
# to get value to set CCDMEAN to, here 18800
sh $scriptdir/make_ccdmean list.base2.Rflatcomb 18800 do_ccdmean_R.cl
# in iraf
cl < do_ccdmean_R.cl

Now look at the narrow band flats (here some MMTF flats in the 6600 filter), sort them into lists of ones that need a matching CCDMEAN (here we had some with 5 sec exposure and some with 3 sec, which need different CCDMEAN):

edit list.base2.flat to make list.base2.flat.6600a, list.base2.flat.6600b
sh $scriptdir/make_ccdmean list.base2.flat.6600a 31500 do_ccdmean_1.cl
sh $scriptdir/make_ccdmean list.base2.flat.6600b 18900 do_ccdmean_2.cl
# in iraf
cl < do_ccdmean_1.cl
cl < do_ccdmean_2.cl
# check that CCDMEAN was set correctly
# in iraf
hselect @list.base2.flat.sub $I,title,ccdmean yes

12. Now that CCDMEAN is set correctly to prevent any unpleasant rescaling surprises, actually do the flat fielding. Here, list.base2.obj_flat is a file with two columns. The first column is a list of object basenames to be flat-fielded and the second column is the corresponding flat basename, for example:

Objects/ccd0877  Flats/ccd0916
Objects/ccd0878  Flats/ccd0917
Objects/ccd0879  Flats/ccd0917
Objects/ccd0880  Flats/ccd0918

Run the divflat script on this file:

sh $scriptdir/make_divflat list.base2.obj_flat do_divflat.cl
# in iraf
cl < do_divflat.cl

13. Check the processed files. In IRAF:

ccdlist Objects/*c2.fits 
  to show that they are OTZF'ed (overscan, trim, zerocor, flatcor) as appropriate
hselect Objects/*c2.fits $I,title,flatcor yes 
  to check that the right flat and scale were used.

14. Clean up some intermediate files:

rm biascomb.list.c? flatcomb.list.c? *.bak

15. If desired, make mosaics of the images. Note that these mosaics are not precisely corrected for the inter-chip gaps, and not corrected at all for the short camera scale distortions and the small chip rotations. makemosaic2.cl (using imtile) will mosaic the images with 50 pixel gaps between chips. Unfortunately it doesn't copy over the image header fields. Although this is an IRAF script, it is written to be run at a shell prompt; chmod +x the file to make it executable. When it runs, IRAF emits a warning about no login.cl found, which you can ignore. There are also scripts named splitmosaic.cl and splitmosaic2.cl which will divide a mosaic back into individual images. Example uses:

mkdir Mosaics
$scriptdir/makemosaic2.cl Objects/ccd0877 Mosaics/ccd0877mos
$scriptdir/makemosaic2.cl Objects/ccd0878 Mosaics/ccd0878mos
$scriptdir/makemosaic2.cl Objects/ccd0879 Mosaics/ccd0879mos

16. For multiple dithered exposures of a single field, it will probably be more accurate to align (shift, register) the individual CCD sub-images and combine them before mosaicing. In the future we will need a routine to correct for camera distortions in each sub image before align/shift/combine.

Note that dithering and combining mutiple exposures at the same wavelength setting is OK at the center of the field, but that near the edge of the field, the wavelength passed by the MMTF can change rapidly enough with radius that a small dither will move an object a non-negligible amount in wavelength.

To give an example of this, if you are observing with the central wavelength (R=0) at 6600 A, the wavelength at R=3900 pixels is 6511.7 A and at R=4000 pixels is 6507.2 A, so dithering by 100 pixels (20 arcsec) can move an object by 4.5 A. (These numbers can be checked by using the MMTF Fabry-Perot calculator.) At the highest resolution settings of the MMTF, FWHM < 10 A or so, this is an effect you need to take into account. If you want accurate data on objects near the edge of the field, it will be better to photometer them in individual images rather than trying to work on the combined image.