CANICA data reduction with IRAF
Reduction of near infrared (NIR) data is laborious due to the amount of frames that are necessary to handle in order to produce an image of a single object. Typically around 10 frames would have to be handled in order to produce an image of net exposure time of 5 minutes in the K-band. The best way to handle these multiple exposure frames is through a pipeline. This manual describes such a pipeline, written for the reduction of data taken with CANICA, the CAnanea Near Infrared CAmera. The pipeline is a script based on various tasks under the IRAF environment.
What are the basic steps in NIR data Reduction?
- Bias subtraction
- Sky frame preparation and subtraction
- Flat fielding
- Image alignment and combining
- Stellar photometry
Data Acquisition with CANICA and File name Nomenclature
CANICA acquires images under Linux environment. Images are acquired by sending an observing sequence containing RA-Dec coordinates and relative shifts through a Java script. Useful observational parameters, such as RA-Dec coordinates, filter, exposure time etc. of each image are written into image headers. The image names are formed from a root name by concatenating a three digit number, which is incremented sequentially until the sequence ends. Typical image names are of the type root_name001_I.fits, root_name002_I.fits, ...... root_name0nn_I.fits. Images taken using the Java sequence until now (June 2003) start with number 002 (001 is reserved for a "training" image). Bias subtraction is done during the acquisition. However for each image, a bias image is also stored. Bias images can be identified by their termination "_Z.fits".
Multiple exposure co-adds are not necessary with CANICA, given that the combination of parameters (image scale, Full well capacity and the sky background) allows single exposures of duration greater than 60 seconds in the K-band.
Basic function of the script
The most important step in the reduction of NIR images is the sky pattern subraction. sky
in NIR images is a combination of the real sky background and a local background originated due to the emission of the warm optics surrounding the detector. The former is expected to be uniform, whereas the latter contribution can be non-uniform over the detector. Relative contribution of the two backgrounds depends on the wavelength. In addition, any vignetting gives a structure to the background. Hence it is necessary to subtract a sky frame from each object frame.
The pipeline starts by first identifying the sky frames in a given sequence. The RA-DEC coordinates are used to separate sky frames from object frames. Sky frames are then combined (imcombine combine=median) to prepare a star-free sky image.
The next step is to flat field the images using a previously prepared flat field frame. The thermal emission of the warm optics makes the task of getting a good flat field image somewhat difficult. Ideally flat field image should be prepared by subtracting a mean night sky image from a mean dawn/dusk image in the same filter, both exposed to same duration.
Centroid positions of a reference star are then obtained in each sky subtracted, flat-fielded image. These centroids are used to find shifts of each frame with respect to the reference frame. Images are aligned using the calculated shifts and then combined (imcombine combine=average, reject=minmax).
Photometry of the reference star (or any other object) is then carried out using the technique of aperture photometry. The combined image is then transformed into the normal astronomical coordinate convention (North to the top, east to the left).
CANICA script names and their usage
The script which does all the reduction is called canobj. For an error-free reduction canobj should be run twice. The first run is a trial run, basically to have a first-hand look at the images. In this run, an aproximate sky frame is prepared and subtracted from all the frames of the sequence. The resulting frames are displayed one by one, so as to enable a quick look at each frame. The display also helps to identify at least one star common to all the object frames. Before running canobj a second time, a script called canref is then run, which reads header parameters, and substitutes suitable values for the parameters of canobj. Now canobj can be re-run without the need to edit any of its parameters. Here is a summary of all the scripts:
- canobj: the main reduction script for canica images.
- canref: reads the object coordinates from image headers and estimates the position of the reference star in all the frames of a sequence. This script also sets all the parameters that are necessary for a successful run of canobj.
- cansky: form a sky using the given frames.
Description of parameters of canobj
cl> epar canobj
Image Reduction and Analysis Facility
PACKAGE = user
TASK = canobj
root = as23k Root Image Name
imbeg = 1 First Image Number
imend = 6 Last Image Number
(sequenc= as23k002.seq) Sequence file [file.seq/preview]
(skyfram= stack) Sky frame [stack/name]
(flattyp= null) Flat-fielding strategy [skynew/skyold/flat/null]
(flatim = ../flatk) Flat field frame Name
(refimag= as23j002ref) Reference Image name
(iref = 2) ref star image number
(skyedit= no) Clean the sky using imedit?
(disp = yes) Display before imcombine?
(submed = no) Remove the horizontal band?
(immos = yes) Mosaic the images?
(dophot = yes) Do photometry?
(xshift = 2.) X-shift[=Xref-Xin] wrt refimag
(yshift = -15.) Y-shift[=Yref-Yin] wrt refimag
(skyout = 600.) Minimum Distance of the off-field sky
(skyin = 150.) Minimum Distance of the on-field sky
(skysect= [551:575,551:575]) Section for skyvalue estimation
(flatsec= [551:575,551:575]) Section for flat normalization
(boxsize= 15.) imalign boxsize
(bigbox = 21.) imalign bigbox
(crval1 = 180.73875) RA of the header (in degrees)
(crval2 = 4.1288055555556) Dec of the header (in degrees)
(crpix1 = 512.5) X-Pixel corresponding to the RA-Dec
(crpix2 = 512.5) Y-Pixel corresponding to the RA-Dec
(cdelt1 = 8.8888888888889E-5) Pixel scale in RA (in degrees)
(cdelt2 = 8.8888888888889E-5) Pixel scale in Dec (in degrees)
(epoch = 2000.) Epoch of RA-Dec
(dowcs = yes) Transform the image to N_up E_left?
(xymag = 2.) XY reduction factor if dowcs=yes
(jhkzp = 20.60 20.40 20.20) JHK Zeropoints
(mode = ql)
| Parameter |
Default/ acceptable values |
Short description |
|---|---|---|
| root |
root file name (e.g. as23k for files of the type as23k002_I.fits, as23k003_I.fits, as23k004_I.fits etc) |
|
| imbeg | the first number of the sequence (2 if the first image is as23k002_I.fits) | |
| imend | the last number of the sequence (7 if the last image is as23k007_I.fits) | |
| sequenc | preview/file.seq |
A filename (e.g. file.seq) containing the positions of a reference star or preview. The default option preview allows a trial run for the sequence. When sequenc=preview, the only parameters that are used are the first three parameters (root, imbeg and imend). So cl> canobj sequenc=preview can be run for any sequence without any prior knowledge of the kind of object it contains. The second run of canobj should have sequenc=file.seq, in which case many parameters of canobj should have been to set to proper values. The complementary script canref helps in setting these parameters. Read the help on canref. The file containing the positions of the reference star is automatically prepared when running canref. The file is a simple text file containing xy pixel coordinates of the chosen common star. In case, telescope pointing is wrong, it may be required to measure the position of the star in each frame, and then correct the entries in the sequence file. Any frame with coordinates outside [1,1024] pixel range implies that the frame is a sky frame. The file should contain exactly the same number of lines as the number of frames in the sequence. |
| skyfram | stack/name | Algorithm for determining the sky frames. With the option stack, sky frames are guessed based on the information in the sequence file sequenc, and then combined (imcombine combine=median zero=median) to prepare a star-free sky frame. Optionally the frame can be edited interactively using imedit (if skyedit=yes). all, sky frame is prepared using all the frames (suitable when the object is stellar, and observations are done with dithering technique).> The option name allows the user to give the name of an already prepared sky frame. |
| flattyp | null / flat / skynew / skyold |
Flat-fielding strategies. With the option null flat fielding is not done. With flat, flat fielding is done using the flat image contained in the parameter flatim. With skynew, the normalized sky frame is used as a flat field image. In this case, sky value (not frame) is subtracted. With skyold, the sky image contained in the parameter flatim is used as a flat field image. Suitable when a good flat field is obtained in the current observing run using night-sky observations. When flattyp=skyold, sky frame formed using the images in the current sequence is used only to estimate the sky value, based on the parameter biassec. Effectively, this option reduces to the optical type of data reduction (i.e. flat fielding first, and then subtract an average sky value). It is not necessary to pre-normalize the flat fields, as the program normalizes the flat image using the average in the parameter flatsec. Warning: flattyp=skynew and skyold are meant for test purposes. Hence values null or flat should be preferred for routine reductions. |
| flatim |
Flat field frame Name (when flattyp=flat) or the name of the previously prepared sky image (when flattyp=skyold) obtained in the current observing run using night-sky observations. In the former case, flat field image should have been prepared by subtracting a mean night sky image from a mean dawn/dusk image in the same filter, both exposed to same duration. |
|
| refimag | as23j002ref | Root name for files that contain the coordinate information of a common star in the reference frame. e.g. for refimag=as23j002ref, the working directory should contain files as23j002ref.coo (xy coordinates of the reference star), as23j002ref.fits (image containing the reference star at the reference position) and as23j002ref.xy (xy coordinates of stars for aperture photometry). These three files are automatically created during the run of canref. Otherwise should be created by the user. It is not necessary that the image refimag .fits be part of the current sequence. For example, it could be an image of the same field, but in another filter. See help on xshift,yshift. |
| iref | 2 | Reference star image number. To which image number, the reference position corresponds (e.g. 2 if the reference coordinates are measured in the image as23k002)? Its value is setup automatically during the run of canref. Otherwise should be entered by the user. It is an error if iref > imend or iref < imbeg. |
| skyedit | yes/no | Clean the sky using imedit? If skyedit=no, script uses the prepared sky (without displaying it) and continues with the rest of canobj. If yes, combined sky frame is displayed with the IRAF command imedit. Enter "q" to quit imedit and continue with canobj. Sky editing is required in only exceptional cases and hence normally enter "q" when the cursor is in the image window. Useful imedit keystroke commands are:
a Background replacement (rectangle) b Background replacement (aperture) d Constant value substitution (rectangle) e Constant value substitution (aperture) u Undo last change (see also 'i', 'j', and 'k') + Increase radius by one - Decrease radius by one Statistics q Quit and save changes Q Quit without saving changesRead help on imedit for further details. |
| disp | yes/no |
Display before imcombine? With disp=yes, bad images can be interactively rejected from the list of images to be aligned and averaged (combined). With disp=no, program runs in non-interactive mode. |
| submed | no/yes | Remove the horizontal band using the wrap technique? (default option "no" should be good for majority of the sequences, and need to be changed to "yes" in very exceptional cases). |
| immos | yes/no | Mosaic the images? |
| dophot | yes/no | Do photometry? |
| xshift,yshift | 0,0 | X-shift[=Xref-Xin] and Y-shift[=Yref-Yin] with respect to the reference image. Especially useful when all the images of a sequence need to be aligned to an image not included in the sequence (e.g. reference image is in the J-band, and we want all the K-images aligned to the J-band reference image). In such a case, mean shift in pixels between the two sets should be entered here. However, if coordinates of the reference star in each image are measured and entered into the "sequenc" file, then XY shifts should be left at their default values of zero. |
| skyout | 600 | Minimum shift of the off-field sky frame. If a frame in the sequence is shifted more than skyout pixels with respect to the reference frame, then it is assumed that the frame does not contain the object of interest, and that frame is used only as a sky frame. |
| skyin | 150. | Minimum shift of the on-field sky frame. While observing point sources, it is not necessary to go completely out of the object field to obtain good sky frames. It is a practice to center the point source in different quadrants for different exposures of a single sequence. The parameter skyin is the minimum shift of a frame with respect to the reference, in order to consider that frame suitable for sky extraction. So if the shift of a frame (wrt to the reference) is less than skyin, then it is not used for sky extraction, and if the shift is more than skyout, then it is not used for object extraction. All frames with shifts greater than skyin are used as sky frames. Frames in a sequence with shifts greater than skyin, but less than skyout, are used both for sky and object extraction. Warning:: For off-field sky frames (shifts > skyout), make sure that each off-field sky in a sequence is distinct from the rest. Also when point sources are moved from one quandrant to the other, and shifts are intermediate between skyin and skyout, make sure that the position of the point source is not repeated in a single sequence. In both these cases, it is recommended to use cansky to get a good sky frame and then give it in the parameter skyfram. |
| skysect | [551:575, 551:575] | Section for skyvalue estimation. This parameter is used only when flattyp=skynew. The section is used to estimate the average sky counts, which will be subtracted from all images. |
| flatsec | [551:575, 551:575] | Section for flat normalization. Flat field frame is normalized to the average of this section. |
| boxsize | 11 | Imalign boxsize. The size in pixels of the box to use for the final centering, during which all the sources in refimag.coo are recentered in each image using the initial estimate of the relative shift for each image. Care should be taken to choose an appropriate value for this parameter, since it is highly data dependent. See imalign. |
| bigbox | 15 | Imalign bigbox. The size in pixels of the box to use for coarse centering. The coarse pass through the centering algorithm is made with the box centered at the nominal position of the first source in the coordinate list. Coarse centering is performed only if the shifts file is undefined. Care should be taken to choose an appropriate value for this parameter, since it is highly data dependent. Large values should be suspect until the final results are checked to see that the centering did not converge on the wrong coordinates, although the usual result for an inappropriate bigbox size is that the algorithm fails to converge and the task aborts. See imalign.
What to do if the task aborts? |
| crval1, crval2 | 0,0 | Right Ascension and Declination (both in degrees) corresponding to the pixel (crpix1,crpix2). When canref is run, the header RA and Dec values of the reference frame are substituted here. The image header of the combined final image is updated with these keywords. |
| crpix1, crpix2 | 512.5,512.5 |
Pixel corresponding to the crval1, crval2 values. When canref is run, these parameters take the values for the center of the chip (canref.crpix=center) or the pixel value of the reference star (canref.crpix=refstar). T he image header of the combined final image is updated with these keywords. |
| cdelt1, cdelt2 | 8.888889E-5, 8.888889E-5 |
Pixel scale in RA and Dec (in degrees/pixel). The default values correspond to 0.32 arcsec/pixel. It is assumed that the detector had been aligned before the observations, such that RA and Dec axes coincide with X-axis, and Y-axis respectively. Both the RA and Declination increase in the same direction as the pixels, i.e. North is up, and East is to the right. |
| dowcs | yes | Transform the image to N_up E_left? If yes, rotates the X-axis by 180 degrees, so that RA increases to the left (i.e. East is to the left). |
| xymag | 2. | XY reduction factor. If dowcs=yes, the image is compressed by the factor xymag , in addition to rotating the X-axis by 180 degrees. |
| jhkzp | 20.60 20.40 20.20 |
JHK Zeropoints to be used when doing photometry of the objects in the file refimag.xy. The script automatically detects the filters and uses the corresponding Zeropoint. It is assumed that the the image header keyword FILTER2 = 1,2,3 for J,H and K bands respectively. For any other value of FILTER2, a zeropoint of 20 is used. |
Description of parameters of canref
cl> epar canref
Image Reduction and Analysis Facility PACKAGE = user TASK = canref imref = 2 No of the ref image (2 for as40k002 etc.) refcoo = as23j002ref file (ref.coo) with ref star position or as40k00 (crpix = center) [center/refstar] XY pixels of RA-Dec (mode = ql)
| Parameter | Default/acceptable values | Short description |
|---|---|---|
| imref | 2 | No of the ref image (2 for as40k002 etc.) |
| refcoo | as23j002ref | file (ref.coo) with ref star position or as40k002ref. File "ref.coo" should contain the position (pixels) of the reference star. Name of the file can be anything that ends with extension ".coo". If the file name does not end with ".coo", then it is assumed that you are giving the value suitable for canobj.refimag. Purpose of this option is to align images in a second filter (e.g. K) to previously analysed filter image (e.g. J). In this special case, files refimag.coo, refimag.xy and refimag.fits should have been created manually or through an earlier run of canref. |
| crpix | center/refstar | [center/refstar] XY pixels corresponding to the header RA-Dec. Default option corresponds to the center of the image [512.5,512.5]. With "refstar" header RA-Dec values are asigned to the reference star. |
Guidelines and examples for successful running of canobj and canref
cl> ls as23*I.fits as23h001_I.fits as23h002_I.fits as23h003_I.fits as23h004_I.fits as23h005_I.fits as23h006_I.fits as23j001_I.fits as23j002_I.fits as23j003_I.fits as23j004_I.fits as23j005_I.fits as23j006_I.fits as23k001_I.fits as23k002_I.fits as23k003_I.fits as23k004_I.fits as23k005_I.fits as23k006_I.fits cl> unlearn canobj cl> canobj as23j 1 6 sequenc=preview
... when images are displayed one by one, identify one star (preferably in image "as23j002_I.fits"), and find its centroid by pressing "a" when the cursor is the image window. Once the canobj trial run is over, edit (with any editor of your preference) a file called "ref.coo" and paste the xy pixel coordiantes of the identified star. You are now ready to run canref.
cl> canref 2 ref.coo
...Hit "Return" for all the querries. You will note that the querried parameters are basically the first 3 parameters of canobj. When the task ends, the RA and Dec of each image in the sequence, as well as the shifts of images with respect to the reference image, appears on the screen. Canref passes all the parameters, that are necessary to align the images, to the parent task canobj. You are now ready to run canobj for the second time. If you have a good flat field, then change the canobj.flattyp=flat and canobj.flatim="flatname". Otherwise keep them at their default values.
cl> canobj
...Hit "Return" for all the querries. Quit "imedit" and "imexamin" tasks that canobj invokes, with 'q", and if all the images look good, keep hitting "Return". A series of messages would appear on the text window. If the script ends successfully, the final rotated image (as23j_2a.fits) is displayed with the coordinate grid superimposed. On text screen, you will see magnitudes of the reference star in each image, followed by its magnitude in as23j_2.fits. Make sure that the coordinates and magnitudes of the star in different images of the sequence, do not differ by more than 2 pixels and 1 mag (approximately), respectively.
Input/Output filenames
| Filename | Examples | Short description |
|---|---|---|
| root | as23k | An example for the root file name |
| iref | 2 | Reference image number. 2===> as23k002_I.fits is the reference image |
| rootiii_I.fits |
as23k002_I.fits, as23k003_I.fits, as23k004_I.fits etc |
Input file name with raw data (bias already subtracted during data acquisition) |
| rootiiisf.fits with iii = imbeg to imend | as23k002sf.fits, as23k003sf.fits, as23k004sf.fits etc | Sky subtracted, and flat-fielded Images. Saved only when the the script aborts due to error in centering stars. |
| rgrootiiisf.fits with iii = imbeg to imend | rgas23k002sf.fits, rgas23k003sf.fits, rgas23k004sf.fits etc | Sky subtracted, and flat-fielded and Aligned Images. |
| root_iref.fits | as23k_2.fits | The final image resulting from combining (imcombine combine=average reject=minmax) all rgrootiiisf.fits. |
| root_irefa.fits | as23k_2a.fits | X-axis of Image root_iref.fits rotated by 180 deg and compressed by xymag (generated only when dowcs=yes). |
| root_iref_sky.fits | as23k_2_sky.fits | Combined (and edited) sky frame (imcombine combine=median zero=median) |
| refimag.fits | as23k002ref.fits | Sky subtracted Image containing the reference star at the reference position. |
| root00iref.seq | as23k002.seq | Text file containing the xy pixels of the reference star in each of the sequence images. |
| refimag.coo | as23k002ref.coo | Text file containing the xy pixels of the reference star(s) in the reference image. |
| refimag.xy | as23k002ref.xy | Text file containing the xy pixels of all stars for which a quick photometry is to be carried out. |
| refimag.mag | as23k002ref.mag | Text file containing the photometric results of the first four stars in refimag.xy. Output of txdump. |
| rgrootiiisf.mag.1 with iii = imbeg to imend | rgas23k002sf.mag.1, rgas23k003sf.mag.1, rgas23k004sf.mag.1 etc. | Unedited output of photometry with phot. |
| root-canica.log | as23k-canica.log | Log file containing the details of reduction with canobj. |
| imexam.log | Log file containing the results of imexamin task | |
Frequently Encountered Problems and Troubleshooting
- What do I do if the task canobj aborts in the second run?
See the help of canobj.bigbox above.
- What do I do if the task canref aborts?
The only reason this could happen is if the header keywords "AR" or "DEC" are empty for at least one image in the sequence. Mostly this error is due to the header of the first image in the sequence, which is probably the "training" image. In such a case, you should edit the header parameters "AR" or "DEC" using "hedit", and put the RA and Dec values corresponding to the next image in the sequence. As of now (Dec 2003), the first two images should have the same coordinates. Even when canref does not abort, if you find that the shifts printed out is different for the first two images, then also, you should correct the header keywords "AR" or "DEC", and re-run canref.
- What do I do if I don't have a good flat field?
- Skyframes have too many "black structures" or "stellar residuals".
- None of the images in the sequence are "off-source". So how do I obtain sky?
- I want my JHK bands to be aligned to each other? How can I do that?
- How reliable are the astrometric parameters putin by canobj?
Frequently asked questions (FAQs)
- Is it correct to refer near infrared images as CCD images?
Technically speaking No. They should be called near infrared array images. Unlike CCDs, NIR array detectors do not use Charge transfer to collect photoelectrons. Instead the amount of charge (or number of photoelectrons) at each pixel is sensed using a capacitor. - What is the basic difference between a NIR observation and a CCD observation?
Observations in the optical and the NIR differ basically due to the differing levels of background values. At optical wavelengths, the background comes mainly from scattering of city lights by atmospheric particles, where as at infrared wavelengths, the thermal emission from the atmosphere and/or the detector/telescope, competes (and even exceedes in the K-band), the emission due to scattering. Observations in the NIR bands are usually background-limited, and not limited by the strength of the source of interest, as the case in the optical. Thus total exposure time required to get the desired S/N ratio should be achieved through multiple exposures in the NIR, where as single exposure could do that in the optical wavelengths. - Is there any difference in the bias frames for NIR array and CCD observations?
- What about flat-fielding in the NIR wavelengths?
- Why subtract sky frame, and not a value in NIR observations?
- How does the photometric accuracy of CCDs compare with that of NIR arrays?
- What are the interesting objects for NIR observations?
- What is the trade-off between sensitivity, and background, for highly reddened objects? (What is the wavelength dependence of sensitivity, background and extinction law in the JHK bands?)
Dirección: Luis Enrique Erro # 1, Tonantzintla, Puebla, Mexico C.P. 72840 Teléfono: (222) 266.31.00 Contacto: difusion@inaoep.mx
Esta obra está licenciada bajo una Licencia Creative Commons Atribución-No Comercial-Sin Obras Derivadas 2.5 México
