ASTROWISE OAC TEAM The ASTROWISE project status at

ASTROWISE OAC TEAM The ASTRO-WISE project: status at OAC The OAC AW team: J. M. Alcalà, F. Getman, A. Grado, M. Pavlov, E. Puddu, M. Radovich, R. Silvotti, A. Volpicelli Topics: • QC tools • Calibration pipeline review • Weight maps • Capodimonte Deep Field: AW vs IRAF • The new VST beowulf Groningen, 17 -21 November 2003 1

QC status (1) ASTROWISE OAC TEAM Groningen, 17 -21 November 2003 2

QC status (2) ASTROWISE OAC TEAM Groningen, 17 -21 November 2003 3

QC status (3) ASTROWISE OAC TEAM Groningen, 17 -21 November 2003 4

ASTROWISE OAC TEAM OAC QC tools 1) Sign test: correlated noise detection in bias frames 2) QCdome and QCtwilight: measure the relative flatness of dome and twilight flat fields 3) Outlayers: count the number of outlayers 4) Histo: check statistics (create histograms and compare with Gaussian distribution) Groningen, 17 -21 November 2003 5

1) Sign test: correlated noise detection in bias frames ASTROWISE OAC TEAM the input frame is splitted in a number n of windows for each window, the excess of elements greater (or smaller) than the median of the whole frame is counted if this number is not compatible with the binomial distribution, the window is flagged the number of bad windows is compared with the number expected from the Gaussian distribution at a fixed number of sigmas and the quality index is obtained by: QI=(n_bad n_expected) / Groningen, 17 -21 November 2003 6

2) QCdome and QCtwilight: ASTROWISE OAC TEAM measure the relative flatness of dome and twilight flat fields the input frame is normalised (divided by its median value) it is splitted in a number n of windows the median of each window is calculated the relative flatness is given by the difference between highest and lowest median for the twilight FFs a cubic spline fit is used instead of the input image Groningen, 17 -21 November 2003 Test of QCdome on WFI frames. Left: V-band dome flats divided by their master. Right: same V-band dome flats divided by the B-band master. 7

ASTROWISE OAC TEAM Fig. 1 Fig. 2 Test of QCtwilight on WFI frames. Fig. 1: V-band dome flats divided by their master (left points) and by the B-band master (right points). Fig. 2: B-band dome flats divided by their master (left points) and by the V-band master (right points). Groningen, 17 -21 November 2003 8

4) Histo: check statistics (create histograms and compare ASTROWISE OAC TEAM with Gaussian distribution) The histo tool applied to a dome flat frame; note that the left panel is in logarithmic scale. Groningen, 17 -21 November 2003 9

Calibration pipeline review: Overscan Correction ASTROWISE OAC TEAM Present algorithm (6 different methods): no overscan correction (default) median (prescan_x) median (overscan_x) median (prescan_y) median (overscan_y) average_per_row (prescan_x) average_per_row (overscan_x) Groningen, 17 -21 November 2003 currently used 10

Calibration pipeline review: Dome Flat ASTROWISE OAC TEAM Previous algorithm: normalise_by_mean (stack_by_median (normalise_by_mean (overscan_corrected (raw_dome) - master_bias ))) 1. overscan correction for each ‘raw_dome’ 2. master_bias subtraction 3. normalisation to 1. 0 of each image by mean value 4. stacking by median 5. normalisation again We proposed: 1. overscan correction for each ‘raw_dome’ 2. master_bias subtraction 3. sigma_clip_average (including internal normalisation) 4. final normalisation Groningen, 17 -21 November 2003 Done ! (07/11/03) 11

Calibration pipeline review: Twilight Flat ASTROWISE OAC TEAM Previous algorithm: normalise_by_mean (stack_by_median (normalise_by_mean (overscan_corrected (raw_twilight) - master_bias ))) 1. overscan correction for each ‘raw_twilight’ 2. master_bias subtraction 3. normalisation to 1. 0 of each image by mean value 4. stacking by median 5. normalisation again We proposed: 1. overscan correction for each ‘raw_twilight’ 2. master_bias subtraction 3. sigma_clip_average (including internal normalisation) 4. final normalisation Done ! (07/11/03) Note that in this case the sigma_clipping technique is even more important in order to remove stars from raw twilight FFs ! Groningen, 17 -21 the November 2003 12

Calibration pipeline review: Master Flat ASTROWISE OAC TEAM Present algorithm: normalise_by_mean ((M_dome / low_freq_component (M_dome)) low_freq_component (M_twilight)) 1. FFT of ‘master_dome’ 2. FFT of ‘master twilight’ 3. divide master dome by its low freq. component 4. multiply by low freq. component of master twilight 5. normalise to 1. 0 by mean value Warning: The presence of bad pixels may introduce artificial frequencies that would be cut by FFT introducing errors. This problem can be critical for WFI data (e. g. CCD#3) other possible solutions: a) smoothing by running window (which can eventually have variable dimensions i. e. smaller windows near the edges) b) cubic spline interpolation Groningen, 17 -21 November 2003 13

Calibration pipeline review: Fringing Pattern ASTROWISE OAC TEAM Present algorithm: norm_by_mean (stack_by_median (norm_by_median ((overscan_corrected (raw_science) - master_bias)/master_flat))) - 1. 0 1. overscan correction for each ‘raw_science’ 2. master_bias subtraction 3. Master_FF correction 4. normalisation to 1. 0 of each image by median value 5. stacking by median 6. normalisation again to 1. 0 by mean value Note that this algorithm contains already the assumption that the amplitude (or scaling factor) of the fringing is proportional to the background ! Otherwise we could not make any stacking. We suggest: To introduce the sigma_clip (with possible modifications) To divide each single corrected scientific frame by its low freq. component before combining them to obtain the fringing pattern, in order to obtain a uniform fringing amplitude across the field. The fringing pattern will then be multiplied by the low frequency component of each scientific frame before being subtracted to that scientific frame. Groningen, 17 -21 November 2003 14

Calibration pipeline review: Reduced science frame ASTROWISE OAC TEAM Present algorithm: ((overscan_corrected (raw_science) master_bias) / master_flat) scaling_factor fringing_pattern Where scaling_factor is calculated with two different methods: a) minimisation of i, j(corrected_science scaling_factor fringing_pattern)2 ; high pixels (stars) are previously removed by a simple sigmaclip rejection b) scaling_factor is proportional to (background stars), where stars are removed using a simple sigmaclip of high pixels. Comments - suggestions: in the first method the present 20 sigma clipping is dangerous in crowded fields where the high presence of objects can lead to a bad solution ! The two scaling factors might be compared each other as a further QC test and saved in the DB for future trend analysis. Groningen, 17 -21 November 2003 15

Calibration pipeline review: fringing scaling factor ASTROWISE OAC TEAM WFI scientific frame at 914 nm (CCD#1) corrected for bias and FF before and after fringing correction Groningen, 17 -21 November 2003 16

Calibration pipeline review: fringing scaling factor ASTROWISE OAC TEAM Residual fringing: the same scientific frame is corrected for fringing using method a) and method b) to calculate the scaling factor. The difference shows a residual amplitude of the order of 0. 3 of the background (or 6 of the original fringing amplitude). Groningen, 17 -21 November 2003 17

Calibration pipeline review: Performance ASTROWISE OAC TEAM INGESTION of 1 WFI extension: ~1 min for VST/Omega. CAM it would take half a day to ingest 100 Giga of raw data ! Total time to produce all the calibration frames ~20 min (WFI frames); most of this time (~15 min) is used to create the Master Flat with FFTs. Fringing pattern scaling factors: computing times are ~150 s and ~10 s respectively using the two methods described in the previous slide. By the way, the 150 s can be easily reduced improving the algorithm (derivative=0 instead of 30 iterations cycle), as already suggested by Mike Pavlov. FFTs are quite heavy in terms of computing time; using smoothing or cubic spline interpolation instead of FFTs would reduce the times needed to produce calibration frames from ~20 to less than 10 minutes Groningen, 17 -21 November 2003 18

ASTROWISE OAC TEAM Pixel and weight maps Current scheme: weight_map = flat (hot_map cold_map cosmic_map) We propose: to flag also high pixels (close to the saturation) in raw scientific frames to compute more realistic weight maps, considering all the errors introduced by each step of the reduction (overscan, bias, FF, fringing): weight_map = (1/D)1/2 (hot_map cold_map saturated_map cosmic_map) where: D(i, j) sigma 2 (i, j) = D[bias_corrected](i, j) D[master_flat](i, j) / master_flat(i, j)4 + D[master_flat](i, j) bias_corrected_frame(i, j)2 / master_flat(i, j)4 + D[bias_corrected](i, j) / master_flat(i, j)2 Groningen, 17 -21 November 2003 19

Calibration pipeline review: a few more comments ASTROWISE OAC TEAM To use average instead of median reduces errors: D[median]/D[average] = Pi/2 1. 57 It can happen that fringing is present also on the dome flats (in WFI data we see sometime this effect, due to the sky light entering the dome). With the present algorithm for the master flat this is critical because fringing is not removed from dome flats (whereas it is removed from twilight flats). Groningen, 17 -21 November 2003 20

Capodimonte Deep Field: AW vs IRAF ASTROWISE OAC TEAM The comparisons will consider two broad band filters (V and R) + one medium band at 914 nm (where the fringing is maximum). As a first step we plan to compare: 1. Master FF 2. Flatness of the reduced images. 3. Residual fringing. 4. Instrumental magnitudes of the point sources (obtained running Sextractor, with same parameters, on single exposures) in the different CCDs and in the different parts of each CCD. 5. Zero points and color terms from the standard fields. Groningen, 17 -21 November 2003 21

The new VST beowulf ASTROWISE OAC TEAM Master: 16 Nodes: MB: Tyan 2880 CPU: two Opteron 244 RAM: 6 GB DDR PC 2700 ECC registered DISK: 40 GB system Case: rack mount 2 U Storage: 1. 5 TB disks + 3 ware 7506 MB Tyan 2880 CPU: two Opteron 244 RAM: 2 GB DDR PC 2700 ECC registered DISK: 40 GB system Case: rack mount 2 U Switch: 24 ports Gigabit Groningen, 17 -21 November 2003 KVM: 18 ports 22