Data Processing and Quality Control Group part of

Data Processing and Quality Control Group part of Data Products Department • leader: Reinhard Hanuschik • QC group: – 7+1 QC scientists (astronomers) – data processing and quality control, all Paranal data DPD: Data Processing and Quality Control Group [www. eso. org/qc] DMO Meet Yourself day January 9 th, 2009

• QC group: – Reinhard Hanuschik – Danuta Dobrzycka – Wolfgang Hummel – Sabine Moehler – Mark Neeser – Isabelle Percheron – John Pritchard – Burkhard Wolff DPD: Data Processing and Quality Control Group [www. eso. org/qc] DMO Meet Yourself day January 9 th, 2009

• QC mission: ensure that science data can be calibrated, to a known and documented level • • • pipeline-process all VLT+VLTI data certify calibration products, store in archive process Service. Mode science data create Service. Mode packages for PIs reprocessing DPD: Data Processing and Quality Control Group [www. eso. org/qc] DMO Meet Yourself day January 9 th, 2009

• supported: – all Paranal instruments: 12 VLT, 2 VLTI instruments [plus 2 survey telescopes (VIRCAM, OCAM)] – La Silla data packing DPD: Data Processing and Quality Control Group [www. eso. org/qc] DMO Meet Yourself day January 9 th, 2009

• statistics – 14 instruments, 8 QC scientists 2 ins/QC – per month: > 18. 000 processing jobs (2006: 13. 000) – 600 GB raw data processed (2006: 410 GB) – 90 SM data packages • reprocessing – UVES 2006 – “nice images”: project with Outreach DPD: Data Processing and Quality Control Group [www. eso. org/qc] DMO Meet Yourself day January 9 th, 2009

• services – www. eso. org/qc : main web site with docu – www. eso. org/HC: Health Check monitor – cal. Checker (calibration completeness) DPD: Data Processing and Quality Control Group [www. eso. org/qc] DMO Meet Yourself day January 9 th, 2009

1 2 3 4 5 6 7 acquisition to processing measuring quality controlling quality trending and feedback science products science deliveries close the loop main workflow: 7 components, to be detailed in the following DPD: Data Processing and Quality Control Group [www. eso. org/qc] DMO Meet Yourself day January 9 th, 2009

Wolfgang’s talk … DPD: Data Processing and Quality Control Group [www. eso. org/qc] DMO Meet Yourself day January 9 th, 2009

• Wolfgang Hummel • VIRCAM: – Commissioning of QC blade cluster – Testing data reduction pipeline, Condor, AB scheduler, DFO infrastructure – multi-detector QC handling (trending, DB, scoring) – ESO Survey Team • HAWK-I: 2009 • Backup for SINFONI and NACO • science: data reduction, calibration, rad. transfer DPD: Data Processing and Quality Control Group [www. eso. org/qc] DMO Meet Yourself day January 9 th, 2009

1 2 3 4 5 6 7 acquisition to processing measuring quality controlling quality trending and feedback science products science deliveries close the loop • 2008: dataflow transmitted via internet (all VLT/VLTI instruments) • survey instruments (VIRCAM) still on disks • incremental processing (as data flow in) • classification, organisation, association (OCA) • cascaded pipeline processing • calibration completeness checker in place DPD: Data Processing and Quality Control Group [www. eso. org/qc] DMO Meet Yourself day January 9 th, 2009

DPD: Data Processing and Quality Control Group [www. eso. org/qc] DMO Meet Yourself day January 9 th, 2009

Sabine’s talk … DPD: Data Processing and Quality Control Group [www. eso. org/qc] DMO Meet Yourself day January 9 th, 2009

1 2 3 4 5 6 7 acquisition to processing measuring quality controlling quality trending and feedback science products science deliveries close the loop Sabine Moehler (FORS 1/2, XShooter, backup MIDI/AMBER) • Quality assessment difficult on raw data § pipeline processing of calibration data • Automatic processing essential § close interaction with DFS colleagues • Quality measurement by pipeline: QC 1 parameter § Spectral resolution, central wavelength, … § Zero points, image quality, … § Lamp intensity, Bias level, read-out-noise, … DPD: Data Processing and Quality Control Group [www. eso. org/qc] DMO Meet Yourself day January 9 th, 2009

1 2 3 4 5 6 7 acquisition to processing measuring quality controlling quality trending and feedback science products science deliveries close the loop • QC 1 Database: store QC 1 parameters § allow to perform individual checks § long-term monitoring (e. g. Seeing Evolution on Paranal, using Image Quality data from FORS 1/2 and ISAAC together with DIMM and Active Optics data (Messenger 132, 11, 2008) DPD: Data Processing and Quality Control Group [www. eso. org/qc] DMO Meet Yourself day January 9 th, 2009

John’s talk … DPD: Data Processing and Quality Control Group [www. eso. org/qc] DMO Meet Yourself day January 9 th, 2009

John Pritchard • QC Scientist for UVES & FLAMES/UVES • QC Scientist for HAWK-I • Backup QC Scientist for ISAAC • Other functions: – pack. LS: Packaging of La Silla Service Mode for EFOSC, EMMI, FEROS, HARPS, Sof. I, Su. SI 2, but soon to disappear completely! DPD: Data Processing and Quality Control Group [www. eso. org/qc] DMO Meet Yourself day January 9 th, 2009

1 2 3 4 5 6 7 acquisition to processing measuring quality controlling quality trending and feedback science products science deliveries close the loop • QC reports – One report per AB, typically a one page, multi-plot graphic showing relevant details of the product(s) of each AB (incl. in Data. Package) • Review – review the QC report and possibly other graphics and/or displays – Does it look ‘normal’? Are the correct number of RAW files included? Unusual structures? Saturation? No light? Star’s in Master Sky. Flats? Pipeline problems/improvements (SDD) • Certification – score. QC: Automatic QC certification based on predefined thresholds for QC 1 parameters. – OK or NOK. If NOK interact with USD in case OB should be repeated, remove products from Calib Cascade, re-reduce dependant Abs after associating to certified Calibs (if necessary), clean QC 1 DB… • MCAL ingestion – Certified CALIB products, logs and plots are then ingested into the ESOArchive DPD: Data Processing and Quality Control Group [www. eso. org/qc] DMO Meet Yourself day January 9 th, 2009

Isabelle’s talk … DPD: Data Processing and Quality Control Group [www. eso. org/qc] DMO Meet Yourself day January 9 th, 2009

1 2 3 4 5 6 7 acquisition to processing measuring quality controlling quality trending and feedback science products science deliveries close the loop AMBER MIDI Isabelle Percheron Main instruments: VLTI/MIDI, VLTI/AMBER Backup: VISIR, CRIRES Coming soon: VLTI/PAOS/PACMAN and VLTI/PRIMA One of PRIMA sub-system: FSU • Providing information on the health of the instrument (mostly through day-time calibrations: daily or non-daily and through night-time calibrations for the interferometric instruments) Ø Customers: Paranal and Scientific community DPD: Data Processing and Quality Control Group [www. eso. org/qc] DMO Meet Yourself day January 9 th, 2009

1 2 3 4 5 6 7 acquisition to processing measuring quality controlling quality trending and feedback science products science deliveries close the loop QC 1 ops. Log FITS files headers qc 1 Parser Pipeline Products QC procedures Location: QC databases (QC 1 database, QC 1 score) Public server QC Garching trend. Plotter database Health Check Plots Paranal DPD: Data Processing and Quality Control Group [www. eso. org/qc] DMO Meet Yourself day January 9 th, 2009

1 2 3 4 5 6 7 acquisition to processing measuring quality controlling quality trending and feedback science products science deliveries close the loop QC 1 database QC 1 score Outliers but older than 6 days trend. Plotter Health Check (HC) Plots DPD: Data Processing and Quality Control Group [www. eso. org/qc] DMO Meet Yourself day January 9 th, 2009

1 2 3 4 5 6 7 acquisition to processing measuring quality controlling quality trending and feedback science products science deliveries close the loop QC databases Outliers within the last 6 data points: detection of potential problems trend. Plotter Trending Plots “long term” HC Plots DPD: Data Processing and Quality Control Group [www. eso. org/qc] DMO Meet Yourself day January 9 th, 2009

Mark’s talk … DPD: Data Processing and Quality Control Group [www. eso. org/qc] DMO Meet Yourself day January 9 th, 2009

Mark Neeser • Instruments supported: SINFONI and Omega. CAM • backup QC for FORS 1 and 2 • member of ESO Survey Team – Preparation and execution of ESO’s public surveys (VST and VISTA for next 5 years). – definition of calibration plans and observing strategies. – monitoring of survey progess. – validating (refereeing) the data products. – conducting science verification for VISTA. DPD: Data Processing and Quality Control Group [www. eso. org/qc] DMO Meet Yourself day January 9 th, 2009

1 2 3 4 5 6 7 acquisition to processing measuring quality controlling quality trending and feedback science products science deliveries close the loop SINFONI: A NIR integral field spectrograph + adaptive optics module science products: • full removal of instrument/sky signature resulting in science-level data for all service mode observations. • final product is a 3 d data cube: x, y (spatial) and z (wavelength) dimensions. • science products (and their associated calibration frames) are QC certified and ingested into the archive. DPD: Data Processing and Quality Control Group [www. eso. org/qc] DMO Meet Yourself day January 9 th, 2009

Omega. CAM: 32 CCD’s ==> 290 Mpixels up to 100 GB/night > 2 x 106 individual CCD images/year VIRCAM: 16 IR detectors ==> 67 Mpixels up to 300 GB/night > 6 x 106 individual detector images/year Solution: Information on demand each QC parameter is scored based on our expertise with the instrument (i. e. a grading of how close a given QC parameter is to its optimum value). DPD: Data Processing and Quality Control Group [www. eso. org/qc] DMO Meet Yourself day January 9 th, 2009

An Example from Omega. CAM A Quality Control (QC) score card per instrument QC score card per detector RAW_TYPE bias flat, dome dark flat, sky std, zeropoint science, object Clicking on any individual detector reveals a detailed analysis DPD: Data Processing and Quality Control Group [www. eso. org/qc] DMO Meet Yourself day January 9 th, 2009

Burkhard’s talk … DPD: Data Processing and Quality Control Group [www. eso. org/qc] DMO Meet Yourself day January 9 th, 2009

Burkhard Wolff • QC for CRIRES, VIMOS • backup for UVES/FLAMES-UVES • maintainer Python QC library DPD: Data Processing and Quality Control Group [www. eso. org/qc] DMO Meet Yourself day January 9 th, 2009

Preparing data packages for PIs 1 2 3 4 5 6 7 acquisition to processing measuring quality controlling quality trending and feedback science products science deliveries close the loop • Incremental packing: after all data from one night have been processed • Data collected: - science: all raw data per run ID plus pipeline products derived from raw data - calibration: products and raw frames needed to process science observations - additional information: ABs, log files, graphical overviews, lists, web pages DPD: Data Processing and Quality Control Group [www. eso. org/qc] DMO Meet Yourself day January 9 th, 2009

Preparing data packages for PIs • Data package structure: • Trigger for package release: - USD sends signal that run is finished - End of period - Operational trigger • Delivery to SAO: package structure and file names (all files incl. products, plots, etc. are available from archive) • Package deleted after SAO has sent DVDs DPD: Data Processing and Quality Control Group [www. eso. org/qc] DMO Meet Yourself day January 9 th, 2009

Danuta’s talk … DPD: Data Processing and Quality Control Group [www. eso. org/qc] DMO Meet Yourself day January 9 th, 2009

• Danuta Dobrzycka • supported instruments: NACO, ISAAC, VISIR • backup support for VIMOS If not for QC VISIR: January 18, 2008 – power outage at Paranal • • May 6, 2008 – high risk earthquake DPD: Data Processing and Quality Control Group [www. eso. org/qc] DMO Meet Yourself day January 9 th, 2009

1 2 3 4 5 6 7 acquisition to processing measuring quality controlling quality trending and feedback science products science deliveries close the loop • close the loop • interaction with PSO: – HC monitor – http: //www. eso. org/qc/ALL/daily_qc 1. html – cal. Checker – http: //www. eso. org/qc/ALL/daily_cal. html – reference frames – http: //www. eso. org/qc/ALL/ref_frames. html • history trending • IOTs • our Paranal visits DPD: Data Processing and Quality Control Group [www. eso. org/qc] DMO Meet Yourself day January 9 th, 2009