Web Based Data Quality Monitoring for the Fermi

Web Based Data Quality Monitoring for the Fermi Gamma-Ray Space Telescope Tony Johnson tonyj@slac. stanford. edu

Launched 11 June 2008 – LAT activated 25 June T. Johnson FERMI Data Quality Monitoring CHEP 2010, Taipei, Taiwan 2/19

T. Johnson FERMI Data Quality Monitoring CHEP 2010, Taipei, Taiwan 3/19

In Orbit: Single Events in the LAT The green crosses show the detected positions of the charged particles, the blue lines show the reconstructed track trajectories, and the yellow line shows the candidate gamma-ray estimated direction. The red crosses show the detected energy depositions in the calorimeter. T. Johnson FERMI Data Quality Monitoring CHEP 2010, Taipei, Taiwan 4/19

Fermi One Year All Sky Map T. Johnson FERMI Data Quality Monitoring CHEP 2010, Taipei, Taiwan 5/19

Fermi MISSION ELEMENTS • • GPS msec Large Area Telescope & GBM DELTA 7920 H - • Telemetry 1 kbps • Fermi Spacecraft TDRSS SN S & Ku • • S - • GN • Schedules Mission Operations Center (MOC) GRB Coordinates Network Fermi Science Support Center White Sands HEASARC Schedules Alerts LAT Instrument Science Operations Center (SLAC) GBM Instrument Operations Center Data, Command Loads T. Johnson FERMI Data Quality Monitoring CHEP 2010, Taipei, Taiwan 6/19

Data Processing Flow at SLAC • • • Downlink from Goddard Space Flight Center – ~8 downloads per day – 15 GB total daily Level 0 Processing – Automatically launched as data arrives – Decode & repackage incoming data • Split science data from telemetry data Level 1 Processing – Full event reconstruction: 750 GB/day – Data Quality Monitoring – Transfer science summary files to Goddard Science Support Center - 200 MB/day • Immediately available to the public ASP (Automated Science Processing) – GRB and Flare detection – Spectral analysis 120, 000 quantities to be monitored – Mixture of Oracle, Root, Fits data T. Johnson FERMI Data Quality Monitoring Level 0 Trending Database (Oracle) Level 1 Data Quality (Root) ASP Results (Fits) CHEP 2010, Taipei, Taiwan 7/19

Level 1 Processing Task Example Digitization Reconstruction • • • ~2000 batch jobs every 3 hours – 800 simultaneous cores All batch automatically submitted using “Data Pipeline” • Very low rate of manual intervention required (<. 01% of jobs) Data verified and available to public (much) less than 24 hours after acquisition T. Johnson FERMI Data Quality Monitoring CHEP 2010, Taipei, Taiwan 8/19

ISOC Control Room • • “Duty Scientists” monitoring data quality daily All of the data processing and data quality monitoring can be done from the web T. Johnson FERMI Data Quality Monitoring CHEP 2010, Taipei, Taiwan 9/19

• • Technologies Used Web: – Many independent web applications • Allows independent development • Shared application framework provides – authentication, authorization » CAS single sign-on – page decoration, sitewide menus – Database utilities – Apache/Tomcat servers • Multiple servers for redundancy • Monitored using Nagios, JMX – Java Server Pages (JSP) • Open Source and Custom tag libraries simplify development – Display. Tag for tabular data » Sorting, filtering, pagination – JAIDA tag library for plotting » Images generated dynamically on server Data Access Tools – Oracle, Partitioning – Java Fits Library – Free. HEP Root IO library T. Johnson FERMI Data Quality Monitoring CHEP 2010, Taipei, Taiwan 10/19

Monitoring Data Processing • Web interface allows – Quick overview of data processing – Flags runs requiring further attention – Allows “drill-down” to isolate/identify problems T. Johnson FERMI Data Quality Monitoring CHEP 2010, Taipei, Taiwan 11/19

Processing Pipeline Web Interface • T. Johnson Pipeline web interface allows – Many views of data processing, down to log files of individual jobs – If jobs do fail they can be “rolled back” directly from the web interface FERMI Data Quality Monitoring CHEP 2010, Taipei, Taiwan 12/19

Data Quality Monitoring • Web interface allows – Show data from single run or aggregate set of runs – View description of each plot – View/Print multiple plots – Customized tree to draw attention to important plots • Can be customized for individuals or groups T. Johnson FERMI Data Quality Monitoring CHEP 2010, Taipei, Taiwan 13/19

Telemetry Trending • • Web interface allows – Dynamic selection of time period – Dynamic overlay of quantities – Customized tree to draw attention to important plots • Can be customized for individuals or groups Cross trending of housekeeping and level 1 data T. Johnson FERMI Data Quality Monitoring CHEP 2010, Taipei, Taiwan 14/19

Automated Alarms Error limit Warning limit • Automated alarms are used to alert duty scientists to anomalies • Use fixed limits and reference histograms • Many quantities are highly orbit dependent, so particle fluxes, geomagnetic variables must be taken into account – 20 different alarm algorithms T. Johnson FERMI Data Quality Monitoring CHEP 2010, Taipei, Taiwan 15/19

Data Quality Monitoring • The trending graphs below show some rate summary plots for the 24 hours around GRB. • Strong correlation with orbital period (~90 minutes) can clearly be seen • This burst was so bright that it can be seen even in the global rate plots. T. Johnson FERMI Data Quality Monitoring CHEP 2010, Taipei, Taiwan 16/19

Automated Science Processing • Used to rapidly detect Gamma Ray Bursts or other flaring events • Enabled timely notification of interesting events to external astrophysical community T. Johnson FERMI Data Quality Monitoring CHEP 2010, Taipei, Taiwan 17/19

Technology reuse • New Scientific Computing Applications (SCA) group formed at SLAC – Part of Particle and Particle Astrophysics (PPA) Division – Aims to encourage reuse of software between experiments • Much of the Fermi monitoring web framework being reused for Enriched Xenon Observatory (EXO) experiment • Starting use for Large Synoptic Survey Telescope (LSST) • Others… • Other related talks at CHEP 2010 – 21 October: 14: 30~16: 00 parallel 47 -- Building Interactive Web Applications for HEP Using the Google Web Toolkit (GWT) – 21 October: 16: 30~18: 00 parallel 48 -- Fermi Gamma-Ray Space Telescope Processing Pipeline and Data Catalog T. Johnson FERMI Data Quality Monitoring CHEP 2010, Taipei, Taiwan 18/19

Conclusions • The Fermi Data Quality Monitoring system has now been used successfully for >2 years – Much of it is publically viewable at: • http: //glast-ground. slac. stanford. edu/ • Acknowledgments – The LAT Data Quality Monitoring system was designed and implemented by: Anders Borgland, Eric Charles, Warren Focke, Martin Kocian, Maria Elena Monzani, David Paneque, Massimiliano Turri (SLAC), Luca Baldini, Johan Bregeon, Melissa Pesce. Rollins and Carmelo Sgrò (INFN and University of Pisa). – Maria Elena Monzani provided many of the plots used in this presentation. T. Johnson FERMI Data Quality Monitoring CHEP 2010, Taipei, Taiwan 19/19