Data Management Status Plans Ice Cube SAC Meeting

Data Management Status & Plans Ice. Cube SAC Meeting UW-Madison May 5 -6, 2008 Greg Sullivan University of Maryland

Data Systems Summary • Systems are all functioning well – kept to original MRE budget (=more strings) • MRE work ramping down – EC done, Data Handling nearly complete • Transitioning to M&O – Exp Control (fully) – Data Handling (90% level) – Filtering, software & simulation (~50% level) • Limited resources in M&O, Grants – Data Rate issues • Need to consider optimizations/tradeoffs with limited resources Data Management Status & Plans May 5, 2008

Cost Profile by L 3 (US & Non-US) These Figures are out of date, but illustrate the ramp down in MREFC Data Management Status & Plans May 5, 2008

Ice. Cube Laboratory and Data Center • Commissioned for operation in January 2007. Data Management Status & Plans May 5, 2008

Question? 4. In view of the higher than planned raw data rate and limited satellite bandwidth, and given our available resources for operations, are we on the right track for optimizing data handling; including plans for triggering, filtering, data archiving, data processing and Monte Carlo production? Data Management Status & Plans May 5, 2008

Data Management Status & Plans May 5, 2008

From Detector Status Page Data Management Status & Plans May 5, 2008

Data Management Status & Plans May 5, 2008

Offline Data Processing in North Data Management Status & Plans May 5, 2008

Distributed MC Production ------ monthly summary ------sys_t = 17636203. 1607 ok = 131697. 0 usr_t = 727670104. 216 real_t = 1139116142. 38 suspended = 862. 0 error = 0. 0 events = 198724628. 0 | grid | sys_t | ok | usr_t | real_t | grid_id | suspend | error | events | +---------+---------+---------+---------+-----+ | GLOW | 2. 4 e+06 | 4. 2 e+04 | 3. 2 e+08 | 4. 8 e+08 | 1 | 60 |0 | 6. 2 e+07 | +---------+---------+---------+---------+-----+ | Fear. The | 5. 6 e+05 | 1. 3 e+04 | 6. 4 e+07 | 9. 3 e+07 | 8 | 1. 4 e+02 | 0 | 2 e+07 | +---------+---------+---------+---------+-----+ | desy | 1. 4 e+07 | 5. 9 e+04 | 2. 4 e+08 | 4. 1 e+08 | 9 | 14 |0 | 8. 9 e+07 | +---------+---------+---------+---------+-----+ | npx 2 | 8. 9 e+04 | 1. 9 e+03 | 7. 4 e+06 | 1. 1 e+07 | 14 | 26 |0 | 4. 1 e+06 | +---------+---------+---------+---------+-----+ | ALi. CEne | 8. 3 e+05 | 1. 5 e+04 | 9. 6 e+07 | 1. 4 e+08 | 24 |1 |0 | 2. 3 e+07 | +---------+---------+---------+---------+-----+ | psu_lio | 1. 2 e+04 | 4. 4 e+02 | 1. 5 e+06 | 2. 3 e+06 | 25 |0 | 6. 6 e+05 | +---------+---------+---------+---------+-----+ | katrina | 0 |0 | 26 | 6 e+02 | 0 |0 | +---------+---------+---------+---------+----- Data Management Status & Plans May 5, 2008

Data Systems “Issues” • Data Rates – Filtering is limited satellite bandwidth • Keep improving filters – Raw Data stream Optimization • Data Warehouse – Insufficient M&O funding for first years • Drawing on contingency • Simulation – Distributed Production going well – Additional stress to data warehouse • Running “online filter” on most simulation Data Management Status & Plans May 5, 2008

Data Rates Parenthetical comment on Ghent meeting decision PDD (IC 80) Actual (IC 40) Estimate IC 86 Trigger rate 1 - 1. 5 KHz 1. 4 KHz ? ? RAW DAQ Rate 1 - 1. 5 MB/s 7. 5 MB/s 15 MB/s (? ) ~ 100 GB/day ~ 650 GB 1. 3 TB (? ) Daily Satellite 12 GB 38 GB 60 GB (? ) Filter rejection 8 17 22 Daily Tape Data Management Status & Plans May 5, 2008

Satellite Stream • Filter (satellite) stream – “Main” Physics from filter stream • Previous EB and ICB decisions – Resources to manage more data are difficult – Should be alright with 60 GB/day • As analyses mature – Cut harder at Pole (e. g. L 2 muon filter…) – Perhaps open some space for speculative filters – Definite need to keep working towards showing analysis to high purity “first” » then cut harder on Pn. F by using “final” sample as gauge of filter efficiency Data Management Status & Plans May 5, 2008

Tape Stream • Looking at ~1 TB/day if no changes – Manageable to write this • Lot of tape, but real burden is operational – Threshold high for spinning through all tape • Don’t have resources to go to all tape under “normal” circumstances • So…What is the optimum use of tape stream? (for IC 58 and later) – Full data, high threshold for processing √ICB last week! ? ? – Reduced data, more “usable” Data Management Status & Plans May 5, 2008