D and CDF DetectorsComputing Bill Lee Fermilab DOE

DØ and CDF Detectors/Computing Bill Lee Fermilab DOE Annual Science & Technology Review July 12 -14, 2010

Acknowledgements • My thanks to my CDF colleagues who assisted me with the preparation of this presentation. § § § 2 Massimo Casarsa Phil Schlabach Richard St. Denis Bill Lee, Fermilab - DOE Science & Technology Review July 12 -14, 2010

Brief Outline Overview of Detectors • Current Operations • Computing • Future Operations • 3 Bill Lee, Fermilab - DOE Science & Technology Review July 12 -14, 2010

Fermilab You Are Here 4 Bill Lee, Fermilab - DOE Science & Technology Review July 12 -14, 2010

CDF and DØ Detectors Muon systems EM and Had Calorimeters Solenoid Tracker Silicon Vertex Detector 5 Bill Lee, Fermilab - DOE Science & Technology Review July 12 -14, 2010

CDF and DØ Collaborations Europe North America 32 institutions • Asia 8 institutions The CDF Collaboration § § 6 19 institutions 15 Countries 59 Institutions 538 collaborators 15% FNAL • The DØ Collaboration § § 19 Countries 86 Institutions 492 collaborators 11% FNAL Bill Lee, Fermilab - DOE Science & Technology Review July 12 -14, 2010

DØ Technical Organization (FNAL) Spokespersons Technical Integration Coordinator G. Ginther Triggermeister Data Quality SMT N. Parua S. Youn A. Jonckheere N. Khalatyan Detectors Muon: T. Diehl CFT/PS: M. Corcoran SMT: Z. Ye CTT: M. Corcoran Cal: J. Sekaric, L. Zivkovic L 1 CAL: S. Cihangir Lum: G. Snow Fiber Tracker/ Preshowers J. Warchol —————— Fiber Tracker J. Warchol Preshowers A. Evdokimov Calorimeter D. Schamberger S. Dyshkant (Deputy) —————— L 1 Cal S. Cihangir D. Edmunds ICD L. Sawyer A. White Solenoid H. Fisk 7 Special Projects M. Johnson Run Coordination S. Gruenendahl, W. Lee Electrical Operations: M. Matulik Mechanical Operations: R. Rucinski Luminosity Monitor I. Katsanos M. Prewitt Central Muon A. Ito —————— PDT’s P. Kasper Trigger counters A. Ito Forward Muon V. Evdokimov —————— MDT detectors V. Malyshev MDT Electronics P. Neustroev Pixel detectors S. Kulikov Pixel electronics T. Fitzpatrick Bill Lee, Fermilab - DOE Science & Technology Review July 12 -14, 2010 L 1 CTT S. Gruenendahl L 1 Muon/Cal Track N. Khalatyan L 2 M. Mulhearn L 2 STT D. Boline V. Parihar L 3/DAQ J. Backus. Mayes G. Watts Online W. Lee Controls G. Savage Global Monitoring E. Cheu V. Sirotenko

CDF Operations Organization (FNAL) Detector Operations Admin. Support Associate Head, Shift Operations JJ Schmidt Trigger Dataset Working Group Massimo Casarsa Philip Schlabach Asscociate Head, Online Systems Jonathan Lewis Heather Gerberich Simone Donati Safety Coordinator Dee Hahn Associate Head, Detector Systems Farrukh Azfar Associate Head, Detector Infrastructure Del Allspach - Steve Hahn Operations Managers Process Systems Data Acquisition Bill Badgett Daily/Weekly Ops Shift Crews Sci-Co Aces Co CSL Willis Sakumoto TSI/Fred Jonathan Lewis Silicon Michelle Stancari Sebastian Carron Trigger L 1/L 2 EVB/L 3 Farm Pasha Murat L 3 Filter Farrukh Azfar COT Bob Wagner Aseet Mukherjee Calorimeter/TOF Larry Nodulman Willis Sakumoto Pierluigi Catastini Pedro Fernandez Bill Noe(Leader) Dean Beckner Cutchlow Cahill Jim Humbert Jim Loskot Bruce Vollmer Wayne Waldon Electrical and Mechanical Sys. Admin. / Database Comp. Div. Monitoring/Valid Kaori Maeshima Pasha Murat DQM M. Martinez-Perez Muon Systems Phil Schlabach Giovanni Pauletta CLC Iuri Oksuzian N. Goldschmidt BSC Ken Hatakeyama Jim Lungu Dervin Allen(Leader) John Bell Roberto Davila Jamie Grado (Bldg. Manager) Lew Morris George Wyatt Slow Controls Steve Hahn(Leader) JC Yun 8 Bill Lee, Fermilab - DOE Science & Technology Review July 12 -14, 2010

CDF and DØ Collaboration Scientific Operations/Computing Effort CDF Operations Offline Management Algorithms Total Effort Today 40 FTE 15 FTE 10 FTE 75 FTE DØ 2009 Effort Operations 33 FTE Computing 13 FTE Management 15 FTE Algorithms 27 FTE Total technical contributions 88 FTE Shifts not included in effort Fermilab continues to provide a significant portion of the effort. • The ongoing streamlining of detector operations has resulted in a reduction of effort without negatively impacting performance. • 9 Bill Lee, Fermilab - DOE Science & Technology Review July 12 -14, 2010

CDF OPERATIONS 10 Bill Lee, Fermilab - DOE Science & Technology Review July 12 -14, 2010

CDF Data Taking Efficiency average • Last 12 months: § • Run II Average: § 11 Recorded 83% of delivered luminosity, 79% with the full detector. 83% acquired, 73% good with full detector. Bill Lee, Fermilab - DOE Science & Technology Review July 12 -14, 2010

CDF Data Taking Performance • In the last 12 months (includes the 2009 shutdown) § § § 12 Delivered: Recorded: With full det. : 2. 13 fb-1 1. 77 fb-1 (83%) 1. 68 fb-1 (79%) Bill Lee, Fermilab - DOE Science & Technology Review July 12 -14, 2010

CDF Total Integrated Luminosity in Run II 9. 00 fb-1 7. 48 fb-1 • • 13 Integrated Luminosity with full detector: 6. 55 fb-1 (73%); Depending on run quality requirements, analyses use 6. 3 -7. 2 fb-1. Bill Lee, Fermilab - DOE Science & Technology Review July 12 -14, 2010

CDF Shutdown 2009: Highlights • Silicon Detector: § § • Drift Chamber: § • Cooling system: replaced the elbows on the ISL cooling lines and attached COT face tubing, sealed a few leaks; Power supply maintenance: replaced aged capacitors in 21 CAEN power supply modules. Recovered many channels replacing blown resistors in the wires readout circuit. Calorimeter: § Plug calorimeter sources maintenance. Front end crates preventative maintenance: replaced fan packs and filters, new fuse installation, heat exchanger and drip sensor cleaning. • Replaced 64 nodes of the L 3 farm. • Tied in new diesel emergency generator. • 14 Bill Lee, Fermilab - DOE Science & Technology Review July 12 -14, 2010

CDF Start-up after Shutdowns • Luminosity delivered and CDF data-taking efficiency in the first 35 days after the last three shutdowns: § § § 15 2007: 11 weeks duration; 2008: 1 week duration; 2009: 12 weeks duration. Bill Lee, Fermilab - DOE Science & Technology Review July 12 -14, 2010

Operations Improvements at CDF • Track fitter upgrade (Giga. Fitter) in the level-2 track trigger: § § • Optimization of the trigger bandwidth: § 16 More powerful FPGAs: 1 board in place of 16 boards, more compact, easier maintenance; More memory available: possibility to extend the track acceptance in impact parameter and momentum. Optimized track trigger selection at level-2 to fill up the bandwidth at low luminosity. Bill Lee, Fermilab - DOE Science & Technology Review July 12 -14, 2010

DØ OPERATIONS 17 Bill Lee, Fermilab - DOE Science & Technology Review July 12 -14, 2010

• 18 DØ has recorded >91% of the delivered luminosity over the past year. Bill Lee, Fermilab - DOE Science & Technology Review July 12 -14, 2010

• Over the past 12 months § § 19 Delivered luminosity: Recorded Luminosity: 2. 14 fb-1 1. 98 fb-1 Bill Lee, Fermilab - DOE Science & Technology Review July 12 -14, 2010

DØ Shutdown 2009: Highlights • • Replace scintillator in luminosity monitor. Recover individual silicon HDIs. § • • Largest fraction of functioning channels in DØ history. Liquid nitrogen dewar vacuum leak repair. Routine maintenance and power supply recovery. § Refurbished rack blowers. New Old 20 Bill Lee, Fermilab - DOE Science & Technology Review July 12 -14, 2010

DØ start-up after shutdowns § § § 21 2007: 11 weeks; 2008: 1 week; 2009: 12 weeks. Bill Lee, Fermilab - DOE Science & Technology Review July 12 -14, 2010

Operational Improvements at DØ • • • 22 Reduced the downtime at the beginning and end of stores. New FPGA programming increased the efficiency of the L 1 central track trigger. Updated trigger lists to address higher peak luminosities. Enhanced monitoring. Documentation improvements to facilitate smoother downtime recoveries. Bill Lee, Fermilab - DOE Science & Technology Review July 12 -14, 2010

Accessing the Detectors • On average CDF and/or DØ access their collision halls 1 -2 times per week. § § • 23 Sometimes there is a high priority need to access the hall. o One of the detectors has a data quality problem. Most other accesses are opportunistic. o Tevatron problem or other issue allows access. A few times per year CDF or DØ will need a long access (>6 hours) Bill Lee, Fermilab - DOE Science & Technology Review July 12 -14, 2010 Installation of Run IIb upgrades

Safety at CDF and DØ • Safety remains a central aspect of the collider experiments at Fermilab. § • Shutdown Safety: § § § 24 Over the past year PPD has not had a DART case. Safety is integrated in all aspects of shutdown activities. A Job Hazard Analysis is a vital portion of the planning of any shutdown job. Personnel are reminded to keep safety first. Bill Lee, Fermilab - DOE Science & Technology Review July 12 -14, 2010

DØ COMPUTING 25 Bill Lee, Fermilab - DOE Science & Technology Review July 12 -14, 2010

DØ Data Reconstruction • Plot includes 125 million events that have been processed twice to remove a calorimeter hot cell. • Currently the D 0 farms are processing data within 3 -4 days after recording. • This is our minimum allowed processing delay (to accommodate calibrations. 26 Bill Lee, Fermilab - DOE Science & Technology Review July 12 -14, 2010

DØ Monte Carlo Production DØ uses farms throughout the world to produce Monte Carlo events. IN 2 P 3 is a dedicated D 0 MC site. Other sites are Grid. Over the last year, the total number of generated Monte Carlo has almost doubled to four billion. 27 Bill Lee, Fermilab - DOE Science & Technology Review July 12 -14, 2010

DØ Processing Time Data taken at higher luminosity values takes longer to process. • Higher occupancy/multiplicity. • Average luminosity is not expected to greatly increase. • 28 Bill Lee, Fermilab - DOE Science & Technology Review July 12 -14, 2010

CDF COMPUTING 29 Bill Lee, Fermilab - DOE Science & Technology Review July 12 -14, 2010

CDF Data Collection for the Past Year Data Type Data Volume (TB) # Events (M) # Files Raw Data 306. 2 1892. 2 340487 Production 404. 0 2516. 3 331081 MC 181. 3 893. 9 224156 Stripped-Prd 14. 140 80. 2 11360 Stripped-MC 0 0 0 Ntuple 149. 5 4810. 9 120416 MC Ntuple 116. 8 1905. 8 100308 Total 1172. 0 12099. 3 1127808 1. 9 billion raw events → 2. 5 billion reconstructed events → 4. 8 billion ntuple events. • Additionally 1. 9 billion Monte Carlo ntuple events were produced. • Almost 1. 2 Peta. Bytes of data. • 30 Bill Lee, Fermilab - DOE Science & Technology Review July 12 -14, 2010

CDF Monte Carlo Production • The North American Grid provides Monte Carlo production. § § 31 Steady usage of NAm. Grid. Peaks tend to occur before conference periods. Bill Lee, Fermilab - DOE Science & Technology Review July 12 -14, 2010

CDFGrid usage • CDF Grid provides the full environment for data handling. § • Peaks show over 30 k queued jobs § 32 Provides the majority of computing for analyses. Also conference dependent. Bill Lee, Fermilab - DOE Science & Technology Review July 12 -14, 2010

CDF Open Science Grid Usage The Open Science Grid (OSG) has provided CDF over 40 million hours of computing over the last year. • The use of the OSG has been fruitful and more resources are expected to come online. • 33 Bill Lee, Fermilab - DOE Science & Technology Review July 12 -14, 2010

Common Tools • DØ and CDF use a variety of tools common to both experiments. § § • The FNAL Computing Division provides common system management § 34 Enstore – the underlying data transport mechanism for moving data from online to tape. o The tape silos are a shared responsibility. SAM – a data storage and retrieval tool GRID – the Open Science Grid is used to distribute processing to locations throughout the world. Glidein. WMS – a work management system which eases submission of computing jobs. Oracle services, farm management, desktop support, security, data storage management, and more. Bill Lee, Fermilab - DOE Science & Technology Review July 12 -14, 2010

FUTURE 35 Bill Lee, Fermilab - DOE Science & Technology Review July 12 -14, 2010

DØ 2010 Shutdown plans • This is a 4 week shutdown beginning next week. • Replace luminosity scintillator. § • And ~16 PMTs. Silicon HDI recovery. § And other individual channel recovery. Alignment measurements. • Calibrations. • Trigger framework maintenance. • General Maintenance. • 36 Bill Lee, Fermilab - DOE Science & Technology Review July 12 -14, 2010

CDF Shutdown 2010 Plans • Silicon detector § § § • Drift chamber § § • 37 Cooling system check. Junction cards reseating. Power supply maintenance: replacement of aged capacitors in power supply modules. Replacement of failing resistors in the wire readout circuit. Low voltage short repair. General preventative maintenance. Bill Lee, Fermilab - DOE Science & Technology Review July 12 -14, 2010

Running in FY 2011 • Staffing will continue to need attention. § • Control room shifters, detector experts, on-call personnel, algorithms, support groups and system administrators. o DOE and Fermilab support of visitors and guest scientists for operational/computing continues to be very valuable to the experiments. (Details in Kilminster/Verzocchi talks. ) Tevatron experiments’ computing budgets are 25% below FY 10 and half of the experiments’ request. This will force the experiments to depend upon beyond warranty CPUs and will not support planned increases in data storage capacity and analysis speed. § Anticipated FY 12 computing budget is ~40% lower than FY 10 and below that required to efficiently and reliably support computing and analysis power of the experiments during active stage of Tevatron data analysis. The experiments expect to continue to maintain the high efficiency of the past year. • Keep the delay between acquiring data and its reconstruction to a minimum. • 38 Bill Lee, Fermilab - DOE Science & Technology Review July 12 -14, 2010

Summary • • • Collider experiments are operating smoothly and efficiently. Offline processing is maintaining pace with data collection. Total delivered luminosity should reach 12 fb-1 with around 10 fb-1 recorded by the end of FY 11. § • Challenges maintaining effective Operations: § § • Both experiments continue to improve on their ability to come out of a shutdown efficiently. The DOE's support facilitates our ability to capitalize on these opportunities. § 39 Increasing pressure on available resources o Personnel and computing. Ageing detectors and infrastructure. CDF and D 0 will take advantage of the shutdown to keep up detector maintenance. § • Will require a dedicated effort from the Tevatron and collider experiments. Fermilab has made and continues to make significant contributions to CDF and DØ. Bill Lee, Fermilab - DOE Science & Technology Review July 12 -14, 2010