LHC CMS Detector Upgrade Project B 04 3
LHC CMS Detector Upgrade Project B 04 -3 Calorimeter Trigger Sridhara Dasu, University of Wisconsin DOE CD 1 Review 26 August 2013 Sridhara Dasu, 26 August 2013 DOE CD 1 Review - US CMS Upgrade - Trigger 1
LHC CMS Detector Upgrade Project The Grand Plan for Trigger Upgrade LHC Calendar § 2012: Discovered the Higgs § 2013 -2014: Make upgrades to handle 14 Te. V + 2 x 1034 cm– 2 s– 1 § 2015 -2017: Discover new high energy physics phenomena using data at 14 Te. V + study the Higgs + EWSB mechanism § 2018 -2019: Make upgrades to handle 14 Te. V + 5 x 1034 cm– 2 s– 1 § 2020 -2022: Program is dependent on what we find by 2018 CMS Plans § § 2013: Finalize trigger designs and prototyping 2014: Construct the trigger modules 2015: Early operation with pre-production modules (Stage-1) 2016 -2017: Operations with fully upgraded trigger (Stage-2) Rapid plan to ensure success in 2015 -2017 run! § Plans backed up by thorough prototyping that is already underway Sridhara Dasu, 26 August 2013 DOE CD 1 Review - US CMS Upgrade - Trigger 2
LHC CMS Detector Upgrade Project Technical Opportunity FPGA & Telecom technology advances § Ubiquitous 10 Gbps links § FPGAs with 72 rx/tx links with built in serdes o Xilinx Vertex-7 platform o Gigantic cores o DSP units, BRAM and Embedded processors § Fiber optic components o Avago mini-pods with 12 x § Ripe time to upgrade from VME to u. TCA LHC long shutdown 2013 -2014 § Sufficient time for copper optical transition for inputs § Ensures that upgraded electronics is ready for operation in 2016 § Partial upgrade operation from 2015 itself Sridhara Dasu, 26 August 2013 DOE CD 1 Review - US CMS Upgrade - Trigger 3
LHC CMS Detector Upgrade Project Calorimeter Trigger Layers 1 & 2 US: UK: Vienna: Sridhara Dasu, 26 August 2013 DOE CD 1 Review - US CMS Upgrade - Trigger 4
LHC CMS Detector Upgrade Project Algorithms Algorithm layers § First layer builds local clusters of energy deposits o Determines energy and cluster type (EM or Hadronic) § Second layer builds triggerable objects from clusters o Electron/Photon, Tau, Jet, MET, … Mapping to architectures § Traditional architecture o Algorithm layer-1 maps to physical layer-1 electronics – Key feature: nearest-neighbor sharing – Sufficient to build clusters locally o Algorithm layer-2 maps to physical layer-2 electronics – All clusters available in one FPGA § Time multiplexing architecture o All primitive data passes through one FPGA o Potential for clever on-the-fly processing, if feasible/necessary Sridhara Dasu, 26 August 2013 DOE CD 1 Review - US CMS Upgrade - Trigger 5
LHC CMS Detector Upgrade Project Current & Upgraded Calo Trigger HCAL Energy HTR ECAL Energy TCC o. SLB HF Energy m. HTR HCAL Energy m. HTR o. RM US Regional Calo Trigger EM candidates UK CTP Layer 2 Calo Trigger MP 7 Region energies Global Calo Trigger Sridhara Dasu, 26 August 2013 o. RSC US Layer 1 Calo Trigger UK DOE CD 1 Review - US CMS Upgrade - Trigger Upgrade L 1 Trigger System Risk mitigation by operating current and adiabatic stages of upgrade CT concurrently Current L 1 Trigger System Copper Optical HCAL Optical Splitters 6
LHC CMS Detector Upgrade Project Calorimeter Trigger Components Type Name Purpose TCC Trigger Concentrator Card Existing ECAL trigger primitives (TP) (France) o. SLB Optical Serial Link Board Transmit ECAL TPs to RCT & CTP (Portugal / France) m. HTR HCAL Trigger & DAQ Board Create and transmit HCAL TPs RCT Regional Calorimeter Trigger Existing trigger system o. RM Optical Receiver Modules Receive ECAL TPs on RCT (Portugal / France for Stage-1) o. RSC Optical Regional Summary Card Convert & transmit RCT output (US DOE NP for Stage-1) CTP Compact Trigger Processor Large FPGA w/ optical & backplane (This Project) CIOX Compact IO cross-point switch Intra-crate sharing switch (This Project) MP 7 Multipurpose Processor Large FPGA w/ only optical links (UK) Sridhara Dasu, 26 August 2013 DOE CD 1 Review - US CMS Upgrade - Trigger 7
LHC CMS Detector Upgrade Project Evolution: Stage-1 Upgrade in 2015 Uses reprogrammed RCT clusters with improved algorithms in Layer-2 Also, brings in all the power of finer grain HF using “Slice Test” of Layer-1 & 2 Improved PU subtraction, Isolation Calculation (e/γ/τ/μ) & Half-tower Position Resolution US Items: o. RSCs, o. SLB/o. RM commissioning CTP Layer 1 Processors RCT RO+Testing CTP HCAL energy 36 m. HTRs Regional Calo Trigger o. RSC UK Items: MP 7 Layer 2 Processors HF energy ECAL energy Layer 1 Processors Add to bring in finer grain EM clusters 2 CTP 7 s 9 CTP 7 s o. RM ECAL & HF Clusters Half-tower position o. SLB Global Calo Trigger 4 x 4 E+H Clusters 2 x 1 E+H Clusters 2 -bit region ID RCT RO + Testing 2 CTP 7 s Layer 22 Spare Calo Trigger Heavy Ion Muon 4 MP 7 s Jets & Sums e. Gamma. Tau GCT fallback remains Layer 2 Processors With just a fraction of final cards or even prototypes, derive many benefits of full upgrade, incl. muon isolation Sridhara Dasu, 26 August 2013 DOE CD 1 Review - US CMS Upgrade - Trigger 8
LHC CMS Detector Upgrade Project Algorithm Studies Stage-1 uses clusters from RCT+HF processed by new m. TCA cards § Adiabatic improvements to calorimeter trigger dataflow § Algorithm stage takes advantage of larger FPGA resources § Performance of the “revised” algorithms is very good o e. Gamma. Tau studied by FNAL/Wisconsin o e. Gamma and muon isolation studied by UCSD o Muon isolation and b-jet tagging with muons studied by Ohio o Heavy Ion jet cleanup studied by MIT o Improved jet finding with PU subtraction by UIC § Further improvements using full ECAL granularity in Stage-1 o Improved e. Gamma. Tau isolation & position (FNAL, Wisconsin) o Pile-up handling with lower LSB for ECAL (FNAL, Wisconsin) o Improved jet position and PU subtraction (Davis) Sridhara Dasu, 26 August 2013 DOE CD 1 Review - US CMS Upgrade - Trigger 9
LHC CMS Detector Upgrade Project Electron / Photon Trigger Improved control on isolation § Factor of 2 -3 reduction in rate with small loss in efficiency Sridhara Dasu, 26 August 2013 DOE CD 1 Review - US CMS Upgrade - Trigger 10
LHC CMS Detector Upgrade Project Tau Trigger Big improvement in efficiency with ~10 X rate reduction! Current tau trigger has large and negative PU dependence Sridhara Dasu, 26 August 2013 DOE CD 1 Review - US CMS Upgrade - Trigger 11
LHC CMS Detector Upgrade Project Jet Triggers Single jet thresholds similar to current § Multi-jet trigger thresholds better in upgrade (PU sub. ) § Small improvement in efficiency turn-on HT = S PTJets Sridhara Dasu, 26 August 2013 Single jet DOE CD 1 Review - US CMS Upgrade - Trigger Quad jet 12
Physics Performance Summary 1. 1 x 1034 cm– 2 s– 1 2. 2 x 1034 cm– 2 s– 1 Current Upgrade W(en), H(bb) 57. 7% 87. 0% 37. 5% 71. 5% W(mn), H(bb) 95. 9% 100% 69. 6% 97. 9% VBF H(tt(mt)) 42. 6% 51. 3% 19. 4% 48. 4% VBF H(tt(et)) 24. 4% 44. 3% 14. 0% 39. 0% VBF H(tt(tt)) 17. 2% 53. 7% 14. 9% 50. 1% H(WW(eenn)) 91. 4% 97. 8% 74. 2% 95. 3% H(WW(mmnn)) 99. 9% 89. 3% 99. 9% H(WW(emnn)) 97. 6% 99. 4% 86. 9% 99. 3% H(WW(menn)) 99. 6% 99. 5% 90. 7% 99. 7% Stop b. Wc e, jets (600 – 450 Ge. V) 55. 8% 68. 2% 50. 3% 64. 8% Stop b. Wc m, jets (600 – 450 Ge. V) 78. 1% 81. 6% 76. 4% 84. 5% RPV Stop jets (200 Ge. V) 70. 1% 99. 9% 43. 6% 99. 9% RPV Stop jets (300 Ge. V) 93. 7% 99. 9% 79. 7% 99. 9% Process (x 2 improvement highlighted) Sridhara Dasu, 26 August 2013 DOE CD 1 Review - US CMS Upgrade - Trigger Average Improvement: 17% (Low Lumi) & 40% (High Lumi) LHC CMS Detector Upgrade Project 13
LHC CMS Detector Upgrade Project Technical Feasibility § Prototypes for all major cards exist & have passed tests § Upgrade is based upon common μTCA hardware platform also used by HCAL & other CMS systems § Upgrade trigger is replacing existing trigger hardware built by the same team → interfaces well understood § Same team wrote software & firmware for existing trigger system → scope & requirements well understood § Requirements are frozen in approved Trigger TDR Sridhara Dasu, 26 August 2013 DOE CD 1 Review - US CMS Upgrade - Trigger 14
LHC CMS Detector Upgrade Project Calo Trigger Team Experience § Wesley Smith § CMS Trigger Project Manager 1994 -2007, § Trigger Coordinator 2007 – 2012 § US CMS L 2 Trigger Project Manager 1998 – present § Sridhara Dasu § US CMS L 3 Manager for Calorimeter Trigger, 1998 – present § Author of original & upgrade cal. trig. Algorithms 1994 – present § Pam Klabbers – Operations & Testing § CMS RCT Operations Manager (more than a decade on RCT project) § CMS Deputy Trigger Technical Coordinator § Tom Gorski – Hardware & Firmware § Over a decade of engineering on the CMS Calorimeter Trigger § Mathias Blake – Firmware § New to project -- 5 years Applications Engineer at Xilinx, 2 years FPGA Engineer at DRS Technologies, 3 years FPGA Design Engineer at Dolby Labs § Jes Tikalsky and Evan Friis – Software § Providing prototype testing software (Friis has experience with current RCT) Sridhara Dasu, 26 August 2013 DOE CD 1 Review - US CMS Upgrade - Trigger 15
LHC CMS Detector Upgrade Project Calorimeter Trigger WBS Detail ← Muon Trigger M&S paid by CMS-France Sridhara Dasu, 26 August 2013 DOE CD 1 Review - US CMS Upgrade - Trigger 16
LHC CMS Detector Upgrade Project HCAL & ECAL Trigger Primitives § Upgraded HCAL Readout & Trigger Electronics (μHTR) is after split that sends data to old electronics (HTR) § μHTR supplies TP to upgrade trigger, HTR continues to send to present trigger § ECAL Trigger Concentrator Cards (TCC) are not upgraded, instead their mezzanines with trigger serial links are replaced (Lisbon): § Optical Serial Link Board o. SLB replaces single copper link to current Regional Calorimeter Trigger (RCT – U. Wisc. ) with two optical links (WBS 401. 04. 06) § One optical link to optical Receiver Module (o. RM) on RCT (WBS 401. 04. 05) o Replaces copper Receiver Module § One optical link to input of Upgraded Calorimeter Trigger: CTP 7 Card § 574 o. SLB’s & o. RM’s with Fibers installed in 2014 § French Contribution to Calorimeter Trigger (no US Upgrade Funds) OSLB Output / o. RM Input: § Presently o. SLB & o. RM in final prototype stage 4 x channels @ 1. 2 Gbps § o. SLB: o. RM: 4. 8 Gbps Sridhara Dasu, 26 August 2013 DOE CD 1 Review - US CMS Upgrade - Trigger 17
3. 3 V Supply § Virtex-7 690 T for processing § ZYNQ for TCP/IP + linux § 60 10 G optical Input links § 36 10 G optical Output links § Function: Find ET clusters & transmit to Layer-2 2. 5 V Supply § Main processor card 1 V 30 A Supply 401. 04. 02: CTP Card Concept 1. 5 V Supply LHC CMS Detector Upgrade Project CXP Module 12 Tx + 12 Rx ZYNQ XC 7 Z 030 EPP Virtex-7 VX 690 T FPGA • Card Count = 46 • 36 total + 8 spares + 2 test setups Sridhara Dasu, 26 August 2013 CXP Module 12 Tx + 12 Rx (CTP-6 BG View) DOE CD 1 Review - US CMS Upgrade - Trigger 12 X Rx CXP Module 12 Tx + 12 Rx 18
LHC CMS Detector Upgrade Project CTP: Virtex-6 Prototype Exists Prototype with two V 6 chips useful for most studies – limited to 6. 4 Gbps IO & fewer (48) links Final V 7 version offers more IO links and at higher 10 Gbps speed and larger logic MMC Circuitry Power Modules Back End FPGA XC 6 VHX 250 T/ XC 6 VHX 380 T 12 x Multi Gig Backplane Connections Dual SDRAM for dedicated DAQ and TCP/IP buffering Sridhara Dasu, 26 August 2013 JTAG/USB Console Interface Mezzanine 4 X Avago AFBR-820 B Rx Module Front End FPGA XC 6 VHX 250 T/ XC 6 VHX 380 T DOE CD 1 Review - US CMS Upgrade - Trigger Avago AFBR 810 B Tx Module 19
LHC CMS Detector Upgrade Project CTP: Basis of Estimate - Hardware § CTP 7 is simpler than CTP 6 prototype § Engineering effort for in-house design based on CTP 6 work § Procurement of PCBs is based on recent CTP 6 purchase § Good estimate of cost and schedule from prototype manufacture § Quotes available for all parts § Virtex-7 690 T is the major cost driver – with recent quote o 3 690 Ts donated by Xilinx o 5 690 Ts purchased from Xilinx o Total of 8 FPGAs in-hand for prototypes § Optical modules based on recent purchase § Miscellaneous parts also have quotes or estimated from prototype purchases (power modules) Sridhara Dasu, 26 August 2013 DOE CD 1 Review - US CMS Upgrade - Trigger 20
LHC CMS Detector Upgrade Project CTP: Basis of Estimate - Effort § Firmware and software efforts estimated from prototype testing + prior RCT experience § Firmware algorithms for testing provide the basis for the core firmware, which is the most complicated o Tom Gorski and Mathias Blake § Trigger algorithm firmware work is based on prior experience managing GCT project (UK) o Wesley Smith was L 1 project manager overseeing GCT § Software effort is based on current RCT o Sophistication of embedded Linux / TCP and shorter timeline for implementation requires professional software effort – Jes Tikalsky o Bulk of testing software from post-doc and graduate student – Pam Klabbers, Evan Friis and graduate students will develop software § Board and Firmware/Software done by the same group (Wisconsin) results in efficiencies Sridhara Dasu, 26 August 2013 DOE CD 1 Review - US CMS Upgrade - Trigger 21
LHC CMS Detector Upgrade Project 401. 04. 03: Crosspoint IO Card – U. Wisconsin § Function: For inter-board connections (2/crate x 3 crates = 6 + spares) § Input/Output: 48 x 4. 8 -6. 4 Gbps Controller (MMC and link mgmt) § Fully tested § Firmware is minimal and exists § CIOX cost well known § Ready for production Sridhara Dasu, 26 August 2013 4 x 4 Lane Bidirectional Multi Gig Backplane Connections § Prototype exists Backplane Rx/Tx Redriver ICs (top and bottom sides) DOE CD 1 Review - US CMS Upgrade - Trigger 4 X Avago AFBR-79 EQDZ QSFP+ Module Positions 22
LHC CMS Detector Upgrade Project 401. 04. 04: CTP Infrastructure: Vadatech VT 894 Crate Test Setup (Final system: 3 crates w/ 12 CTP 7 ea. + 2 test setups + spare = 6) Sridhara Dasu, 26 August 2013 DOE CD 1 Review - US CMS Upgrade - Trigger UW CTP-6 BU AMC 13 Vadatech MCH UW Aux U. Wisconsin designed backplane with dense card interconnects manufactured & installed in commercial Vadatech VT 892 Crate available in Vadatech Catalog UW CTP-6 TTC Downlink 23
LHC CMS Detector Upgrade Project CTP: Risk and Mitigation Strategy § The dominant risks are: § Availability of ALL trigger primitive inputs from ECAL, HCAL in optical format § Validation of large optical link plant in limited time § Fully validated trigger algorithms in firmware § Control and operations software § Risk mitigation strategy: § Continue to provide fully operational current trigger system in parallel with upgrade commissioning § Partial operation of the upgrade systems provide tangible benefits from 2015 onwards § Firmware or Software issues are mitigated by provisioning qualified additional temporary personnel o Ability to hire short term qualified staff at UW / FNAL / Elsewhere Sridhara Dasu, 26 August 2013 DOE CD 1 Review - US CMS Upgrade - Trigger 24
LHC CMS Detector Upgrade Project Calorimeter Trigger Risk Mitigation § Goal (Original): § Provide readout of original RCT descoped during construction project § Present readout through GCT input buffer not workable with trigger evolution § Uses connection to a single CTP 6 prototype or CTP 7 card for DAQ readout § optical Receiver Summary Card (o. RSC) paid by DOE Nuclear § § § § VME Slave Interface Card Fits in current RCT Crates (1 per crate) 18 Cards in 2015 System Receives RCT Jet Sum Card Output to GCT on Copper “SCSI” Data Cables Provides direct optical input to GCT o Bypassing old optical conversion cards Planned use for Heavy Ion Triggers Prototype under test o. RSC Function: Conversion to optical o. RSC Input: RCT egt and jet clusters 80 MHz Parallel ECL o. RSC Output: 6 x copies at 6. 4 Gbps optical § Multiple optical outputs provide: § Inputs for upgrade calorimeter trigger & parallel operation of old & new trigger wherever ECAL & HCAL electronics not available in 2015 Sridhara Dasu, 26 August 2013 DOE CD 1 Review - US CMS Upgrade - Trigger 25
LHC CMS Detector Upgrade Project Prototypes Status (Optical Links) CTP 6 to CTP 6 (48 x 6. 4 Gbps links) o. RSC to CTP 6 (11 x 6. 4 Gbps links) § Input/Output verified – low bit error rate – clear eyes o. RSC to MP 7 (4 x 6. 4 Gbps links) Sridhara Dasu, 26 August 2013 DOE CD 1 Review - US CMS Upgrade - Trigger 26
LHC CMS Detector Upgrade Project HCAL TPG to CTP Integration Test § 12 -links tested at 6. 4 Gbps – no errors Sridhara Dasu, 26 August 2013 DOE CD 1 Review - US CMS Upgrade - Trigger 27
LHC CMS Detector Upgrade Project Running Linux on CTP 6 FPGA – Peta. Linux § Startup screen via USB console § Root login, pinging controller PC at 192. 168. 1. 2 § Running on CTP 6 back end FPGA (`VHX 250 T) § Important step towards CTP-7 ZYNQ-based Linux Sridhara Dasu, 26 August 2013 DOE CD 1 Review - US CMS Upgrade - Trigger 28
LHC CMS Detector Upgrade Project Demonstrators & Critical Integration Tests § m. HTR to CTP 6 integration test at Madison (done) § Wisconsin/Minnesota : 12 links operated successfully § o. RSC to CTP 6 demonstrator at Madison (done) § Test 2 -3 o. RSCs and interface before shipment to CERN § o. RSC to CTP 6 platform at Prevessin (done) § 2 o. RSCs emulate RCT + 1 CTP 6 for RCT readout and algorithm tests § Full platform for system operations and control software development § o. RSC - MP 7 Integration test at Prevessin (done) § Wisconsin/Imperial July 2013 Milestone § CTP 6 - o. RSC-o. RM/RCT - o. SLB/TCC Integration Test at Prevessin § Wisconsin/Lisbon/LLR October 2013 Milestone § CTP 6 - o. RSC-o. RM/RCT - o. SLB/TCC -u. HTR Integration Test at 904 § Wisconsin/Lisbon/LLR/Minnesota November 2013 Milestone § CTP 7 - MP 7 Integration test at Prevessin § Wisconsin/Imperial March 2014 Milestone Sridhara Dasu, 26 August 2013 DOE CD 1 Review - US CMS Upgrade - Trigger 29
LHC CMS Detector Upgrade Project Adiabatic Upgrade Timeline § 2013 § § M&O + R&D Activity : o. RSC construction, installation Firmware development for RCT readout and testing using CTP 6 Lisbon / France : o. SLB and o. RM construction, installation Minnesota / Wisconsin : m. HTR to CTP 6 testing § 2014 § Ensure fully working legacy RCT system with o. RSC + CTP 6 RO § CTP 7, CIOX, … construction, validation § Firmware development for CTP 7 testing and 2015 operation § 2015 § Commission and operate Stage-1 system § Firmware development and integration of Stage-2 system § 2016 § Commission and operate Stage-2 system Sridhara Dasu, 26 August 2013 DOE CD 1 Review - US CMS Upgrade - Trigger 30
LHC CMS Detector Upgrade Project CT Milestones Activity ID Activity Name Target Completion Milestone T 1020 L 2 – Deliver Stage-1 Layer 1 to CERN 15 -Nov-2014 15 -May-2015 T 1030 L 2 – Deliver Stage-2 Layer 1 to CERN 10 -Mar-2015 10 -Sep-2015 T 1040 L 2 – Test operation at CERN 15 -Aug-2015 15 -Feb-2016 T 1070 L 2 – Test High-Luminosity Capability 28 -Sep-2016 28 -Mar-2017 T 3310 L 3 – Commission CTP 7 for Initial Ops 15 -Apr-2015 15 -Jul-2015 T 4000 L 3 – Commission CIOX for Final Ops 04 -Sep-2016 04 -Dec-2016 T 3320 L 3 – Commission CTP 7 for Final Ops 29 -Sep-2016 29 -Dec-2016 T 4740 L 4 – o. SLB-o. RM Optical Fibers Complete 15 -Sep-2014 T 4090 L 4 – CIOX Software and Firmware Complete 10 -Dec-2014 T 4210 L 4 – CIOX Production Complete 08 -Jan-2015 T 3710 L 4 – CTP 7 Production Complete 08 -Jan-2015 T 4510 L 4 – CTP Infrastructure Production Complete 29 -Jul-2015 T 4300 L 4 – CIOX Commissioning Complete 09 -Sep-2015 T 3980 L 4 – CTP 7 Test, Install and Commissioning Complete 30 -Sep-2016 T 3520 L 4 – CTP 7 Software and Firmware Complete 30 -Sep-2016 Sridhara Dasu, 26 August 2013 DOE CD 1 Review - US CMS Upgrade - Trigger 31
LHC CMS Detector Upgrade Project Conclusion § Calorimeter Trigger Prototypes are in good state § Both CTP 6 and CIOX validated § CTP 7 is in design stage and is based on CTP 6 experience § Crate infrastructure is also ready to go § Calorimeter Trigger Cost Basis is solid § Hardware costs based on purchase orders or firm quotes § Firmware and Software estimates are based on prior experience o Experienced team with risk mitigation by ability to add people § Calorimeter Trigger Risk Analysis § Cost and schedule risks with mitigation plans are presented § Calorimeter Trigger Team ready to get started § CD 1 approval will be much appreciated! Sridhara Dasu, 26 August 2013 DOE CD 1 Review - US CMS Upgrade - Trigger 32
- Slides: 32