LHC CMS Detector Upgrade Project B 04 2

LHC CMS Detector Upgrade Project B 04 -2: Stage 2 Calorimeter Trigger Wesley Smith, U. Wisconsin L 2 Manager, WBS 401. 4 October 14, 2015 Wesley Smith, 14 October 2015 DOE Review -- B 04 -2: Stage 2 Calorimeter Trigger 1

LHC CMS Detector Upgrade Project Calorimeter Trigger WBS Detail ← Muon Trigger Institutions: UI – Chicago Fermilab MIT Rice Wisconsin Wesley Smith, 14 October 2015 M&S paid by CMS-France DOE Review -- P 04: Trigger Upgrade 2

Calo. Trigger Upgrade in Parallel: Split inputs from ECAL and HCAL Current L 1 Trigger System 2015: ECAL, Legacy RCT & associated Cal. Trigger Cards* - STAGE 1 ECAL Energy (TCC) HCAL Energy (HTR) US HCAL Optical Splitters HF Energy (μHTR) o. SLB Regional Calo Trigger o. RM o. RSC EM candidates UK Region energies Global Calo Trigger 2016: HCAL u. HTR & assoc. Cal. Trigger Cards - STAGE 2 HCAL Energy (μHTR) Layer 1 Calo Trigger US Layer 2 Calo Trigger UK Entire Summary Upgrade L 1 Trigger System LHC CMS Detector Upgrade Project § ECAL: optical Serial Link Board (OSLB) and optical Receiver Mezzanines (o. RM) connect to Present and Upgrade Calorimeter Trigger o All installed and being commissioned in CMS Global Runs § HCAL: optical splitters drive both HTRs and μHTRs o All u. HTRs installed and driving Upgrade L 1 T (+ Current HF RCT via o. RMs) Wesley Smith, 14 October 2015 DOE Review -- P 04: Trigger Upgrade 3

LHC CMS Detector Upgrade Project Stage-2 Upgrade in 2016 Layer-1 CTP 0 CTP 1 CTP 2 … CTP 17 USCMS 3 m. TCA Crates 18 CTP 7 s Each covering all h DAQ via AMC 13 s Layer-2 MP 7 Algorithm Card(s) (9+1) Imperial HW + FW/SW USCMS SW Full flexibility for traditional or time-multiplexed architecture Highest precision location of objects with dynamic clustering m. TCA Global Trigger DAQ via AMC 13 Wesley Smith, 14 October 2015 DOE Review -- B 04 -2: Stage 2 Calorimeter Trigger 4

LHC CMS Detector Upgrade Project CTP 7 Card Status – U. Wisconsin § CTP 7 Design § § Virtex-7 690 T main data processing FPGA ZYNQ So. C FPGA with dual ARM Cortex-A 9 CPU Embedded Linux Operating System running on the ZYNQ 81 RX and 62 TX Multi. Gigabit I/O links, multi-rate, LHCsynchronous or asynchronous link operation § True Triple-Link-Rate, Multi-Clock-Domain Card o 6. 4 and 4. 8 Sync Rx, 10 G Async Tx in same MGT quad § Heavy duty power and cooling infrastructure § First two units delivered in Dec, 2013 § Excellent results in checkout § Full Production Complete Mar ‘ 15, 2015 § Excellent checkout results continued o 6 Preproduction, 50 Production CTP 7 s complete o 38 total CTP 7 s shipped to CERN Wesley Smith, 14 October 2015 DOE Review -- B 04 -2: Stage 2 Calorimeter Trigger 5

LHC CMS Detector Upgrade Project CTP 7 Card Details CTP 7 Features • Virtex-7 690 T for processing ZYNQ for TCP/IP + Linux AXI Chip 2 Chip conects V 7 to ZYNQ 67 10 Gbps optical Input links 48 10 Gbps optical Output links • • Stage-1: Read Out • Receives data from 18 o. RSCs Formats & provides data on L 1 trigger Input: 2 fibers each @ 10 Gbps Output: to AMC 13 • • • Minipods: 31 Rx + 12 Tx CXP Module 12 Tx + 12 Rx Virtex-7 VX 690 T FPGA CXP Module 12 Tx + 12 Rx ZYNQ XC 7 Z 030 EPP Stage-2: Layer-1 • • Receives data from ECAL/o. SLB and HCAL/u. HTR Initial processing and TMT mulitplexing and send to Layer-2 for processing + TPG RO Wesley Smith, 14 October 2015 DOE Review -- B 04 -2: Stage 2 Calorimeter Trigger 6

LHC CMS Detector Upgrade Project CTP 7 Functions § Combines 3 different calorimeter subsystems: ECAL, HCAL, and HF § Align and decode input data from the calorimeters § Apply tower-level calibration in lookup tables § Build combined trigger tower words, streaming them to Layer-2 § Event-by-event DAQ readout for data quality monitoring § Captures inputs and outputs up to +/- 2 BX at reduced rate by means of Remote Procedure Call service running on the ZYNQ Wesley Smith, 14 October 2015 DOE Review -- B 04 -2: Stage 2 Calorimeter Trigger 7

LHC CMS Detector Upgrade Project Production CTP 7 s ← in Stage 2 Rack Wesley Smith, 14 October 2015 CTP 7 Without CXPs + heatsinks ← Front Back→ DOE Review -- B 04 -2: Stage 2 Calorimeter Trigger 8

LHC CMS Detector Upgrade Project CTP 7 Builds Build Number 1 2 3 4 Time Frame Q 4/2013 Series (PCB Scheduled Rev) Qty Reject/ Fail Yield Qty Falcon (A) 2 0 2 Q 2 -Q 3/ 2014 Raven (B) 6 0 6 Q 1/2015 Eagle (C) 8 0 8 Q 1 -Q 2/ 2015 Eagle (C) 42 0 42 NB: One Build 4 CTP 7 had a connector bent in shipping to CERN, to be repaired Wesley Smith, 14 October 2015 DOE Review -- B 04 -2: Stage 2 Calorimeter Trigger 9

LHC CMS Detector Upgrade Project Layer 1 Installation Design 48 V Power Panel o. RSC 10 G Patch Panels 9× ECAL/H F CTP Crate #1 Fiber Feed-Thru 48 V Power Panel 96 LC to 8 MPO 12 Patch Panel CTP Crate #2 9× ECAL/H F Fiber Feed-Thru 96 LC to 8 MPO 12 Patch Panel 48 V Power Panel CTP Crate #3 2 × 7 k. W@48 V Power Chassis S 2 E 10 Back Side Wesley Smith, 14 October 2015 S 2 E 10 Front Side § ECAL/HF Fibers enter back side of rack as individual LC connections § 3 U Optical Feed-thru panels pass MPO 12 fibers internally through rack to front side § Rx connections made to CTP 7 cards as in-line MPO 12 -MPO 12 connections on front side § 3 CTP 7 u. TCA Crates, each covering 120° of φ and all of η § 1 U Power Panel delivers 48 V from existing RCT internal power supplies to crate Power Modules DOE Review -- B 04 -2: Stage 2 Calorimeter Trigger 10

LHC CMS Detector Upgrade Project Stage 2 Layer 1 Installation Wesley Smith, 14 October 2015 DOE Review -- B 04 -2: Stage 2 Calorimeter Trigger 11

LHC CMS Detector Upgrade Project CTP 7 Crate Installation at CERN § All 4 Final CTP 7 Crates installed and operating § 1 for Stage 1 (rack 9) § 3 for Stage 2 (rack 10) Stg. 1 Stg. 2 o. RSC Stg. 2 Stage 2 Legacy RCT x 9 o. RSC Stg. 2 Wesley Smith, 14 October 2015 DOE Review -- B 04 -2: Stage 2 Calorimeter Trigger 12

LHC CMS Detector Upgrade Project Replacing a CTP 7 Transmitter § Defective transmitter → § Avago CXP – one channel failed in operation § Hot swap replacement: § Disconnect optical fiber (MPO 12) § Pull handle on CXP on CTP 7 § CXP slides out, taking care of its own shutdown § Go to cabinet § Get replacement CXP § Insert new CXP into CTP 7 § CXP reinitializes, turns back on and starts operating § Plug back in optical fiber § Elapsed time: 3 min o Mostly walking over to cabinet! Wesley Smith, 14 October 2015 DOE Review -- B 04 -2: Stage 2 Calorimeter Trigger 13

LHC CMS Detector Upgrade Project Cal. Trig. Stage-2 Layer-1 Status § Hardware installation finished in April ‘ 15 § 1152 Layer-1 input fibers connected in June’ 15 § All 576 ECAL links tested and mapping validated § All 72 HF links tested and mapping validated § All 504 HBHE links tested and mapping validated § 864 Layer-1 output fibers connected June‘ 15 § All links tested and mapping validated § Firmware with production-ready DAQ functionality completed in August’ 15 § Full resolution input TPG DAQ readout § Layer-1 timed in, commissioned and stress-tested during TS 2 in c. DAQ § DAQ Status September ‘ 15: § All 3 Layer-1 FEDs timed in, commissioned and stress-tested in c. DAQ during TS 2 § Operating in Global Data-taking since 9/25. Wesley Smith, 14 October 2015 DOE Review -- P 04: Trigger Upgrade 14

LHC CMS Detector Upgrade Project ZYNQ Features § ZYNQ So. C Controller and embedded Linux § ZYNQ FPGA runs embedded Linux automatically loaded from a micro. SD card and is responsible for ancillary functions on the CTP 7. It provides Gb. E TCP/IP communication for the card, as well as specialized Linuxbased software: Integrated Eye Scan Engine, Xilinx Virtual Cable Service, dedicated Memory Service, Remote Procedure Call Service, and MMC Initialization Agent. § AXI (Advanced e. Xtensible Interface) interconnect protocol § AXI Chip-to-Chip Xilinx IP core is employed to seamlessly extend the memory space of the ZYNQ FPGA to the Virtex-7 FPGA. § ZYNQ-based Xilinx Virtual Cable (XVC) § § § Replaces physical cable with TCP/IP connection to Xilinx toolset Useful where JTAG cable not easily employed Does not require JTAG infrastructure in u. TCA crate ZYNQ connects to V 7 JTAG pins through programmable logic Cable Server runs on ZYNQ embedded CTP 7 Linux system Allows remote debug of Virtex-7 firmware via LAN connection to ZYNQ Wesley Smith, 14 October 2015 DOE Review -- B 04 -2: Stage 2 Calorimeter Trigger 15

LHC CMS Detector Upgrade Project Integrated Eye Scan (ZYNQ-based Application Example) § Familiar Eye Scan diagrams available in IBERT § Scanning engine ported to CTP 7 ZYNQ § Low-latency access to links via AXI interface § Parallel Eye Scans in ~same time as a single scan § Runs in parallel in CTP 7 FW during data-taking Wesley Smith, 14 October 2015 DOE Review -- B 04 -2: Stage 2 Calorimeter Trigger 16

LHC CMS Detector Upgrade Project CTP 7 Firmware § Enough resources in V 7 690 T FPGA � § Careful planning went into link allocation § Firmware logic split into 2 slices: Eta+ and Etao 2 × ½ of η × 20° in φ § Excellent results using Xilinx Vivado Toolchain � § Low resource utilization § Timing constraints met § No need for manual floorplanning o Was in the back pocket if needed § FW fully operational at P 5 in data running since Sept. 25. § No problems. Wesley Smith, 14 October 2015 DOE Review -- B 04 -2: Stage 2 Calorimeter Trigger 08 -April -2015 1

LHC CMS Detector Upgrade Project Calorimeter Trigger Firmware Eta+ I/O Eta- L 1 Logic Eta- I/O Layer-1 690 T FPGA Floorplan Post-Implementation Resource Utilization Resource Support Logic (AXI Chip 2 Chip, Register Files) Eta+ I/O Eta+ L 1 Logic DAQ I/O + Buffering Wesley Smith, 14 October 2015 Utilization [%] LUTs 22 Registers 27 BRAM Memories 30 GTH I/O * 81 * 65/80 GTHs utilized • 64 GTHs for Calo inputs and L 2 outputs • 1 GTH for 5 Gbps DAQ DOE Review -- B 04 -2: Stage 2 Calorimeter Trigger 08 -April -2015 1

LHC CMS Detector Upgrade Project Stage-2 Layer-1 Software § Integration with the RCT Trigger Supervisor complete § 18 x CTP 7 and 3 x AMC 13 configuration tested in local mode o Link masking, alignment and alignment checks o DAQ configuration (AMC 13 and CTP 7 DAQ FW) § Running now in global data-taking § Achieved Milestone of Parallel Operation with Stage-1 § Data from Stage-2 / Layer-1 is now routinely in runs at P 5 § Completing Layer-1 Data Quality Monitoring § Data unpacker + Offline DQM is running on recorded data § Results on next slide § Online SW Longer Term Plan § Migrate Layer-1 from Trigger Supervisor to new online framework as soon as it becomes available for production use at P 5 o We are up to date at P 5 with most recent release o We have implemented and tested command sequences for CTP 7 configuration o New Gatekeeper pulls info from our new configuration XML file - database transition should be transparent Wesley Smith, 14 October 2015 DOE Review -- B 04 -2: Stage 2 Calorimeter Trigger 19

LHC CMS Detector Upgrade Project RCT + Stage-2 Layer-1 DQM Non-Zero ET RCT Regions For Triggering BX and +/- 2 BX § Count of non-zero regions § Used for PU estimate for Stage -1 § RCT Readout (Stage-1) used to validate Stage-2 / Layer-1 § ECAL TPGs sourced from ECAL/TCC or Stage-2 / Layer-1 readouts ⇒ 100% agreement! § Stage-2 / Layer-1 now used to monitor transient issues from ECAL § Used for both Stage-1 and Stage-2 Wesley Smith, 14 October 2015 DOE Review -- B 04 -2: Stage 2 Calorimeter Trigger 20

LHC CMS Detector Upgrade Project § § Cal. Trig. CIOx Status Design and layout complete Parts for 5 boards are in hand RFQ for PCBs will be sent next week Plan to make 3 prototypes for testing Wesley Smith, 14 October 2015 Essential custodial functions removed from individual CTP 7 s onto a single board connected to CTP 7 s via the backplane – savings in cost and circuitry DOE Review -- B 04 -2: Stage 2 Calorimeter Trigger 21

LHC CMS Detector Upgrade Project Schedule – Cal Trig. Milestones WBS L 3 Milestone 402. 4. 3 MPCM Production Complete Feb-2014 ✓ 402. 4. 4 o. SLB-o. RM Optical Fibers Complete Jun-2014 ✓ 402. 4. 3 MPC-EMUTF Optical Fibers Complete Jul-2014 ✓ 402. 4. 3 MSM Production Complete Apr-2015 Aug-2015 A 402. 4. 4 CTP 7 Production Complete May-2015 Apr-2015 A 402. 4. 3 EMUTF Production Complete Jul-2015 Nov-2015 S 402. 4. 4 CTP 7 Infrastructure Complete Jan-2016 S 402. 4. 4 CIOx Production Complete Mar-2016 S 402. 4. 3 EMUTF Infrastructure Commissioned Apr-2016 Aug-2015 A 402. 4. 3 MS-EMUTF Commissioned Dec-2016 Sep-2016 S 402. 4. 4 Calorimeter Trigger Complete Dec-2016 Sep-2016 S 402. 4. 3 Muon Trigger Complete Dec-2016 Sep-2016 S 402. 4. 3, 4 Project objective KPP complete Mar-2017 S Wesley Smith, 14 October 2015 Milestone Date DOE Review -- B 04 -2: Stage 2 Calorimeter Trigger Schedule/Act ual Date 22

LHC CMS Detector Upgrade Project Calorimeter Trigger EVM Wesley Smith, 14 October 2015 DOE Review -- B 04 -2: Stage 2 Calorimeter Trigger 23

LHC CMS Detector Upgrade Project Calorimeter Trigger EV curve Wesley Smith, 14 October 2015 DOE Review -- B 04 -2: Stage 2 Calorimeter Trigger 24

LHC CMS Detector Upgrade Project Cal Trig. Resources, FTE Profiles Wesley Smith, 14 October 2015 DOE Review -- B 04 -2: Stage 2 Calorimeter Trigger 25

LHC CMS Detector Upgrade Project Trigger Plans § Commission full Calorimeter Trigger Stage-2 (Layer-1 + Layer-2) system and operate in parallel for rest of 2015, then operate as main trigger for 2016 Run § Install CIOx cards to provide additional diagnostics and programming access. § Improve performance using information from datataking during 2016 run § Improvements in firmware, software, DQM Wesley Smith, 14 October 2015 DOE Review -- B 04 -2: Stage 2 Calorimeter Trigger 26

LHC CMS Detector Upgrade Project Conclusions § Stage 2 Calo. Trig. Hardware major production complete. § Remaining Service Card, CIOx provides improved operations, diagnostics and serviceability, but is not mission critical. § Calorimeter Trigger Stage-2 Layer-1 in full operation as of end of September § Running in parallel with Stage-1, taking data § Firmware mature, robust and proven in data-taking § Online software is working with previous online systems and ready for integration with new online framework when available. § Online and Offline DQM are working and providing detailed feedback. § Development of improved firmware and online/offline software continuing. § Optimization based on data-taking and operational experience § Preparations underway for higher luminosity running o Increased Pileup § Fully expect Stage-2 Calorimeter Trigger will finish successfully on time and within budget Wesley Smith, 14 October 2015 DOE Review -- B 04 -2: Stage 2 Calorimeter Trigger 27

LHC CMS Detector Upgrade Project Additional Slides Wesley Smith, 14 October 2015 DOE Review -- B 04 -2: Stage 2 Calorimeter Trigger 28

LHC CMS Detector Upgrade Project Calorimeter Trigger Components Type Name Purpose TCC Trigger Concentrator Card Existing ECAL trigger primitives (TP) (Paid by FR) o. SLB Optical Serial Link Board Transmit ECAL TPs to RCT & CTP (Paid by PT) m. HTR HCAL Trigger & DAQ Board Create and transmit HCAL TPs RCT Regional Calorimeter Trigger Existing trigger system o. RM Optical Receiver Modules Receive ECAL TPs on RCT (Paid by FR) o. RSC Optical Regional Summary Card Convert and transmit RCT output (paid by DOE Nuclear) CTP Calorimeter Trigger Processor Large FPGA w/ optical & backplane CIOX Compact IO cross-point switch Intra-crate sharing switch MP 7 Multipurpose Processor Large FPGA w/ only optical links (UK) Wesley Smith, 14 October 2015 DOE Review -- B 04 -2: Stage 2 Calorimeter Trigger 29