Installation and Commissioning of the CMS Regional Calorimeter

Installation and Commissioning of the CMS Regional Calorimeter Trigger P. Klabbers, S. Dasu, T. Gorski, K. Grogg, M. Grothe, M. Jaworski, J. Lackey, C. Lazaridis, J. Leonard, P. Robl, W. H. Smith, M. Weinberg Physics Department, University of Wisconsin, Madison, WI, USA TWEPP 2007 September 4, 2007 The pdf file of this talk is available at: http: //indico. cern. ch/contribution. Display. py? contrib. Id=76& session. Id=5& conf. Id=11994 See also the CMS Level 1 Trigger Home page at http: //cmsdoc. cern. ch/ftp/afscms/TRIDAS/html/level 1. html P. Klabbers, U. Wisconsin, TWEPP September 2007 CMS Regional Calorimeter Trigger - 1

CMS Detector P. Klabbers, U. Wisconsin, TWEPP September 2007 CMS Regional Calorimeter Trigger - 2

Trigger & DAQ Systems Level-1 Trigger Event Manager Detector Frontend Readout Systems Run Control Builder Networks Filter Systems Computing Services Level-1 Trigger Requirements: • Input: 109 events/sec at 40 MHz at full Luminosity of 1034 cm-2 s-1 • Output: 100 k. Hz (50 k. Hz for initial running) • Latency: 3 sec for collection, decision, propagation P. Klabbers, U. Wisconsin, TWEPP September 2007 CMS Regional Calorimeter Trigger - 3

Calorimeter Trig. Overview (located in underground counting room) Copper 80 MHz Parallel 4 K 1. 2 Gbaud serial links w/ 4 Highest ET: 2 x (8 bits E/H/FCAL Energy Lumi. Isolated & non-isol. e/ + fine grain structure bit) nosity Central, forward, jets, + 5 bits error detection code Info. E , E from each crate x y per 25 ns crossing US CMS: UK CMS: Imperial/Bristol US CMS HCAL: Wisconsin Global Cal. Trigger BU/FNAL/ Sorting, ETMiss, ET Regional Maryland/ Calorimeter Princeton Trigger Calorimeter Global Electronics Trigger Interface Receiver CMS: Processor Electron Isolation CMS ECAL: Vienna Jet/Summary Lisbon/ Global Muon Trigger Palaiseau 72 f x 60 H/ECAL Iso Mu Min. Ion Tag Towers (. 087 f x. 087 for <2. 2 &. 174 -. 195 , >2. 2) HF: 2 x(12 f x 12 ) P. Klabbers, U. Wisconsin, TWEPP September 2007 Min. Ion & Quiet Tags for each 4 f x 4 region CMS Regional Calorimeter Trigger - 4

CMS Calorimeter Geometry EB, EE, HB, HE map to 18 RCT crates Provide e/ and jet, , ET triggers P. Klabbers, U. Wisconsin, TWEPP September 2007 CMS Regional Calorimeter Trigger - 5

Trigger Mapping P. Klabbers, U. Wisconsin, TWEPP September 2007 CMS Regional Calorimeter Trigger - 6

Calorimeter Trig. Algorithms Electron (Hit Tower + Max) • 2 -tower ET + Hit tower H/E • Hit tower 2 x 5 -crystal strips >90% ET in 5 x 5 (Fine Grain) Isolated Electron (3 x 3 Tower) • Quiet neighbors: all towers pass Fine Grain & H/E • One group of 5 EM ET < Thr. P. Klabbers, U. Wisconsin, TWEPP September 2007 Jet or ET • 12 x 12 trig. tower ET sliding in 4 x 4 steps w/central 4 x 4 ET > others : isolated narrow energy deposits • Energy spread outside veto pattern sets veto • Jet if all 9 4 x 4 region vetoes off CMS Regional Calorimeter Trigger - 7

Regional Calorimeter Trigger Crate Data from calorimeter Trigger Primitive Generators on Cu links at 1. 2 Gbaud • Into 126* rear Receiver Cards VME 48 V DC Power 2 Std. VME Slots Custom Point-to-point Dataflow on AMP Stripline HD Conn. Designed by J. Lackey 160 MHz point to point backplane, differential ECL with 0. 4 Tbit/sec dataflow • Designed to incorporate algorithms: Non-Isolated Electron, Tau & Jet Triggers • 18* Clock&Control, 126* Electron ID, 18* Jet/Summary Cards *Spares • Use 5 Custom Gate-Array 160 MHz Ga. As Vitesse Digital ASICs not • Phase, Adder, Boundary Scan, Electron Isolation, Sort included • 18 crates and backplanes needed, 26 crates and 28 backplanes produced P. Klabbers, U. Wisconsin, TWEPP September 2007 CMS Regional Calorimeter Trigger - 8

RCT Cards Clock & Control Receiver 126/158* 7 per crate 18/25* 1 per crate Master Clock Crate Input Data Sharing on cables for e/ algo delay adjust Receiver Mezzanine Card (RMC) Provides 160 MHz and 120 MHz clocks, reset, BC 0 to entire RCT crate Generated on board or received from Master Clock Crate fed by CMS Trigger Timing and Control (TTC) System Fans out signals and adjusts phase to all boards Clock and Reset timing set with delay adjust P. Klabbers, U. Wisconsin, TWEPP September 2007 Adder ASICs Receives 8 x 8 E/HCAL towers on 1. 2 GB Cu Links (Vitesse 7216 -1) on RMC’s Phase ASIC: Deskew, Mux at 160 MHz and provide Test Vectors cycle LUTs Memory LUT at 160 MHz Adder ASIC: 8 inputs at 160 MHz in 25 ns. BSCAN ASIC: Synchronizes shared data & Diff. Output at 160 MHz to backplane Reg. Sum ETs, MIP, sent to Jet/Sum. Card 50 Tower ETs & vetoes sent to Electron ID Card *used/total produced CMS Regional Calorimeter Trigger - 9

RCT Cards Jet Summary Electron Identification 18/25* 1 per crate 126/154* 7 per crate EISO ASICs Sort ASICs Forward Calorimeter Input Output to Global Calorimeter Trigger Processes a 4 x 8 tower region at 160 MHz Uses Sort (disabled) ASIC for BP receive and the EISO ASIC fully implements e/ algos LUT for ET conversion Highest ET iso and non-iso e/ per 4 x 4 region sent to JSC for sorting and top 4 of each type forwarded to Global Calorimeter Trigger 7 cards send candidates over backplane to Jet Summary card • 14 Isolated e/ one per 4 x 4 region • 14 Non-Iso e/ } P. Klabbers, U. Wisconsin, TWEPP September 2007 Electron/photon/muon • Sort ASICs receive data on BP & find top iso. & non-iso. ) • 14 Quiet Bits by threshold on JS • 14 Min. Ion bits from RC • Forward Calorimeter (HF) functionality • RMC on board & LUTs for HF ET’s • Region energies • HF and 4 x 4 tower sums (regions) to GCT for jets, t, & calculation of global quantities total & missing ET * used/total produced CMS Regional Calorimeter Trigger - 10

New Master Clock Crate and Cards RCT shares data on SCSI-type cables among its 18 crates at 160 MHz • Margins are tight • The MCC would create and fan out 160 MHz and 120 MHz clocks, Re. Sync and Bunch Crossing Zero to 18 RCT Crates’ Clock and Control Cards • Clock Input Card (CIC) - 1/5* • Input from CMS Trigger Timing and Control system • Provides machine clock, bunch Single system for 18 crates crossing information, and CMS-wide (normal operation) control signals via custom modules • On-board oscillator for standalone testing • Coarse and fine delay for each signal of up to 25 ns • Clock Fan-out Card Midlevel (CFCm - 2/7* & Clock Fan-out Card To Crates (CFCc - 7/13*) • Accepts inputs from CIC and fans out to crates • Fine delay (100 E 195’s) on boards for adjustment of delay to each crate • Signal distribution on thirty-six four-pair lowskew cables (Kerpen Mega. Line® 8) with VGA Double for 12 or 6 crates type D-Connectors. All cables will be the (to allow two different tests at once -) same length. Produced and Installed in USC 55 CMS Regional Calorimeter Trigger P. Klabbers, U. Wisconsin, TWEPP September 2007 *used/total produced 11

Test Card: Jet Capture Card Signals Out FPGAs Translators JSC Inputs CCC Signals In Delays Receives all the data from one RCT crate: 14 Reg. Sums, 8 HF ET’s, 8 e/ , and bits on 6 SCSI type cables - 10 boards produced • Located in spare VME slot of RCT crate • Clocks, reset, and BC 0 from Clock and Control Card on low-skew cable (Kerpen Meg. Line® 8) • Triggers out on programmable thresholds • Different data capture modes • Used in final validation of RCT crates • Used in integration tests with HCAL and ECAL Trigger Primitive Generators • Trigger Output used in Magnet Test and Cosmic Challenge Phase II for an HCAL Trigger P. Klabbers, U. Wisconsin, TWEPP September 2007 CMS Regional Calorimeter Trigger - 12

ECAL and HCAL Input to RCT Both HCAL HTR (HCAL Trigger and Readout) and ECAL TCC (Trigger Concentrator Card) use a Serial Link Board (SLB) with the Vitesse V 2716 -1 link chip on it • Configurable mezzanine card • Two Altera Cyclone FPGAs synchronize data for V 2716 and calculate Hamming Code • Clocking separate • Ensures data in time between subsystems HTR SLBs • Up to six SLBs send Tower Energies (a. k. a. Trigger Primitives or TPs) • Front-end data received on Fiber TCC • Up to nine SLBs send Tower Energies Both Receive front-end data on Fiber Initial tests as early as 2004 - now working regularly together in pattern tests (more later…) P. Klabbers, U. Wisconsin, TWEPP September 2007 TCC CMS Regional Calorimeter Trigger - 13

RCT Output to GCT Each RCT crate is connected to 3. 5 GCT Source Cards Source Card Crate • RCT output differential ECL • 63 cards needed • 2 RCT-GCT cable inputs/card • 45 Cards for Regional Sums • Duplication needed by GCT on =0 boundary for jet algo - only one input used • 9 source cards have inputs from two different crates • 18 Cards for iso-e/ and noniso-e/ candidates, muon bits • Source card sends data on optical fiber to GCT crate on lower level of electronics hall • GCT turns regional sums into jet candidates and sorts jet and e/ candidates and sends to Global Trigger (GT) RCT Crate Front First tests with RCT only in July 2006 successful - more in May 2007 including capture of patterns from ECAL TCC (later slides). P. Klabbers, U. Wisconsin, TWEPP September 2007 CMS Regional Calorimeter Trigger - 14

Commissioning Underground at CMS 10 racks with 18 empty crates, all fans and rack monitoring hardware installed in July 2006 From Dec 2006 to Aug 2007 18 crates tested and installed and two more spare crates in USC 55 • All RCT card sets tested first at aboveground integration facility at CERN Prevessin site • Nearly all cabling as of Aug 2007 - input HCAL, HF, ECAL barrel, and output to GCT all installed Front of Racks Full system = 18 Crates Extra cabling and cards for extensive testing! P. Klabbers, U. Wisconsin, TWEPP September 2007 Crate Rear of Racks Input cabling nearly complete and currently under test. Crate Front CMS Regional Calorimeter Trigger - 15

Commissioning: Rack Monitoring System Custom-built hardware installed in USC 55 in July 2006 • Monitors power supplies, temperatures, fans • Configurable - alarm set points, number of fans, power supplies connected… • Ability to turn on and off system, check for and acknowledge alarms • Connects to network via a COMTROL serial-to-ethernet port This year, control software was developed using PVSS (Prozessvisualisierungs- und Steuerungs-System) Rack 1 PVSS Control Panel Rack 1 Crate A Temperatures Rack Monitor Card and power chassis Power & Temp. • Fully Implemented in USC 55 • Exploits all above functionality • Keeps values in database • Histograms available P. Klabbers, U. Wisconsin, TWEPP September 2007 Fans & Monitoring CMS Regional Calorimeter Trigger - 16

Commissioning: Trigger Supervisor CMS Trigger Supervisor • An online software framework to configure, test, operate, and monitor the trigger components and manage communications between systems • Set up as individual cells - with the central cell handling multiple systems at once with SOAP commands • Enables central configuration of trigger systems for day-to-day running and interconnection tests • RCT Trigger Supervisor handles • Crate configuration • User configurable or accepts a key from the central cell for a specific configuration • Keys for normal running, internal tests, and multi-system interconnection tests Crate Selection Card Selection Command Selection • Crate monitoring • RCT hardware registers and errors (not yet implemented) • 18 crate system installed and ready for use for crate operation in patterns and link testing P. Klabbers, U. Wisconsin, TWEPP September 2007 CMS Regional Calorimeter Trigger - 17

Commissioning: RCT Intercrate Testing Uses the ability of the RCT to cycle the addresses of its input LUTs on the Receiver cards • First done with Jet Capture Card receiving RCT output • Initially just two crates tested all internal/external connections • Up to 6 crates - with all data sharing cables • Unconnected edges quiet • Software steps through crates • • Configure crates and program all LUTs first Predict output Capture on JCCs Compare to prediction (Emulator - more on next slide) • Limit means physically moving setup to test next set of crates - this should be expanded to 18 RCT crates • Also is standalone command line code - would like to automate • To expand to all 18 RCT crates Global Calorimeter Trigger Source Cards must be used to capture output (in development) • Based on code for above tests • Eventual automation and integration into trigger supervisor • GCT Source Cards are very flexible - multiple capture options including BC 0, output patterns, and Re. Sync P. Klabbers, U. Wisconsin, TWEPP September 2007 CMS Regional Calorimeter Trigger - 18

CMS L 1 Calorimeter Trigger Pattern Tests: Trigger Emulator A software package • Its goal: to reproduce exactly the L 1 Trigger hardware response, including: • Look. Up Tables (LUTs) • Hardware and Firmware registers • Any other configuration options • It is to be used for hardware validation and monitoring • Currently in use by the calorimeter trigger to predict the response of the full chain to patterns injected at the trigger primitive level • ECAL and HCAL have files of trigger primitives that are inputs for their pattern generators and for the TE • RCT, GCT, and GT capture their output and input for direct comparison with the emulator • In this way the hardware and the emulator are fully vetted • Bugs are tracked down and fixed in firmware, hardware and software • In reverse: simulation can be used to inject physics patterns into the hardware • Validation of algorithms P. Klabbers, U. Wisconsin, TWEPP September 2007 CMS Regional Calorimeter Trigger - 19

HCAL to RCT in MTCC* Phase II One full RCT crate used: • Received 56 HCAL links (448 trigger towers) - top and bottom HCAL sections • JCC Output 2 bits - each an OR of MIPs (all top or all bottom) • Logical AND done in discrete NIM logic after RCT • Trigger sent and timed in at Local Trigger Control (LTC) of MTCC • Integrated 1 -Nov-2006 • Total rate ~1 Hz • Included in runs at 4 T! • Run started with Central Run Control, TS, RCT/HCAL, DAQ • RCT Trigger Supervisor implemented *Magnet Test and Cosmic Challenge P. Klabbers, CMS Week Trigger Meeting, 05 December 2006 HCAL to RCT input RCT output 20

Results from MTCC From Running Nov. 1: Drift Tube Trigger vs. Calorimeter Trigger time in 25 ns bunch crossings (coincidence = 0 bx) P. Klabbers, CMS Week Trigger Meeting, 05 December 2006 21

ECAL TCC and HCAL HTR to RCT Pattern Tests 2005 to June 2007 HCAL/ECAL sent patterns of Trigger Primitives to the RCT • Used Common Clocking and worked in two locations • Prevessin Integration Facility (2005 - early 2007) • Local Timing and Control board fans out signals to TTCci which distributes in correct form to each individual sub-system • USC 55 • LHC Clock distribution in place - all systems receive the same clock fanned out from the LHC machine interface • Can use Global Trigger or Local Timing and Control board to send orbit signals and other 8 -bit global commands to all systems • RCT Look. Up Tables (LUTs) can be programmed to send data via RCT Sums and/or e/ path • HCAL HTR (HCAL Trigger and Readout) to RCT • Used JCC to capture data: alignment and value checked • RC receives TPs and creates 4 x 4 tower sums • Verified value and alignment at JCC • Single crossing with all TPs filled an increment by 5 pattern from HCAL HTR verified as 4 x 4 sums • ECAL TCC (Trigger Concentrator Card) to RCT • Also used JCC to capture: alignment and value checked • Single crossing with all TPs filled sent to Sums path - result seen as 4 x 4 sum • A single tower per bunch crossing sent in incrementing pattern become isolated electrons, as expected P. Klabbers, U. Wisconsin, TWEPP September 2007 CMS Regional Calorimeter Trigger - 22

ECAL/HCAL/RCT/GCT Pattern Tests May 2007 to July 2007 HCAL HTR & ECAL TCC to RCT to GCT in USC 55 • One RCT crate used • Common clocking used • Verified phase of data arriving • RCT can check phase of 120 MHz data clock with 40 MHz LHC clock and verify alignment of data • HCAL HTR and/or ECAL TCC • Sent constant data on one Bunch Crossing • Sent increment pattern • RCT uses an identity LUT and sends output to GCT Source Card • GCT SC captured data • ECAL TCC data was captured at each step and sent through the GCT SC to the GCT Concentrator Card (just before the Global Trigger) • Checked with L 1 Trigger Emulator (Software) predictions • Found bugs in firmware and software • Repeated tests and verified fixes were correct P. Klabbers, U. Wisconsin, TWEPP September 2007 CMS Regional Calorimeter Trigger - 23

Full Trigger Chain: ECAL/RCT/GT Pattern Tests in July and August 2007 ECAL TCC sends patterns • Simple one trigger tower (iso-e) increment to RCT • RCT uses an identity LUT and sends output to GCT Source Card • GCT SC to GCT concentrator to Global Trigger PSB • Data captured at all levels and compared to L 1 Trigger Emulator prediction • Data matched software emulator except for a phi index mismatch (understood) • Good hardware/emulator agreement with more complex patterns and RCT e/ LUT’s LSB of 0. 5 (same as emulator): • Simulated 35 Ge. V electrons across multiple trigger towers in one 4 x 4 calo tower region - compared to emulator • Duplicate the 35 Ge. V electrons in another RCT crate and check synchronization - compared to emulator • Next steps this fall • Complex patterns to test sort algorithms, isolation in RCT, GCT, and GT • Join monthly Global Runs to test “slices” of detector & DAQ • Long term pattern verification (available from GT) • Expand to larger numbers of RCT crates P. Klabbers, U. Wisconsin, TWEPP September 2007 GLOBAL TRIGGER GLOBAL CALORIMETER TRIGGER CMS Regional Calorimeter Trigger - 24

Conclusions CMS Regional Calorimeter Trigger boards produced and installed • Test Board Jet Capture Card produced to validate input and output and help with integration tests • 10 racks with 18 fully populated crates and two spares crates installed. • 90% of input data cables, Master Clock Crate clock distribution in place, and 30% of output cables connected to Global Calorimeter Trigger source cards - another 30% shortly - remainder in October • Commissioning and integration continues this fall • Scheduling is critical - all other systems are commissioning • First steps - full checkout of installed input cables including mapping locations and cable quality checks • Continued verification of trigger algorithms and trigger emulator via increasingly complex pattern tests • Long term tests to verify repeated patterns are error free • Expansion of scope - more crates as systems become available after commissioning • Become involved in global runs (a few days once per month) • Slice of calorimeter trigger at first - full RCT in November run Though system is nearly complete, there is great deal of work ahead. The Regional Calorimeter Trigger will be ready for collisions in 2008! P. Klabbers, U. Wisconsin, TWEPP September 2007 CMS Regional Calorimeter Trigger - 25