Laboratoire de lAcclrateur Linaire Orsay France and CERN

Laboratoire de l’Accélérateur Linéaire, Orsay, France and CERN Olivier Callot on behalf of the LHCb collaboration Implementation and Performance of the LHCb trigger n The challenge n The solution: a three level system n Level-0 is synchronous, custom hardware n Level-1 is software, limited latency, on part of the data n HLT is software, same infrastructure, on the complete event n Performance n Future developments Beauty 2003, Carnegie Mellon University, Pittsburgh, PA, USA

The challenge u. LHC environment n 40 MHz crossing rate n Only about 3 x 107 crossing per second have interacting beams. u. LHCb environment n Luminosity l Single n Visible around 2 x 1032 cm-2 s-1 interaction preferred, to match B decay to its production vertex cross-section is about 70 mb l Producing n About at least 2 tracks in the LHCb acceptance 12 millions visible events per second. production cross-section around 500 μb. n l ~100, 000 Olivier Callot pairs per second… Implementation and performance of the LHCb trigger Beauty 2003 2

n But interesting decays have low branching fraction l 5 10 -6 for B ππ l 20 10 -6 for B J/Ψ( μμ)K 0 s n Interesting events have a total rate of a few Hz. l Acceptance n We and selection keep only order of 1 %. foresee to write on storage about 200 Hz. l Including side bands and reasonably wide cuts u. Requirements n Reduction factor around 20, 000 n Best possible efficiency on physics signal l E. g. lifetime cuts are applied offline to remove background èNo need to trigger on events that will be rejected l No dead time n Affordable, Olivier Callot debug-able, robust, … Implementation and performance of the LHCb trigger Beauty 2003 3

The solution u. Three level system n Level-0 : hardware system with fixed latency l Custom electronics l Pipe-line operation, synchronous, fixed latency : 4 μs l Reduce the rate to 106 events per second n Level-1 : Software analysis on part of the data l Reduced and packed data from only few detectors l Run on a PC farm, about 1800 CPU l Average latency 1 ms, maximum 58 ms. l Reduce the rate to 40, 000 events per second n High Level Trigger (HLT) : Software on the complete event l Tracking with almost final accuracy l Select interesting physics decays u. Coordination Olivier Callot by the Timing and Fast Control system Implementation and performance of the LHCb trigger Beauty 2003 4

Collision point Level-0 Level-1 Olivier Callot Implementation and performance of the LHCb trigger Beauty 2003 5

Level-0 u. Select high p. T particles n. B meson have a high mass. At least one decay product should have a few Ge. V p. T. u. Work for all types of particles n Hadrons, using large local deposits in HCAL n Electrons, photons, 0 using large local deposits in ECAL n Muons, using tracks in the Muon system u. Reject busy events n Multiple interactions, re-interaction, … l Easier to trigger on, more complex to process, not used for physics l Better to lower a bit the thresholds, and have more useful events. Olivier Callot Implementation and performance of the LHCb trigger Beauty 2003 6

Level-0 Calorimeter u. Local deposits n Use 2 cells x 2 cells area l ~6000 in ECAL, 8 x 8 to 24 x 24 cm 2 l ~1500 in HCAL, 26 x 26 or 52 x 52 cm 2 u. Work with ET n Converted from ADC value, on 8 bits n Need to access neighbours l Easy when on the same board l Dedicated backplane for most of the connections n Only highest ET interesting l Select u. Sign locally particle type with PRS-SPD n Same geometry address point to point link l But card-to-card communication LVDS links l Same Olivier Callot Implementation and performance of the LHCb trigger Beauty 2003 7

u. HCAL: n Only add ECAL deposits when high enough l The geometry is not the same u. Performed n 10 in the cavern Gy/year, SEU l anti-fuse PGA l triple voting u. Send candidates to Selection Crate n Via ~200 optical links to the barrack n Single crate to select the highest ET of each type n Send to L 0 Decision Unit u. Produce n Reject Olivier Callot also SPD multiplicity and Total HCAL ET busy or empty events Implementation and performance of the LHCb trigger Beauty 2003 8

Level-0 Muon trigger u. Straight line search in M 2 -M 5 n Chambers with projective geometry n Infinite momentum same cell l FOI to allow non-infinite momentum ! u. Extrapolation to M 1 shielding high occupancy n But OK with prediction from M 2 -M 3 n Before u. Momentum from M 1 -M 2 n Assuming from Primary Vertex n Using LUT l No Olivier Callot computation Implementation and performance of the LHCb trigger Beauty 2003 9

u. Processed n Each chamber data sent to the barracks chamber has 4 layers l OR of the chambers 100% efficiency n Logical pads created: l from physical pads, or l from crossing strips n Optical link l 148 ribbons of 12 fibres l Binary data BER not too important l But measured < 10 -15 from link, estimated 10 -11 with SEU. u 4 crates, each handling ¼ of the chamber n 15 cards, handling each a sub-region n Data exchanged via a dedicated backplane l High n 4 x 2 Olivier Callot speed point-to-point links muon candidates to L 0 Decision Unit Implementation and performance of the LHCb trigger Beauty 2003 10

Level-0 Pile-Up system u 2 silicon planes backwards n Measure l From Z of the Primary Vertex the ratio of the radii n Histogram u. Digital n Same Interaction region and search for second peak data sent to the barracks optical link technology u. Dedicated boards n 4 identical, one event / 100 ns to each. n Peak parameters sent to L 0 Decision Unit. Olivier Callot Implementation and performance of the LHCb trigger Beauty 2003 11

Level-0 Decision Unit u. Synchronisation n All of the data inputs at 40 MHz u. Apply thresholds n Five types of particles for Calorimeter n Eight muons select highest and pair u. Global variables n Reject busy events or multiple interactions n Not used for very clean channels l Di-muons Olivier Callot are always accepted Implementation and performance of the LHCb trigger Beauty 2003 12

Level-1 u. Select long lived particles n Tracking l Select n But in the Velo high Impact Parameter tracks multiple scattering can fake IP l Estimate n Confirm that there are high p. T particles with high IP u. Software n Send on a PC farm at 1 MHz events fragments to a CPU ! l 126 n Trick: p. T from deflection up to TT sources, around 30 bytes/event each Group 25 events in the same packet l Fit in a single Ethernet frame, this reduce the overhead l Less interrupts at destination n Big Olivier Callot (but available today) IP router to connect sources and CPUs. Implementation and performance of the LHCb trigger Beauty 2003 13

u. Same infrastructure for HLT n Same Ethernet network, same sub-farms and computing nodes n Nodes are running HLT / reconstruction in the background n When a Level-1 event arrives, it interrupts, and run at high priority l Minimal u. Separate n Available latency for Level-1, maximal use of the CPU. HLT inputs from “TELL 1” l Common to group links n Events also grouped by packet of ~10. Olivier Callot system Front-end Electronics FE 126 Links 1100 / 25 = 44 k. Hz 5. 5 GB/s readout board n Allows u. TFC Level-1 Traffic Switch FE FE FE Switch FE FE FE HLT Traffic TRM 323 Links 40 / 10 = 4 k. Hz 1. 6 GB/s Switch Multiplexing Layer 64 Links L 1 Decision Readout Network Gb Ethernet Level-1 Traffic HLT Traffic Mixed Traffic Sorter 94 Links 7. 1 GB/s Storage System SFC Switch CPU CPU Implementation and performance of the LHCb trigger CPU … SFC Switch CPU CPU 32 Links CPU Beauty 2003 TFC System 94 SFCs ~1800 CPUs CPU Farm 14

Level-1 software u. High n. R speed Velo tracking sensors, 45°sectors, 512 strips/sector, 40 μm to 100 μm pitch (Busy sector) Occupancy < 1% n Straight line search in R-Z view, forward and backward tracks l Around n Vertexing, l σZ tracks with high impact parameter, 0. 15 to 3 mm l about Olivier Callot using the sector number as Phi measurement ~ 60 μm, σX, Y ~ 25 μm n Select n Full 58 (+ ~30 backward) tracks 8. 5 per event space tracking for those tracks Implementation and performance of the LHCb trigger Beauty 2003 15

u. Momentum measurement using TT n Only 0. 15 T. m between Velo and TT n Measures p. T with 20 -40% accuracy n Reject most low momentum tracks l They u. Decision n For can fake high impact parameter. using two highest p. T tracks those with high IP. Minimum bias Signal events Olivier Callot Implementation and performance of the LHCb trigger Beauty 2003 16

u. In fact, add some “bonus” for identified particles n Di-muons l Mass n Photons J/Ψ peak events also 3 Ge. V ET photons are rare. l u. The are easy to tag whole processing takes around 8 ms NOW n We know how to improve it by ~20% n Moore’s law should give a factor 6 n 1 ms in 2007 seems feasible l We Olivier Callot need 1000 CPU for Level-1. Implementation and performance of the LHCb trigger Beauty 2003 17

HLT algorithms u. Tracking n Extend with (almost) full accuracy Velo tracks across the magnet (p)/p ~ 0. 6 % u. Confirm Level-1 decision n Select from the 3 D Velo the tracks to extend, measure accurately the momentum, confirm the decision l Gain a factor 2 without sensible loss l In about 25 ms on today’s CPU u. Full n All reconstruction tracks, around 50 ms now u. Physics n Not Olivier Callot 95% confirmed Input selection yet coded or timed Implementation and performance of the LHCb trigger Beauty 2003 18

Timing and Fast Control u. Distribute decisions using RD 45’s TTC system n Level-0 decision at 40 MHz n Level-1 decision at 1 MHz n Events’ destination at 40 k. Hz (L 1) / 4 k. Hz (HLT) l Simple u. Readout round-robin assignment, not a central event manager Supervisor as ‘chef d’orchestre’ n Emulate l 0. 5% the Level-0 occupancy front-end to avoid buffer overflow dead time at 1 MHz, guarantee to never exceed 1. 11 MHz n Receive throttle signals from the readout boards in case they can’t cope with the rate l In this case, disable temporarily the trigger, until the system is in a less busy state n Critical component l Prototype Olivier Callot exists and works well. Implementation and performance of the LHCb trigger Beauty 2003 19

Performance u. Measured on selected events n This means used for physics analysis n Minimum bias retention adjusted l 1 MHz at Level-0 èSharing also optimised l 40 u~30% k. Hz at Level-1 for hadronic channels n 60 -70 % with di-muons n Also around 30% for e/γ/π0 channels L 0 efficiency L 1 efficiency Total L 0 L 1 efficiency Olivier Callot Implementation and performance of the LHCb trigger Beauty 2003 20

Future developments u. TDR has been submitted last month n Approval u. Level-0 expected in November is mature n Hardware implementation has started l Fully synchronous independent of history l Many testing features included u. Level-1 and HLT hardware being prototyped n Commercial components (switches, router, PCs) l Routing available today, even if too expensive l Relies on Moore’s law to get powerful enough PC in 2007 n Connection Olivier Callot and control of the farm is the issue. Implementation and performance of the LHCb trigger Beauty 2003 21

u. Level-1 and HLT software being developed n Prototypes l Level-1 l HLT show that we can stay within the time budget code is in the correct speed range, need ~1000 CPU code is expected to take around 500 CPU èL 1 confirmation in about 25 ms today 4 ms in 2007 = 160 CPU èFull tracking in 60 ms today 10 ms in 2007 = 200 CPU èEvent selection should be < 40 ms today 140 CPU n Event selection is still at the prototype stage l We have code for a few channels, few ms today. l But we have many channels to select, and want to be quite open for the selection (if possible) LHCb will be able to efficiently select the events useful for CP violation studies in B decays from the beginning of LHC operation Olivier Callot Implementation and performance of the LHCb trigger Beauty 2003 22