GSI CBM Volker Lindenstruth Kirchhoff Institute for Physics

GSI CBM Volker Lindenstruth Kirchhoff Institute for Physics Chair of Computer Science University Heidelberg, Germany Phone: +49 6221 54 4303 Fax: +49 6221 54 4345 Email: ti@kip. uni-heidelberg. de WWW: www. ti. uni-hd. de Chair of Computer Science (www. ti. uni-hd. de)

Overview L 0 L 1 10 MHz Reaction Counter 200 k Ch, 20 Tbin 6 Layers, 10 MHz 5000 k. B RAW 500 k. B Hits <2 k. B Tracklets 4. 5 M Ch 7 Layers 500 k. B RAW 50 k. B Hits 70 k Ch 2 Layers 70 k. B RAW 7 k. B Hits 1 MHz 33 k. Hz DAQ 16 k. Hz L 0 Trigger 2. 5 TB/s <10 GB/s Tracklet search TRD <2 GB/s TRD Layer Tracking tracks L 1 TMU 100 MB/s 100 GB/s 84 k Ch 100 k. B RAW 10 k. B Hits <1 k. B Rings L 2 RICH 500 GB/s 50 GB/s Cluster search Silicon 70 GB/s TOF <1 GB/s Ring search rings 33. . . 1000 k. Hz <1 GB/sec Si-Vertex Processor L 2 Vertex 7 GB/s Readout 9 GB/s @ 16 k. Hz Chair of Computer Science (www. ti. uni-hd. de) HLT / DAQ 2

Overall Observations and Axioms • No physics or detector requirement for maximum trigger latency • Can afford to implmement large derandomizing Buffers at L 0 • Move Data only if necessary – (keep RAW data in FE until L 2 accept) • Process local data locally – ship only results (tracklets, rings, (displaced) vertices) Chair of Computer Science (www. ti. uni-hd. de) 3

TRD Trigger Architecture Reaction Counter Aggregate In: Aggregate Out: Rate: L 0 Trigger TRD Tracklet search TRD Layer Tracking 2. 5 TB/s <2 GB/s 1 MHz tracks L 1 TMU RICH Ring search Silicon Cluster search rings Si-Vertex Processor L 2 Vertex TOF Readout HLT / DAQ Chair of Computer Science (www. ti. uni-hd. de) 4

ALICE TRD Overview Detector . i n z B su bd iv B Chamber 5 18 subdivisions in f 6 Layers Stack ws 24 Multi-Chip-Module Tracklet Processor 8 Chips (144 Channel) preamp • 1156032 analog channels • 10 MHz digitization rate • 18+3 channels per module • 64224 sources 1 trigger bit • track processing on chamber • trigger/readout integrated • maximum latency 6µs • ~20 space points per tracklet • 4 -6 tracklets (layers) per track o dr pa AGND 18 Channels Chair of Computer Science (www. ti. uni-hd. de) APWR Signal layers DGND DPWR 5

TRD Electronics Chain L 1 trigger To CTP Tracklet TRD PASA ADC Preprocessor TPP Tracklet Processor TP Tracklet Merger TM GTU To HLT & DAQ event buffer Store RAW data until L 1 A detector 6 layers 1200000 Channels charge sensitive preamplifier Shaper ADC 10 Bits 10 MHz Chair of Computer Science (www. ti. uni-hd. de) dig. filter (pedestal, tail corr. ) preprocess data fit trackets @L 1 A compress & ship RAW data Readout tree 270 GB/sec Merge tracklets Into tracks Compute Ro. I Ship RAW data @L 2 A 6

TRD Trigger Timing Data ship Calculate S fit Calculate Tracklets relevant pipeline ADC output Global Tracking Drift time 998 1996 Tracklet TRD PASA ADC Preprocessor TPP event buffer Chair of Computer Science (www. ti. uni-hd. de) 2994 3992 Tracklet Preprocessor TPP Tracklet Processor TP 4990 Tracklet Merger TM 5990 t [ns] GTU event buffer 7

Tracklet Fit (Tracklet Preprocessor) During Drift Time: N = hit count yi = position xi = time bin sum yi = position sum xi yi = time bin*position sum yi 2 = position² sum After Drift Time: a = intercept b = slope 2 = track quality merge track segments in pad row Pipelined Calculation : AP-1 AP 10 6 Ax Ax/Ac 7 LUT yi 10 Build Ratio Calculate Position Chair of Computer Science (www. ti. uni-hd. de) 6 xi yi 2 xi yi xi 2 Calculate fit summands 10 11 xi yi xi 2 yi 2 12 xi yi 2 Calculate sums xi yi xi yi yi´ 2 HC Amplitude AP+1 10 Sum Memory 8

Tracklet (MIMD) Processor Shared I-Mem Projection of Pre-Processor data inside the MIMD Four ALUs • Four Harvard CPUs coupled by Shared data via • multi port instruction memory GRF and D-Mem • multi port data memory • Global Register File (GRF) • Register based interface to Pre-Processor • Two stage pipeline (fetch/decode, execute/write back) • Architecture can be adopted to any general purpose processor Chair of Computer Science (www. ti. uni-hd. de) 9

Tracklet Track Trigger (Tracklet Merger) L 02 L 10 Readout Tree R 02 L 10 L 20 L 12 M 112 (L 21) M 100 f 1 f 2 f 3 OC 48 R 01 M 111 R 00 Sublayer 1 L 01 M 110 L 00 L 11 R 20 R 12 M 113 (R 21) R 11 GTU Readout Board (RB) 4 3 2 1 0 9 8 7 6 5 14 13 12 11 10 19 18 17 16 15 Chair of Computer Science (www. ti. uni-hd. de) 10

Tracklet Preprocessor FF IMEM Event Buffers IMEM • 4. 5 M transistors • 180 nm DSM process • all clocks gated • 21 digital filters (Xtalk + 1/t) • 18 full channels • multi-event buffers • 3 x 10 Bit, 10 MHz ADC • ADC 0. 1 mm² • PLL clock recovery • 4+1 2. 5 Gbit Links • BIST, JTAG • 4 RISC processors • redundant, fault tolerant nonfig. network • full custom QPM • tape-out May 12 • 5000 x 5000 µm die • 1. 8 V Core • TOX = 4 nm ADC FF DMEM Core 120 DMEM TRAP 1 Die ID Chair of Computer Science (www. ti. uni-hd. de) Tracklet Merger 11

ALICE TRD Global Tracking Unit · global tracking is performed module wise (540 modules). 1 1 optical receiver 1 1 deserializer 16 & chamber 6 x 1 1 6*2*2. 5 GBd fiber channel optical receiver 1 1 histogram engine depattern serializer 16 recognition deserializer 16 6*2*2. 5 GBd PECL 6*2*16 Bit/120 MHz TTL CTP module chamber GTU Module (x 5 x 18) · input of one GTU are 6*2 2. 5 GBd optical fiber channel links which are converted into 6*2 16 Bit/120 Mhz data streams. · readout order of individual chambers are optimized for histogramming and pattern recognition in parallel. · histogramming engine fills a data field with tracklet candidates regarding their position. · looking for patterns with middle plane · a cluster of at least 4 candidates · adequate similar angle · high momentum · adequate quality · quality and position of found hit is sent to central trigger processor · GTU operates also for raw data readout and compression Chair of Computer Science (www. ti. uni-hd. de) 12

CBM TRD Outlook • Detector Occuppancy critical • Interaction rate > drift time • L 0 provides more time-stamp kind of information • Multi-Event pile-up has to be detected and sorted out electronically • Large numbers of Channels and large event buffers can be well integrated • Digitize as early as possible • Use digital filters • Interference of digital circuitry into analog front-end even more critical Chair of Computer Science (www. ti. uni-hd. de) 13

L 1/L 2 Si-Vertex Trigger Processor Reaction Counter Aggregate In: Rate: L 0 Trigger TRD Tracklet search TRD Layer Tracking <5 GB/s 1 MHz tracks L 1 TMU RICH Ring search Silicon Cluster search rings Si-Vertex Processor L 2 Vertex TOF Readout HLT / DAQ Chair of Computer Science (www. ti. uni-hd. de) 14

LHCb Vertex Position 2 D tracking Primary vertex Secondary tracks 3 D tracking Secondary vertices i. Kisel Chair of Computer Science (www. ti. uni-hd. de) • VELO STATISTICS (2 D) • • Ref. B efficiency (%|Nmc)) : 0. 9801 | 853 Refprim efficiency (%|Nmc)) : 0. 9741 | 5179 Refset efficiency (%|Nmc)) : 0. 9598 | 20797 Allset efficiency (%|Nmc)) : 0. 9161 | 26565 Extra efficiency (%|Nmc)) : 0. 7581 | 5768 Clone probability (%|Rec)) : 0. 0288 | 700 Ghost probability (%|Rec)) : 0. 0205 | 498 MC tracks/event found : 64 : 380 events PRIMARY VERTEX RESOLUTION: s(x, y) = 16 mm, sz = 47 mm 15

Scheduled Subevent Transfer @ LHCb Packets of 128 Bytes are assembled in PCI Cards and distributed in PC cluster, implementing event building and load balanced data distribution at rates exceeding 1. 9 MHz. Hardware initiated DMA through Scalable Coherent Interface Final System will implement about 300 PCs. Maximum Latency 2 ms. SCH 128 byte packets @ a rate of 1. 96 MHz hardware initiated MHZ Transfer Rates Demonstrated in 3 x 10 Torus RS http: //www. ti. uni-hd. de/LHCb Chair of Computer Science (www. ti. uni-hd. de) 16

MHz Event Rate in PC Clusters 64 B @ PCI 32/33, 2 x. ORCA interleaved Address (1 cc) Wait (4 cc) Data (16 cc) 128 B @ PCI 64/66, 1 x. ALTERA Address (1 cc) Wait (7 cc) Data (16 cc) Idle (1 cc) fmax = 1. 50 MHz Idle (10 cc) fmax = 1. 94 MHz Chair of Computer Science (www. ti. uni-hd. de) 17

Trigger Processor Farms Reaction Counter Aggregate In: Rate: L 0 Trigger TRD Tracklet search TRD Layer Tracking 9 GB/s 16 k. Hz tracks L 1 TMU RICH Ring search Silicon Cluster search rings Si-Vertex Processor L 2 Vertex TOF Readout HLT / DAQ Chair of Computer Science (www. ti. uni-hd. de) 18

Receiving Detector Data into Cluster • Probably cheapest memory available is in commodity computers • Use that memory as event- and latency buffer • Perform first level processing (e. g. cluster finder) where data exists • Avoid unnecessary copying of data • Use programmable FPGA to off-load host processors (FPGA Coprocessor) • Make PCI(-X) baseline bus enabling use of all commodity computers • Use CMC form factor for optical detector data link circuitry on mezzanine Chair of Computer Science (www. ti. uni-hd. de) 19

Modular Cluster Communication Package New Event Publisher Subscriber New Event Publisher Load balancing Fan-out Data Block Distributor New Event Data Block Merger Event M Block 0 Event M Block 1 Subs Data Block Collector Subscriber Publisher New Event Subscriber Bridging between Nodes Subscriber. Bridge. Head • Subscribes to upstream • Gets input data from publisher • Sends data over network Merging Code Publisher Event M Block 0, 1 … Publisher. Bridge. Head • Publishes to downstream • Reads data from network Chair of Computer Science (www. ti. uni-hd. de) 20

System benchmark • pp pileup removal at 270 Hz Rate >270 Hz (TCP/IP, fast ethernet) SM CPU: 100% EG PM EM SM Slice Merger EG Event Gatherer PM Patch Merger EM Event Merger EG Event Gatherer Netto: 6*131 k. B/sec EG T Netto: 6*1. 8 MB/sec T T EG T T ES ES ES CF CF CF AU AU AU FP FP FP 4 5 6 1 2 3 Chair Patches of Computer Science (www. ti. uni-hd. de) CPU: 60 -70% Brutto: 6*2. 4 MB/sec T EG CPU: 75 -100% Brutto: 6*374 k. B/sec T T Tracker (2 x) ES Event Scatterer CF Cluster Finder AU ADC Unpacker FP File Publisher 21

Publisher / Subscriber Fault Tolerance Not a single event lost During error recovery Chair of Computer Science (www. ti. uni-hd. de) 22

Fault Tolerance in large Clusters -PCI Linux PC in every PC -BIOS access -Hardware controlling -POST codes -Floppy emulation -Remote access to a broken PC -Detecting hard- & software errors -Starting self-acting error correction -Web interface for Monitoring & interacting - Part of Data. GRID Project Chair of Computer Science (www. ti. uni-hd. de) 23

Cluster RAID B noncritical data local redundant synchronous backup: parity = d(A) Å d(B) Å d(C) Å d(D) other‘s backup incremental backup: paritynew = parityold Å dold(X) Å dnew(X) 25% data with backup A local exclusive local redundant local exclusive 25% local redundant 25% other‘s backup local redundant other‘s backup C other‘s backup = 25% of local redundant D Chair of Computer Science (www. ti. uni-hd. de) 25% local exclusive local redundant other‘s backup Distributed Block Device Enabling implementation of Distributed, reliable mass storage within cluster Cluster RAID 24

Chair of Computer Science (www. ti. uni-hd. de) 25

Heidelberg HLT cluster Chair of Computer Science (www. ti. uni-hd. de) 26

Summary - Conclusions • PC Clusters are a powerful tool up to 1. . . 2 MHz Subevent Rates and 2. . . 4 GB/sec transfer rates – Data easily transferred into L 1 / L 2 Caches – Scalability due to Moores law • At higher (data or transaction) rates Systolic FPGA (Co)Processors • ASICs for large channel counts • Complex low-power readout and (pre)trigger circuitry possible and likely to benefit much from Moores Law • High integration density and high-speed processes allow implementation of very inexpensive and low-power ADCs • Co-existence of ADC and clocked circuitry demonstrated to work well • Challenging GSI CBM Program does not seem to present any unsurmountable problems w. r. t. trigger and readout Chair of Computer Science (www. ti. uni-hd. de) 27