Data Transport in Particle Physics Experiments Tony Gillman

Data Transport in Particle Physics Experiments Tony Gillman Particle Physics Department Rutherford Appleton Laboratory Tony Gillman – Particle Physics Department 1 RAL – 21 st January 2009

Data transport – scope Ø A very generic title… Ø “Transport” is meaning here “movement of signals and data” – Ø How are data transferred all the way from detectors to computers? Ø What happens to the signals during this journey – transmission media, formats, … Ø I will aim to cover a broad range of topics – Ø Analogue signal handling – and some of the pitfalls… Ø Analogue to Digital conversion techniques – the good and the bad… Ø Data serialisation and deserialisation – why bother… Ø Digital data transport media – copper vs silica Ø Purpose: give some idea of the problems of getting data from experiments Ø The first-level trigger of the ATLAS detector at the CERN LHC neatly illustrates many of these techniques, so will be used as a general case study Tony Gillman – Particle Physics Department 2 RAL – 21 st January 2009

The data challenge Ø Current generation of experiments will generate prodigious data volumes Ø ATLAS will produce ~1 Petabyte (1015 bytes) per second Ø In addition, the instantaneous data rates can be extremely high Ø The LHC collision rate is 40 MHz → bursts of new data arrive every 25 nsec Ø How do we transfer these data from the detectors into the data acquisition electronics → massive communication problem Ø Triggering removes the need to transport all of these data – Ø Store data for ~2 sec in pipeline memories Ø First-level trigger decides from which events to accept and transport the detector data Ø 40 MHz → 75 k. Hz max – (higher-level triggers reduce this much further) Tony Gillman – Particle Physics Department 3 RAL – 21 st January 2009

ATLAS trigger system Ø Still a challenge even to get trigger data into trigger electronics Ø Transport of data must be almost error -free, or trigger rate can become unacceptably high Ø Latency (time delay between collisions and trigger decision) must be short, to minimise data storage requirements (remember 106 Gigabytes per sec!) Ø Every part of signal chain must therefore be as fast as possible Tony Gillman – Particle Physics Department 4 RAL – 21 st January 2009

ATLAS level-1 calorimeter trigger 400 Mbit/s Jet / ET (JEP) Analogue Receivers Analogue tower sums (~7200) To CTP DAQ/Ro. I Pre. Processor (PPr) 400 Mbit/s DAQ 1 Gbit/s e/ , /hadron Clusters (CP) To CTP DAQ/Ro. I 1 Gbit/s Readout Driver (ROD) Real-time signal (data) path Tony Gillman – Particle Physics Department To ROS 5 Readout data path RAL – 21 st January 2009

ATLAS level-1 calorimeter trigger Ø Data are transported from detectors (calorimeters) to trigger processing electronics to generate ACCEPT signals to feed Central Trigger Processor Ø Signals undergo transformations at several stages in their journey… Tony Gillman – Particle Physics Department 6 RAL – 21 st January 2009

Analogue signal transmission Ø Calorimeter signals are of two types: Ø Liquid Argon calorimeter – Ø Bipolar, 75 nsec FWHM Ø Differential, ± 2 V max Ø Tile calorimeter – Ø Unipolar, 50 nsec FWHM Ø Differential, ± 2 V max Tony Gillman – Particle Physics Department 7 RAL – 21 st January 2009

Tile. Cal analogue trigger cable Transport medium: 16 shielded twisted-pair channels + global shield Characteristic impedance: 88 Ω 10 % Cable delay: ≤ 4. 76 nsec/m Inter-pair delay skew: <2. 5 nsec (70 m cable) Attenuation: -0. 06 d. B/m Crosstalk: <0. 2% (70 m cable) Bandwidth: 13 MHz at -6 d. B Tony Gillman – Particle Physics Department 8 RAL – 21 st January 2009

Imperfections in transmission lines Ø Pre-installation measurements on Tile. Cal analogue cables showed bad timing skew Ø Inter-pair skew (tpdn ≠ tpdm) – excessive, up to 20 nsec, but could be calibrated out Ø Intra-pair skew (tpdn+ ≠ tpdn-) – excessive, up to 28 nsec Ø This effect is totally unacceptable – result is to change the shape and amplitude of resultant differential signal, because of varying levels of dispersion Good Pair (tower 4, PMT 19) Bad Pair (tower 4, PMT 19) Resultant signal Positive signal Negative (inverted) signal Tony Gillman – Particle Physics Department 9 RAL – 21 st January 2009

S-parameter measurements Ø S-parameters characterise transmission-line performance in the frequency domain Ø For the suspect cable, S-parameters were measured for the 4 propagation modes – 1. differential-mode → differential-mode (signal attenuation) 2. common-mode → common-mode (signal attenuation) 3. differential-mode → common-mode (mode conversion) Ø Common-mode signal will radiate and couple to adjacent signal pair 4. common-mode → differential-mode (mode conversion) Ø Cable susceptible to radiation and resultant differential-mode signal will degrade S/N ratio Ø Mode conversion is caused by asymmetries in differential transmission-lines Tony Gillman – Particle Physics Department 10 RAL – 21 st January 2009

S-parameter measurements Ø First step was to measure characteristic impedance Z 0 of cables in two modes – common-mode and differential-mode and terminate cables under test in both ways using a single network Ø Measure transfer function of cables over frequency range up to 50 MHz in each of four modes using sine waves Test setup for common → common mode and common → differential mode measurements Tony Gillman – Particle Physics Department Test setup for differential → differential mode and differential → common mode measurements 11 RAL – 21 st January 2009

S-parameter measurements “Bad” Pair “Good” Pair Ø “Bad” pair exhibits severe attenuation at high frequencies → signal dispersion Ø Common → differential conversion is extremely large >15 MHz (compare with differential → differential mode!) Ø Conclusion: The entire batch of cables from this manufacturer was rejected Tony Gillman – Particle Physics Department 12 RAL – 21 st January 2009

ATLAS analogue trigger cabling Tony Gillman – Particle Physics Department 13 RAL – 21 st January 2009

ATLAS analogue trigger cabling Tony Gillman – Particle Physics Department 14 RAL – 21 st January 2009

Analogue → Digital conversion Ø Digital signals have many advantages over analogue signals (noise immunity, crosstalk, processing capability, …), so preferable to digitise detector signals as early as possible in signal chain Ø Analogue-to-digital converters (ADCs) are mixed-signal devices Ø Digital Output = Input signal / VREF = AIN / VREF x 2 N Ø AIN = Analogue Input Voltage Ø VREF = Vmax - Vmin (Reference Voltage) Ø N = No of output bits (resolution) Ø Analogue signal resolution = VREF / 2 N Ø This is the fastest type of converter, also known as a Flash ADC (FADC) Ø The delay between the clock and the digital output data appearing is latency Ø Low latency essential in many applications ATLAS level-1 trigger) (e. g. Clock Tony Gillman – Particle Physics Department 15 RAL – 21 st January 2009

ADC performance – some notes For an n-bit converter… Ø Dynamic range in d. B – 20 log (2 n -1) Ø Signal-to-Noise Ratio (SNR) = rms Signal / rms Noise (integrated over 1/2 clock period) Ø Several sources of noise – Ø Quantisation noise Ø Clock jitter Ø Electronic circuit noise Ø Fundamental limit on ADC performance is quantisation noise – LSB / sqrt 12 Ø SNR for ideal ADC = (6. 02 n + 1. 76) d. B Ø Nyquist limit – highest frequency component permitted ≤ ½ sampling frequency Ø If f(Ain) > ½ fs aliasing will occur → increased noise Ø Avoid aliasing by passing signal through low-pass filter before ADC comparators Tony Gillman – Particle Physics Department 16 RAL – 21 st January 2009

ADC performance – timing jitter Ø Clock jitter leads to aperture uncertainty Ø For a sine wave signal (V = A sin wt) → d. Vmax = 2 p A f dt Ø Aperture uncertainty therefore translates to a noise source, degrading the ADC resolution for high-frequency signals Ø Magnitude scales with the input signal frequency Ø The effect only becomes significant if dt > (2 n p f)-1 Ø The demands on clock jitter are very severe… ADC resolution Input frequency 44. 1 k. Hz 192 k. Hz 1 MHz 100 MHz 8 28. 2 ns 6. 48 ns 1. 24 ns 124 ps 12. 4 ps 10 7. 05 ns 1. 62 ns 311 ps 31. 1 ps 3. 11 ps 12 1. 76 ns 405 ps 77. 7 ps 7. 77 ps 777 fs 14 441 ps 101 ps 19. 4 ps 1. 94 ps 194 fs 16 110 ps 25. 3 ps 4. 86 ps 486 fs 48. 6 fs 18 27. 5 ps 6. 32 ps 1. 21 ps 121 fs 12. 1 fs 24 430 fs 98. 8 fs 19. 0 fs 1. 9 fs 190 as Tony Gillman – Particle Physics Department 17 RAL – 21 st January 2009

ADC performance – ENOB Ø Overall effect of aperture uncertainty is to reduce the Effective Number Of Bits (ENOB) of the ADC at high frequencies Ø N. B. An n-bit ADC will not resolve to n bits at its full analogue bandwidth unless clock jitter is kept below these limits Tony Gillman – Particle Physics Department 18 RAL – 21 st January 2009

Digital signal transmission Ø To transfer parallel data between sub-systems, convert to serial bitstreams to reduce the number of data paths and connector pins – increases reliability (but also latency!) Ø Serialising-deserialising (Ser. Des) chipsets can drive serial bitstreams at ~Gbit/s rate Ø Very common technology for serial links is Low-Voltage Differential Signaling (LVDS) Cable chosen for trigger – shielded Twin-ax (2 parallel cores – Z 0 = 100 W) Ø Many advantages: Ø Ø Ø Low-voltage power supplies Good noise immunity Low power dissipation Small signal swing → high data rates “Gigabits at Milliwatts” Tony Gillman – Particle Physics Department 19 RAL – 21 st January 2009

Eye patterns – digital data Source Tony Gillman – Particle Physics Department 20 Destination RAL – 21 st January 2009

Pre-compensation techniques Ø Adding a passive pre-compensation network (high-pass filter – CR or LR) to the LVDS driver outputs boosts the highfrequency components of the signal to compensate for the cable dispersion No pre-compensation LR pre-compensation N. B. overshoot Tony Gillman – Particle Physics Department 21 RAL – 21 st January 2009

ATLAS Pre. Processor Module ANALOGUE MCMs Digital data outputs DIGITAL Processor ASIC Flash ADCs Signal flow LVDS Serialisers Analogue signal inputs Signal flow Tony Gillman – Particle Physics Department 22 RAL – 21 st January 2009

Beware!!! Ø Installing Cu signal cabling can produce unexpected effects – Cable Discharge Event (CDE) Ø Static electricity on the jacket material of the cable induces a charge in the cable wires Ø Mechanisms – Ø Tribocharging (friction), produced as cables are pulled across surfaces Ø Electromagnetic fields can induce charge build up on cables, e. g. from electronic light ballasts Ø This may have been an issue for our 8000 LVDS cables installed under-floor between racks Ø As a precaution, we “discharged” cables after installation but before connecting any modules Ø N. B. This is another reason why using fibre-optic cabling has advantages Tony Gillman – Particle Physics Department 23 RAL – 21 st January 2009

Optical fibres Ø Cylindrical dielectric waveguide transmitting light along its axis by total internal reflection, consisting of a core covered by a sheath of cladding (ncore > ncladding) Ø As an alternative to Cu cabling for digital data transmission, it has many benefits – Ø Huge bandwidth Ø Immunity from EMI, ground-loops and crosstalk Ø Small volume for cable plant Ø Two types available – Multi-mode and Single-mode (usual material is silica) – Ø Multi-mode fibres – large core diameter (few tens of mm) allows multiple path lengths → intermodal dispersion limits Bandwidth x Distance product Ø Reduce intermodal dispersion by using graded-index silica – transit time variations → zero Ø Single-mode fibres – small core diameter (few mm) forces lowest-order (axial) mode, low dispersion → high Bandwidth x Distance product Ø Propagation delay ~ncore / c (~5 nsec/m – similar to Cu cable) Tony Gillman – Particle Physics Department 24 RAL – 21 st January 2009

Optical fibres – some available types Ø Step-index Multi-Mode fibres – Ø Ø Cheap Large core diameter → easy to couple light in/out High intermodal dispersion → low bandwidth Suitable for short links and low data rates Ø Graded-index Multi-Mode fibres – Ø Large core diameter → easy to couple light in/out Ø Reduced intermodal dispersion → increased bandwidth Ø Suitable for medium-range links/low data rates or short links/medium data rates Ø Step-index Single-Mode fibres – Ø Small core diameter → harder to couple light in/out Ø Wide bandwidth Ø Suitable for long-range links and high data rates Tony Gillman – Particle Physics Department 25 RAL – 21 st January 2009

Optical fibres – ATLAS level-1 trigger Ø Data transmitted to level-2 trigger and DAQ via Readout Driver modules (RODs) – distance ~10 m, total bandwidth >250 Gbyte/s Ø Chosen to use Multi-Mode fibres driven by laser diode transmitters (Infineon) operating at 850 nm, mounted on trigger modules Ø Total no of fibres feeding Readout Driver modules (RODs) ~320 56 mm Ø Transmitters are driven from Agilent G-link transmitters at 960 Mbaud (800 Mbit/s) Ø Receivers are dual Stratos devices mounted on 20 RODs Tony Gillman – Particle Physics Department 26 RAL – 21 st January 2009

ILC Vertex Detector Ø International Linear Collider will be an accelerator ~35 km long colliding bunches of e- and e+ at energies of 500 Ge. V – physics to complement that from the LHC Ø VXD will be based on Si detectors e. g. CCDs – forming ladders Tony Gillman – Particle Physics Department 27 RAL – 14 th February 2007

ILC Vertex Detector Ø 5 concentric barrels of ladders, on radii ranging from 15 mm - 60 mm Ø Thickness <0. 1% X 0 per barrel (target) Ø ~109 pixels – each 20 m Ø ILC will generate many spurious hits from beamstrahlung during bunch crossings Ø To minimise these background hits, CCDs must be read out quickly – Ø Readout time of 50 s for inner barrel (highest background hit density) Ø Readout time of 250 s for each outer barrel (lower background hit density) Tony Gillman – Particle Physics Department 28 RAL – 14 th February 2007

Background hit rates Ø Accelerator beam parameters – Ø ~1 msec bunch-train Ø 337 nsec inter-bunch gap Ø 5 Hz repetition rate (200 msec dead-time) Barrel Radius no CCD dimensions CCDs Ladders Readout per clock time ladder barrel rate Background hits per bunch-train 1 15 mm 100 mm 13 mm 1 8 50 MHz 50 sec 760 K 2 26 mm 125 mm 22 mm 2 8 25 MHz 250 sec 370 K 3 37 mm 125 mm 22 mm 2 12 25 MHz 250 sec 140 K 4 48 mm 125 mm 22 mm 2 16 25 MHz 250 sec 30 K 5 60 mm 125 mm 22 mm 2 20 25 MHz 250 sec 30 K Tony Gillman – Particle Physics Department 29 RAL – 14 th February 2007

Readout data volumes Ø So how much data will the VXD generate? Ø Total no of pixels clocked out during each bunch train ~4. 109 Ø To read out every pixel (assuming ≤ 1 byte/pixel) raw data volume ~20 Gbyte/s Ø This is unnecessary, most pixels are empty – only ~0. 5% occupancy Ø Sparsify data in real-time in Readout chips Ø Digitise signals in on-chip ADCs (5 bits OK) Ø Look for 2 x 2 pixel clusters with signal >cluster threshold → 6 bytes per cluster 26 bits (h-f addressing) 20 bits for 4 x 5 -bit ADC values 2 spare bits – parity, etc Ø 20 Gbyte/s → 40 Mbyte/s Tony Gillman – Particle Physics Department 30 RAL – 14 th February 2007

Data acquisition task Ø Total sparsified data volume per bunch train ~8 Mbytes (~40 Mbyte/s) Ø To read this out in real-time requires peak data transfer rate >8 Gbyte/sec Ø Readout chips require de-randomising FIFOs → reduce average data transfer rate Ø Provide each Readout chip with primary memory to store sparsified data (+ address tags) Ø ~1 Mbyte/CPR (Barrel 1) → ~10 Kbyte/CPR (Barrel 5) Ø Read data out to DAQ during 200 msec dead-time after each bunch train Ø Total sparsified data rate from VXD ~40 Mbyte/s (split between ±h) Tony Gillman – Particle Physics Department 31 RAL – 14 th February 2007

Data collection Ø Many ways to collect the data from all CPRs – this is only one possibility Ø Empty CPR primary memories sequentially at 50 MHz on to byte-wide ring-buses at ends of each barrel Ø Serialise the data from each ring-bus at 400 Mbit/s and drive differential LVDS signals (or optical links) into 2 DAQ cards (±h) Ø DAQ cards de-serialise the LVDS data, combine the 5 data streams, re-format, assemble and store the data for the entire bunch crossing (taking ~80 msec) Ø 2 optical fibres/DAQ card export data to main DAQ + import readout control signals Tony Gillman – Particle Physics Department 32 RAL – 14 th February 2007

“Galvanic” links need space… ~10% of digital data links of ATLAS L 1 Calo trigger in a Birmingham test-rig Small part of ATLAS L 1 Calo data link system installed underground Ø Data from ILC Vertex Detector could be transported on a single fibre! Ø Upgraded L 1 Calo for Super-LHC will probably use fibres for all data transport Tony Gillman – Particle Physics Department 33 RAL – 14 th February 2007