READOUT LINK Filippo Costa Filippo Costa readout link

READOUT LINK Filippo Costa

Filippo Costa - readout link 2

READOUT LINKS Filippo Costa - readout link 3

LEGEND SIU Source Interface Unit CMC Common Mezzanine Card HOLA High-speed Optical Link for Atlas TELL 1 Trigger ELectronics Level 1 DDL Detector Data Link ROL Readout Link SLINK 64 Simple Link Interface Gb. E Gigabit Ethernet RORC Readout Receiver Card ROBIN Readout Buffer INput module FRL Front end Readout Link NIC Network Interface Card CONVERTER CARD READOUT LINK Filippo Costa - readout link DAQ RECEIVER CARD 4

General picture FEE Converter card readout link DAQ receiver card PC or crate (VME, Compact PCI) Filippo Costa - readout link 5

Filippo Costa - readout link 6

Filippo Costa - readout link 7

Filippo Costa - readout link 8

Filippo Costa - readout link 9

IT’s Filippo Costa - readout link 10

THE Filippo Costa - readout link 11

SAME Filippo Costa - readout link 12

THING (almost) Filippo Costa - readout link 13

Filippo Costa - readout link 14

~2500 CHF (~1300 per link) Filippo Costa - readout link 15

~2500 CHF (~1300 per link) ~3300 CHF (~1100 per link) Filippo Costa - readout link 16

~2500 CHF (~1300 per link) ~3300 CHF (~1100 per link) 1550 CHF (~800 per link) Filippo Costa - readout link 17

ALICE ~2500 CHF (~1300 per link) ATLAS ~3300 CHF (~1100 per link) CMS 1550 CHF (~800 per link) Filippo Costa - readout link LHCb > 6000 CHF (~300 per link) 18

# links protocol Throughput Flow control Main buffering 1600 Slink optical 160/200 MB/s Yes On board. 3 x 64 MB 500 DDL optical 200 MB/s Yes On the PC memory 8 GB 600 Slink 64 copper 400 MB/s Yes On board. 64 KB 400 Ethernet NO On board. 128 KB 4 x 1 Gb/s Filippo Costa - readout link 19

No general software tools to debug data flow. Filippo Costa - readout link 20

• S-Link is a that can be implemented in an FPGA (not resources demanding) Filippo Costa - readout link 21

• Very robust link. • Cost is pretty low. Filippo Costa - readout link 22

GOL Filippo Costa - readout link 23

IN GENERAL Filippo Costa - readout link 24

RUN 2 Filippo Costa - readout link 25

We are busy updating the experiment for you and will be back shortly • Faster readout and higher data throughput for TPC and TRD. • Higher data throughput between HLT to DAQ. Filippo Costa - readout link 26

• PROTOCOL UPGRADE: • faster DDL protocol from 2 Gb/s to 6 Gb/s. • HARDWARE UPGRADE: • INPUT: from 2 ch @ 2 Gb/s to 12 ch @ 6 Gb/s. • OUTPUT: from PCIe gen. 1 x 4 to PCIe gen. 2 x 8. • Current RORC firmware implemented on new hardware. • Same software framework, only driver upgrade to handle the new hardware. Filippo Costa - readout link 27

We are busy updating the experiment for you and will be back shortly • 64 -bits PCI technology , obsolete and causes problems for upgrading DAQ PCs. • Number of readout links will increase. • Reduce amount of RACK space. • Increase data throughput from ROS to HLT after LS 1. Filippo Costa - readout link 28

• Use of the ALICE CRORC, but is called ROBINNP. • HARDWARE UPGRADE: • INPUT: from 3 ch @200 MB/s to 12 ch @ 200 MB/s (6 Gb/s) • OUTPUT: from PCI 64 bit 66 MHz to PCIe gen. 2 x 8. • Bigger buffering capacity (up to 4 GB of DDR 2/3). • New firmware (the PPC present in the ROBIN will be moved out and implemented in software). Filippo Costa - readout link 29

Same hardware – different features ALICE ATLAS 12 ch @ 6 Gb/s 12 ch @ 2 Gb/s (in average, if needed 6 Gb/s) No use of internal memory Up to 4 GB of DDR 2/3 used to buffer events in the board Filippo Costa - readout link 30

ALICE CRORC FIRMWARE Push Buffer Descriptor #5 Buffer Descriptor #4 Buffer Descriptor #3 Buffer Descriptor #2 Software Buffer Descriptor #1 Receive Address FIFO Firmware Receive Report FIFO Receiver DMA controller DMA report #5 DMA report #4 DDL DMA report #3 DMA report #2 DMA report #1 PC memory, managed by PHYSMEM PCI bus Filippo Costa - readout link D-RORC 31

ATLAS ROBINNP firmware Filippo Costa - readout link 32

We are busy updating the experiment for you and will be back shortly • Hardware is aging (Myrinet) … needs for new components and replacements. • New components are not compatible with the current hardware. • Faster readout and higher data throughput, needed by trigger and HCAL. Filippo Costa - readout link 33

Filippo Costa - readout link 34

• The existing DAQ receiver card (FRL) stays in place to support legacy FEDs. • New FEROL card has 2 x 10 Gb/s and 2 x 6 Gb/s channels. • No CMC , but IPCORE, same interface to the FEE. • The new 6 Gb/s link has re-transmit feature. • 500 MB internal buffer. • 2 input links are treated independently and can share the same 10 Gb/s. • The second link 10 Gb/s will be used to receive 10 Gb/s in future (1 Input 1 Output) • 10 Gb/s reduced TCP/IP implemented in FPGA. Filippo Costa - readout link 35

Filippo Costa - readout link 36

ABORT/RST Filippo Costa - readout link 37

CONGESTION CONTROL Filippo Costa - readout link 38

Filippo Costa - readout link 39

The don’t have special plan for the upgrade during LS 1 … however … Filippo Costa - readout link 40

Look (again) what LHCb has done Filippo Costa - readout link 41

They have achieved goals that other experiments are planning in the future: • Common readout unit (TELL 1) used by all the detector groups. • Use of Ethernet protocol. Filippo Costa - readout link 42

RUN 3 Filippo Costa - readout link 43

LS 2 Online Upgrade ~ 2500 DDL 3 s 10 Gb/s (~1 TB/s in input) CLK ITS RORC 3 FLP TPC RORC 3 FLP TRD RORC 3 FLP EMCal RORC 3 FLP PHOS RORC 3 FLP TOF RORC 3 FLP Muon RORC 3 FLP FTP RORC 3 FLP EVB farm L 0 L 1 Trigger Detectors DAQ and HLT Filippo Costa - readout link 44

ALICE readout link SIU 3 RORC 3 SIU IP CORE TBD Det. Read-Out FPGA XILINX / ALTERA / ACTEL FPGA Custom DDL 10 Gb/s Ethernet @ 10 Gb/s PCIe over fibre GBT Filippo Costa - readout link 45

DDL 3 (UDP Ethernet) DDL optical fibre SFP+ 10 Gb/s embedded in the motherboard XILINX VIRTEX 6 Filippo Costa - readout link 46

DAQ+DCS proposal of sharing DDL 3 Detector Readout Electronics DCS data FPGA readout data taking in progress… Physics DATA DCS DATA Physics DATA Our proposal is to share the same physical link between the 2 data traffics: 99% PHYSICS data 1% or less, DCS monitoring data Filippo Costa - readout link 47

Detector link Readout electronic GBT TELL 40 10 Gb. E (production and early tests) 40 Gb. E, Infiniband or PCIe Reduced TCP/IP or advanced UDP Filippo Costa - readout link 48

CONGRATULATIONS TO ALL OF US !!!! Filippo Costa - readout link 49

Some remarks … • ALICE, ATLAS, CMS and LHCb have different physics goals, but from the readout link point of view they try to achieve the same thing, move data from A to B. Why things are so different? Requirements and data throughput are not the same, but features could be adapted, instead writing them from scratch. • Common effort to standardize the link and the electronics, ultimate goal: • common readout unit for detector groups, • use of commercial hardware and industrial-standard protocols. • How to increase the knowledge sharing between all the experiments groups? It is always a joy to share good experience with a protocol or a link, but maybe we could learn more from things that went wrong (and usually are not showed at conferences) Filippo Costa - readout link 50

Philippe Farthouat Benedetto Gorini Markus Joos Niko Neufeld Christoph Schwick Petr Zejdl Dominique Gigi Filippo Costa - readout link 51