DUNE DAQ Firmware Development David Cussans 26 th

DUNE DAQ Firmware Development David Cussans, 26 th September 2018

Introduction • For SP-TPC • “Pre-processing” in FPGA • Lossless compression • Trigger primitive generation • Filtering , pedestal subtraction • Hit finding • Buffering • “ 10 -second buffer rolling buffer” • trigger latency, S. N. pre-trigger data • “ 100 -second S. N. buffer” • Written to “few times a month” (slow readout) 2 David Cussans | Collaboration Meeting, Sept ‘ 18

Hardware Layout • Indicates topology , not geometry …. David Cussans | Collaboration Meeting, Sept ‘ 18

Hardware Layout • E. g. Host pre-processing module on Felix …. David Cussans | Collaboration Meeting, Sept ‘ 18

Hardware Layout • …. Or x. TCA David Cussans | Collaboration Meeting, Sept ‘ 18

DPM status / test Plan Babak Abi 6

7 DPM Design current status includes : • ZYNQ: XCZU 15 EG-1 FFVB 1156 E • DDR 4 : MT 40 A 1 G 16 WBU-083(twin die) in two separate banks • M. 2 NMVe: 512 GB on DPM connected to PCIe Gen 2 on top of DDR 4 s • 48 samtec x 10 Gb/S line(24 GHT TX and 24 GTH RX connecte to 2 connecter) • PCB stack is 20 layers in small form factor (89 mm x 110 mm). Using PTH via and FR 408 for reducing the cost. It is very 1) 4 x DDR 4 (8 GB) connected to PL. 2) 4 x DDR 4 (8 GB) connected to PS • • • samtec M 2 NMVe DDR 4 PL ZYNQ DDR 4 PS complex pcb for signal integrity checks and PCB routing. So it needs massive and extensive signal integrity simulation. The GTH Full simulation (3 D full EM wave solver, Hyperlinynx ) and compliances for COM (10 GBASE-KR) and DER 0 = 1 e-012 passed. PCIE gen 2 also passes compliances checks. Last tuning on DDR 4 for assuring the maximum performances are in progress, almost done. In negotiation with PCB manufacturer for assurance of impedance controlled PCB tolerance 5% , fixed via structure and impedance test coupons, probe test, assembly line ……… samtec SSD PCIe We might have to make some changes to PCB to make it easier for manufacturer for a better impedance control during the production, this means a bit more work (and of course time).

8 DPM tests plan • Hardware, independent of firmware progress but if any art of firmware would be ready could help here too. • • 1. 2. 3. After final semi-review (and probably a small design re-spin) submitting to manufacturing in UK. We are aiming for 6 board prototyping in 2 steps of 3+3 boards. DCs and mechanical checks Performance tests: DDR 4 TX/RX 10 Gb. E and bit rate error PCIe SSD What is the test platform, • • COBs Oxford and RAL and UCL has the current generation and SLAC has also the new generation, capable of checking items 1 and 3 and partially 2. Carrier board, for access to all TX/RXs and should be able to do full tests. • Two variation of carrier boards : 1. SLAC based, I will start to working on PCB after DPM submission. 2. Minimum board just provide the PS and all TX<->RX are looped in samtec cons.

9 48 TX/RX 10 GB/s

Firmware Development

Firmware Layout • Block development divided between individuals/groups David Cussans | Collaboration Meeting, Sept ‘ 18

Firmware Development to TDR • Use off-the-shelf evaluation boards • DPM hardware fabricated and tested in parallel • Stages: 1. Simulation 2. Exercise firmware entirely in Eval board • Load short buffer with input data, play through algorithms, repeat. 3. Feed data to Felix 4. Feed data from external source • 12 Need to “sanitize” proto. DUNE data to remove “stuck bits” David Cussans | Collaboration Meeting, Sept ‘ 18

Framework • Prototype framework exists • Already synthesizable • Based on CMS “IPBus” to write/read/control over Gbit/s Ethernet. • Interface between blocks: • Being documented. • Dave Newbold. Due next week. • AXI-4 Stream with extensive use of user-bits • Will standardize on ZCU 102 and KCU 105 FPGA Eval boards • Use SFP for Gbit/s Ethernet interface to PL (ZCU 102 Phy coupled to PS) 13 David Cussans | Collaboration Meeting, Sept ‘ 18

Col. Data Reception • Leading up to TDR will use proto. DUNE WIB data format • Use proto. DUNE code from SLAC • Status: • Needs integrating with development framework. 14 David Cussans | Collaboration Meeting, Sept ‘ 18

Data Blocking and formatting • Read from data source • Short buffers (not streaming WIB data) , if stand-alone. • Col. Data reception ( streaming WIB data) • Cut into fragments ( length parameterizable ) and pass to processing units • Algorithms see uninterrupted stream of data – fragment size has no effect. • Status: • Prototype framework exists. 15 David Cussans | Collaboration Meeting, Sept ‘ 18

Filtering for hit-finding • Kostas Manolopoulos(RAL) • Status: 16 David Cussans | Collaboration Meeting, Sept ‘ 18

Hit Finding • Firmware implementation • Kunal K. (Bristol) • Comparison with Monte-Carlo simulation • Joel Greer (Ph. D, Bristol) and Jim Brooke (Bristol) • Status: 17 David Cussans | Collaboration Meeting, Sept ‘ 18

Compression • Inspired by work on proto. DUNE data by Ivan Furic, Dave Newbold (Bristol) has make an initial look at compression based on Fibonacci encoding of sample differences. • Much lower in resource usage than table based compression ( Huffman, APE, etc. ) • Similar performance for white noise on a low frequency background ( … which is what initial proto. DUNE data indicated) • Status: • Implemented for a different neutrino experiment. Needs porting to DUNE framework and testing 18 David Cussans | Collaboration Meeting, Sept ‘ 18

Buffer Management • Erdem Motuk (UCL) • Will assume • data selection commands ordered in time. • Granularity of request same as fragment • Status: • Working on hardware interface to DDR 4 19 David Cussans | Collaboration Meeting, Sept ‘ 18

Writing to SSD • Two approaches: • Using FPGA-Processor subsystem (Zync ARM cores) • Programming logic only • Roy Wastie (Oxford) constructing set-up attaching NVME M. 2 SSD to a Zync Ultrascale+ prototyping board. • Already done at SLAC • RAL TD purchasing evaluation licence of interface to SSD implemented in PL. • In discussions with UK firm to write SSD PL-only interface. • Will have source code – multi-site 20 David Cussans | Collaboration Meeting, Sept ‘ 18

Vertical Slice Test

Vertical Slice Test • Firmware only one component • Software, • firmware/software comparison, • Link to physics performance • Iterative • Start with firmware running entirely in single Eval board • Finish with WIB data flowing through system to Felix and host PC. 22 David Cussans | Collaboration Meeting, Sept ‘ 18

Version Management

Version Management • Use CERN Git. Lab • Access control • Continuous integration facility for firmware synthesis • Project name (preliminary) DUNE-SP-TDR-DAQ • https: //DUNE-SP-TDR-DAQ/framework • (Don’t want to collect multiple sub-projects unconnected with DAQ firmware. ) 24 David Cussans | Collaboration Meeting, Sept ‘ 18

Summary • Firmware for processing DUNE SP-TPC data divided into blocks and distributed • Filtering/Hit finding • Buffer management • Super Nova long buffer • Will use off-the-shelf Eval boards in run up to TDR • DPM used when ready. • Vertical slice needs planning and involvement of multiple groups. 25 David Cussans | Collaboration Meeting, Sept ‘ 18

Backup 26

2 7 PCB stack up 20 layers – no back drill samtec

2 8 PCB stack up 20 layers – no back drill 20 layers but still very busy board samtec

2 9 GTH lines Compliance: COM (10 GBASE-KR) All passes 10 GBASE-KR

3 0 GTH lines Compliance: plots 1 • Insertion loss and Fitted attenuation are important

3 1 PTH VIA full EM 3 D simulation 1 Many simulation with different via structure regards to PCB stack is done to get the best via for each GTH line. Next slides are an example of via Full EM 3 D study and modification for one of GTH line

3 2 PTH VIA full EM 3 D simulation 2 Ser. Des. Results_20180815 -0200 Before VIA modification C: workdpm_lmb_zynq_ultrascale_c 01OutputSer. Des. Results_20180817 -0157 After VIA modification

3 3 PTH VIA full EM 3 D simulation 3 Before VIA modification After VIA modification

3 4 PTH VIA full EM 3 D simulation 4 Before VIA modification After VIA modification

3 5 DDR 4 timing and signal integrity check and plots

3 6 DDR 4 timing and signal integrity check and plots

3 7 DDR 4 timing and signal integrity check and plots

3 8 DDR 4 timing and signal integrity check and plots