DUNE FD DAQ Firmware David Cussans DUNE UK

DUNE FD DAQ Firmware David Cussans DUNE UK DAQ Meeting, Daresbury Lab 11/Dec/19

Outline • Where is it? - Where does Firmware fit into DUNE FD DAQ? • What is it? - What does it do? • How are we making it? - What are the functional blocks? - Who is writing them? - (Timing covered in separate talk) 2 David Cussans | DUNE DAQ Firmware

Firmware: Where it? You are Here 3 David Cussans | Firmware Overview

Firmware: Where it? You are Here (part of upstream DAQ) 4 David Cussans | Firmware Overview

Firmware: Where it? Buffer You are Here (part of upstream DAQ) 5 David Cussans | Firmware Overview

Firmware: Where it? Trigger You are Here (part of upstream DAQ) 6 David Cussans | Firmware Overview

Firmware: What’s In it? • Concentrate on SP TPC - DP TPC will need different handling, but much development already done. - Photon Detection Systems produce much less data than TPCs • Other systems need different “front end”, but then same functionality - E. g decompress DP TPC - PDS zero suppress data 7 David Cussans | Firmware Overview

Firmware: What’s In it? • Working to demonstrate in firmware: - hit finding ( main focus at the moment ) • + pedestal subtraction and filtering - Buffering • O(10 s) pipeline • O(100 s) non volatile • Sizes driven by Super Nova requirements - Compression (to maximize effective size of buffer) - Receiving trigger messages, transmitting event fragments to builder • Debate about where to put functionality: server or FPGA. - Do both then choose. - Will feed into technology choice 8 David Cussans | Firmware Overview

Buffer/Pipeline: Event , Super Nova • Two buffers - O(10) seconds in RAM. • Pipeline – all data passes through - O(100) seconds in non-volatile storage • Activated when possible Super Nova detected. Stores all data • Combination with O(10 s) buffer gives view before SN detected. Super Nova SN Detected O(10 s) pipeline O(100 s) non-volatile storage 9 David Cussans | Firmware Overview

Embarrassingly Parallel • Ten fibres per APA @ 9. 6 GBit/s - 12 -bit ADC samples @ 2 MSample/s - Each fibre 2560 wires. - Induction and collection wires on same fibre • Treat each fibre in parallel - How safe is this? Is it sufficient to perform common mode noise subtraction on a per-fibre basis? 10 David Cussans | Firmware Overview

Firmware: What’s in it? 11 David Cussans | Firmware Overview

Firmware: Reorganize • Data on fibres has complicated encoding. • Francesco Gonnella: Birmingham, David Cussans: Bristol 12 David Cussans | Firmware Overview

Firmware Framework • Dave Newbold, RAL • 2 MSample/s from ADC, but logic can run much faster - Run processing logic ~ 200 MHz - Data arrives packets containing all channels for a single “time tick” - Rearrange into packets with multiple time ticks for a single channel. - Save/restore state so processing transparent to block size 13 David Cussans | Firmware Overview

Firmware: Pedestal Subtraction, Filter • Kostas Manoulopolous: RAL 40 pipelines Per APA 14 David Cussans | Firmware Overview

Firmware: Hit Finding • Kunal Kothekar: Bristol 15 David Cussans | Firmware Overview

Firmware: Lossless Compression • Patrick Dunne, Abbey Waldron: Imperial 16 David Cussans | Firmware Overview

Firmware: Buffer Management • Erdem Motuk: UCL One DRAM interface Per APA 17 David Cussans | Firmware Overview

Firmware: Non-Volatile Buffer • Roy Wastie: Oxford, Commercially written firmware: TBD Two NVMe SSD Per APA 18 NVMe Interface David Cussans | Firmware Overview

Firmware: Build Environment • Alessandro Thea, RAL • Challenging to build firmware: - Traditional approach: One engineer and their Workstation • We have distributed team • Mix of framework , user code • Multiple hardware platforms • Tools developed within LHC to facilitate collaborative firmware development - CMS IPBus-build (ipbb) • Felix firmware build tools not yet configured to facilitate distributed development ( e. g. separation of framework and processing code) 19 David Cussans | Firmware Overview

Firmware - a work in progress • Standalone hit finding in ZCU 102 , KU 105 eval boards - No input from WIB yet • Fully integrated with FLX 712 - Missed December deadline - … but run extended to Easter • Delay gives time to move effort to hit finding 20 David Cussans | Firmware Overview

Current Work • Working to integrate hit finding with Felix FLX 712 PCIe board - Demonstrate hit finding in Firmware on Proto. DUNE • Good source of Dune-Like-Data • Full chain ( WIB Offline ) present • Integrating hit finding code with Felix code has uncovered unexpected features in both • Xilinx Ultrascale fabric ( on Felix ) slightly slower than Ultrascale+ fabric (on Zynq eval board used for hit-finding development) • FLX 712 does not have RAM or NVMe interfaces. Developing buffering firmware on COTS FPGA boards in parallel. - Will likely use COTS boards to demonstrate FPGA buffering in Proto. DUNE-2 – needs further discussion. 21 David Cussans | Firmware Overview

Towards Proto. DUNE 2 • Demonstrate functionality in Proto. DUNE 2 • Build relationship with ATLAS Run 2 Felix developers - Felix firmware already builds for Ultrascale+ - Join Felix firmware team to enhance Felix tools • Framework better able to host project specific functionality in • Hardware used - under discussion. One model: - COTS FPGA board then new Felix board • FLX 712 continues to run as-is 22 - Felix firmware developers moving to Ultrascale+ ( then Versal) for future developments - FLX 712 not widely used in DUNE outside Proto. DUNE. Support limited. - Minimize number of different hardware platforms. David Cussans | Firmware Overview

Summary • Working to demonstrate Upstream DAQ functionality - Trigger primitive production • Moving effort to this to recover from slippage - O(10 s) Buffer - O(100 s) Buffer - Suitable firmware design flow • Focus on hit finding in Proto. DUNE at DAQ sprint • Then focus moves to Proto. DUNE 2 - Demonstration of DUNE functionality. 23 Presenter Name | Presentation Title