Super B Factory DAQ Trigger Online Paris Super
Super B Factory DAQ, Trigger & Online Paris Super B Factory Workshop 9. 5. 2007 S. Luitz and G. P. Dubois-Felsmann Super B Workshop S. Luitz and G. P. Dubois-Felsmann 1
Outline n n n n n A Caveat Rate and Event Size Estimates FCTS and Data Flow Event Builder and Level-3 Trigger Farm Logging and Data Quality Monitoring Control Systems & Logbook Configuration Management Hardware Reuse, Cost and Labor Estimates R&D Projects Summary Super B Workshop S. Luitz and G. P. Dubois-Felsmann 2
A Caveat n As you will certainly notice right away, our Super. B CDR DAQ design is heavily based on the existing Ba. Bar DAQ & Online system n Nice: n n Build on the Ba. Bar success Ba. Bar is the source of the most reliable performance and effort data available to us Not a radically new design, we think we understand most issues But keep in mind: n n n The design parameters are fairly challenging for an HEP DAQ system There may be other ways to build such a system (e. g. with LHC-style technology) Not a radically new design, we think we understand most of the issues but there may be some unforeseen surprises Super B Workshop S. Luitz and G. P. Dubois-Felsmann 3
Looks like Ba. Bar, doesn’t it? Super B Workshop S. Luitz and G. P. Dubois-Felsmann 4
Rate and Event Size Estimates/Assumptions n Event size n n Ba. Bar: ca. 35 k. Byte average Super. B: educated guess based on Ba. Bar: 75 k. Byte average n n Possibly smaller cells in DCH More samples needed to detect pile-up and overlapping events Higher occupancy due to background events Trigger Rates – Maintaining the Ba. Bar Open Trigger Philosophy n L 1 -Accept n From Ba. Bar: n n n n ~20 nb (Ba. Bar physics filter output) 20 k. Hz @ 1 e 36 ~50 nb Bhabhas (Ba. Bar acceptance) 50 k. Hz @ 1 e 36 10% scaled backgrounds of Ba. Bar ~25 k. Hz Operational Ba. Bar experience: 50% headroom desirable Would require L 1 and Front End DAQ to handle 150 k. Hz L 1 -Accept rate Bhabha veto could bring down L 1 -A rate (with headroom) to ca. 100 k. Hz Assume Bhabha veto for now L 3 -Accept n n Assume ~20 nb + ~5 nb for monitoring triggers 25 k. Hz @ 1 e 36 L 3 accept rate Moderately optimistic extrapolation from Ba. Bar Super B Workshop S. Luitz and G. P. Dubois-Felsmann 5
Trigger n Foresee Ba. Bar-like trigger structure: n Level-1 n n n Level-2 n n n Trigger in data path acting on partial event information Currently not foreseen (event sizes are small!) Level-3 n n n Pre-event-readout trigger based on drift chamber and calorimeter information Level-1 -Accept initiates event readout Software trigger running on server farm, acts on complete events. Refines Level-1 calculations, allows more complex operations Level-4 n “Offline” filter Super B Workshop S. Luitz and G. P. Dubois-Felsmann 6
Fast Control and Timing System n n Similar to Ba. Bar, but faster fiber links to front-end electronics (2. 5 GBit/s) Design based on “CE” modular DAQ system currently developed at SLAC n FPGA/System-On-A-Chip n n Feature extraction as close to front-ends as possible n n PPC general-purpose CPUs VHDL-style “DSPs” Various link options, including 10 Gig. E and 2. 5 GBit/s fiberoptic serial links Proof-of-concept in Ba. Bar DCH electronics “No” mandatory dead-time between triggers allowed n n Frontend readouts that can “keep up” with a (very short) minimal allowed command spacing Trigger queuing and fully addressable front end readout buffers Super B Workshop S. Luitz and G. P. Dubois-Felsmann 7
Event Build and Level-3 Trigger Farm n Ba. Bar-style network event builder and Level-3 farm n n 10 Gig. E outputs of CE modules are fed into a commodity network switch Event contributions are sent to Level-3 trigger farm nodes n n n 100 k. Hz x 75 k. Byte 7. 5 GByte/s or 60 GBit/s n n n Can be handled by current generation of network switches The Ba. Bar Cisco 6500 -720 switch could handle this rate Will require at least 120 farm nodes for networking reasons n n n Farm nodes connected to the switch by 1 Gig. E Take advantage of cheap (and possibly long) copper links. Copper 10 Gig. E is limited to a few meters. Use lightweight UDP/IP protocol with simple retransmit Most of UDP protocol done by NIC Can’t utilize 1 Gig. E links more than 50% 8 CPU cores per node would allow ~5 ms / event / CPU Level-3 output rate: 25 k. Hz 210 Hz per node Super B Workshop S. Luitz and G. P. Dubois-Felsmann 8
CE-Module Modular DAQ system design currently developed at SLAC for LSST, Peta. Cache and other projects Super B Workshop S. Luitz and G. P. Dubois-Felsmann 9
Logging and Data Quality Monitoring n Logging n n n Log to RAID-1 (mirrored) disks local to farm nodes, 2 -4 TByte/node 16 MByte/s (210 Hz x 75 k. Byte) per node Send data from farm nodes to archival storage and further processing Rate requirement per node: 16 MByte/s write, 2 x 16 MByte/s read Within reach of current disks Each farm node holds part of a run (“Runlet”) n n n Runlets expected to be preserved, probably no further merging Bookkeeping system will need to deal with this Data quality monitoring n n Foresee additional farm of ca. 40 machines to run fast online data quality monitoring Alternatively data quality monitoring application could run on the L 3 farm (e. g. using low-overhead VM technology). Interesting if excess CPU available. Histogram collection via DHP-like distributed histogramming mechanism GUI/Web based display Super B Workshop S. Luitz and G. P. Dubois-Felsmann 10
Control Systems & Logbook n Experiment (“Run”) Control n n Operator interface for DAQ Sequences DAQ configuration, calibration, partitioning, etc. Automate operations as much as possible Detector (“Slow”) Control n n n Control detector and detector support systems Monitor and record detector and environment conditions Based on toolkit that provides interface to hardware n n n Ba. Bar uses EPICS Depending on hardware, a different system may be more appropriate Should share a timebase with the FCTS Unified operator interface / GUI Electronic Logbook n Integrated with Experiment and Detector Control Super B Workshop S. Luitz and G. P. Dubois-Felsmann 11
Configuration Management & Release Control n n All hardware and software configurations defined in configuration database Software release management and tracking n n What was running where at any given time in any part of the DAQ and Online system With more “traditional software functionality” implemented in FPGAs, FPGA firmware is considered “software” and must be included in the configuration and release management scheme Super B Workshop S. Luitz and G. P. Dubois-Felsmann 12
Hardware Re-usability & Cost Estimates n Most Ba. Bar Online system hardware not reusable for Super B factory: n n DAQ and Trigger – for obvious reasons Computing & Networking n n n Need to take advantage of Moore’s Law Useful life of commodity computing 3 -4 years Basis for online computing cost estimate n Ba. Bar experience with 2 upgrades shows: n n “Boxes don’t get cheaper, they get faster” “A farm node always costs $3 k, now it has 4 cores” Super B Workshop S. Luitz and G. P. Dubois-Felsmann 13
Labor Estimates n What about re-using online software? n Concerns: n Tied to hardware May be sub-optimal n Very specific optimizations for Ba. Bar operations n n n n n Apply lessons learned But a lot of code will need to be written from scratch Given the challenges, we estimate the Super B Factory effort to be comparable to the initial Ba. Bar online computing effort Apply lessons learned to development model n n n E. g. detector controls In-depth expertise may no longer be available Different expertise and tools available in LHC era Documentation sub-optimal or non-existent, would require reverse-engineering Concepts and frameworks can (and should) be re-used n n As recent optimizations of feature extraction codes have shown E. g. feature extraction code to be developed by “professionals” More core effort for e. g. the slow controls system to avoid the “ 3 different drivers for the same hardware” syndrome 80 -90 FTE-years (based on Ba. Bar effort estimates) Super B Workshop S. Luitz and G. P. Dubois-Felsmann 14
R&D Projects – further study n n Detailed evaluation of channel counts, and per-channel DAQ requirements Study backgrounds and occupancies and their effect on event sizes Study attainable performance of L 1 - Bhabha veto Assess cost of extending L 1 -Accept capacity to 150 k. Hz n n n Headroom, no Bhabha veto, later lumi upgrades Investigate consequences of overlapping events, triggers and trigger queuing Queue modeling for data path to determine buffering requirements Study adaptation of the current “Level-4” Ba. Bar physics filter to Level-3 quantities Look at control system alternatives to EPICS Super B Workshop S. Luitz and G. P. Dubois-Felsmann 15
Summary n n n “Ba. Bar++” design of Online and DAQ seems feasible for Super B Factory Hardware re-use very limited Software re-use needs careful consideration n In many cases a redesign with lessons learned probably the better approach All challenges can (probably) be solved with technology available today – just at a higher cost More R&D in many areas needed for more detailed design Super B Workshop S. Luitz and G. P. Dubois-Felsmann 16
- Slides: 16