Paving the DAQ road from NSCL to FRIB
Paving the DAQ road from NSCL to FRIB Giordano Cerizza LECM 2018 WG Session II This material is based upon work supported by the U. S. Department of Energy Office of Science under Cooperative Agreement DE-SC 0000661, the State of Michigan and Michigan State University designs and establishes FRIB as a DOE Office of Science National User Facility in support of the mission of the Office of Nuclear Physics.
Status of the Art @ NSCL § NSCLDAQ 11. 3 • Several updates and bug fixes § Spec. Tcl 5. 0 (with ROOT support) • Spectra - ROOT-based histogram displayer • ROOT tree created while unpacking data – it maintains same data structure • ROOT macro support – allows to run ROOT macros from the Spec. Tcl env § DDAS 2. 0 Pros: solid and stable framework for existing experiments Cons: analysis applications show aging limits as detector electronics and DAQ become more computationally expensive moving toward multi-core/GPU/FPGA-based software G. Cerizza LECM 2018, Slide 2
Towards FRIB § Plan to develop a new analysis framework more flexible at each stage Source • • Event Builder • • Raw Data NSCL DAQ Framework • Unpacker Storage • Configuration • User Defined 1 User Defined 2 … NSCL Analysis Framework Visualization All components can be parallelized Components can be individually enabled and disabled Framework is GUI controlled Framework will require a common set of (user) parameters and variables. Data can be written out of and read into each stage. Third-party libraries can be incorporated into user defined routines to increase functionality Visualization API will be defined to enable use of different visualizations (Spec. Tcl/ROOT is likely to be provided by default) (presented at the DAQ committee – March ‘ 18) G. Cerizza LECM 2018, Slide 3
How to Get There? § From single- to multi-core applications • 0 MQ – high-performance asynchronous messaging library » Benchmark tests: I/O speed test data reading + trace fitter for Pixie 16 digitizers • Spec. Tcl parallelization using 0 MQ as framework for data pipelining and processing (ongoing) § Local computing cluster for online analysis • Tested on “Lifetime Measurements within the N = 20 Island of Inversion” (Crider/Liddick e 16032) § Exploring GPU/FPGA programming at the HPC G. Cerizza LECM 2018, Slide 4
0 MQ Middleware: Algorithm (1) § Worker-driven load-balancing pattern REQ: request REP: reply t da Scanning a 3 GB NSCLDAQ evt file by reading variable-size chunks of data (I/O speed test*) t en s a do Ready! § Benchmark test 1 Sender ne Ready! 30 MB 4. 5 Worker 2 … Worker N 4 Time (s) Worker 1 10 MB 3. 5 Sink exit 3 2. 5 2 1. 5 Example: 10 k messages distributed over 8 worker threads 8 CPU cores 16 CPU cores 24 CPU cores 28 CPU cores 1 128 k. B 0. 5 0 1000000000 Memory chunk (k. B) * on XEON E 5 -2690 2. 60 GHz 28 x 2 CPU cores G. Cerizza LECM 2018, Slide 5
0 MQ Middleware: Algorithm (2) § Benchmark test 2: trace fitting for Pixie 16 digitizer Crider/Liddick (e 16032) – June 21 st -29 th Computing Cluster for Online Analysis (June 21 st to June 29 th) - 56 nodes x 8 cores Intel(R) Xeon(R) CPU E 5620 @ 2. 40 GHz G. Cerizza LECM 2018, Slide 6
Multi-threaded Spec. Tcl § Goal: distribute computational load (i. e. user event processors) over many cores to speed up analysis Multithreaded safe with N processors (0 MQ framework) Data. Source - single box (via inproc) - multi box (via tcp) TKRun. Control Buffer. Decoder Evt. Processor (…) Evt. Processor Pipeline 1 TCLAnalyzer event 1 event 2 event. N Multithreaded safe with N processors (0 MQ framework) - single box (via inproc) - multi box (via tcp) Evt. Processor (…) Evt. Processor Pipeline event 1 step 1 event 2 event. N Sink Evt. Processor (…) Evt. Processor Pipeline 2 event 1 step 2 event. N Sink Histogrammer G. Cerizza LECM 2018, Slide 7
FPGA-Based Systems § MSU FPGA Taskforce @ ICER (Institute for Cyber Enabled Research) • Collaboration between NSCL and CMSE dept. § Two FPGA nodes based on Altera/Intel Technology • Intel(R) Xeon(R) Silver 4114 CPU @ 2. 20 GHz • Intel Arria 10 FPGA § Benchmarking • Comparative studies on data fitting, functional fitting, machine learning based on NSCL data (i. e. trace fitting for Pixie 16) • Testing the following possibilities: shared memory parallel, shared network parallel, GPU, and FPGA § Stay tuned for results… G. Cerizza LECM 2018, Slide 8
- Slides: 8