Environment for Realtime Streaming Acquisition and Processing Reactive

Environment for Real-time Streaming, Acquisition and Processing Reactive micro-services based datastream processing framework. V. Gyurjyan, D. Lawrence, C. Timmer, N. Brey, B. Raydo, D. Abbott, C. Larrieu, M. Batagllieri, L. Cappeli, T. Chiarusi, C. Pellegrino

Outline • Triggered vs. stream readout • Framework unification effort at the JLAB • Reactive actor model based programming paradigm － Reactive vs. passive programming － Micro-services vs. monolithic architecture － Event vs message driven communication • ERSAP reactive micro-services based data-stream processing framework for SRO and data stream processing － Our experience for stream data processing: CLAS 12 and Glue. X • Framework level workflow orchestration － Data-processing performance optimization across diverse hardware and software infrastructures. 2

Triggered vs. SRO trigger type strob L 1 ack EB ROCs user L 2 TS FE Modules EH L 1 accept EB user EH L 2 accept L 3 accept clear FE Modules ER ER busy TS Trigger supervisor • > data volumes ROC Readout controller • > storage requirements EB Event Builder EH Event Hub ER Event Recorder 3

Batch vs. stream processing push pull Consumer (active) Data Source S 1 Consumer (pasive) Data Source • • S 1: Proactive, responsible for change in S 2: Passive, unaware of the dependency • • S 1: Broadcasts it’s own result S 2: Subscribes S 1 change events and changes itself Passive programming S 2 S 1 S 2 Publisher/Producer Subscriber/Consumer t 2 t 1 t 0 Feedback to control backpressure 4 Reactive programming Enables event/message driven stream processing

System architecture • Reactive event driven actor (micro-service), data-stream pipe, and orchestrator. • Stream of data quanta, flowing through directed graph of actors. • Application is a network of independent “black box” actors. S 4 Data processing station S 1 Service engine S 5 S 8 S 9 S 10 S 2 Data–stream pipe S 3 S 6 • Data moves across actors not instructions S 7 • Actors communicate with each other by exchanging the data quanta across predefined connections by message passing, where connections are specified externally to actors. • User provided data processingle-threaded algorithms (engines) are presented as fully scalable actors in the framework. 5 Workflow manager

Data processing station: actor • User engine run-time environment. • Engine follows data-in/data-out interface. • Engine gets JSON object for run-time configuration. Runtime Environment Communication User provided code Engine Data Processing Actor (micro-services) Data Processing Station 6 Multi-threading Configuration

ERSAP architecture DPE Data processing environment FE Front end DPE SC Service container Orchestration Layer DPE SC Local Registration SC Registration FE gateway security Local Registration DPE SC SC SC Service Layer Data Meta-Data Zero. MQ(x. Msg) POSIX Shared Memory (FIPC) 7 in-Memory Data-Grid

Actor communications Publish/Subscribe P 2 P Communication Transient Data 0 MQ/POSIX_SHM/Data-Grid • • Meta-description Serialization Data-Stream Pipe 8

Workflow orchestrator Application Monitoring, Real-time Benchmarking Command-Line Interface Application Deployment and Execution Hardware Optimizations Service Registration/Discovery Orchestrator Exception Logging and Reporting Data-Set Handling and Distribution Farm (batch or cloud) Interface 9

Actor deployment optimization In-Memory Data-Grid Meta-data Detector-2 POSIX Shared Memory Detector-1 1 2 3 4 In-Process SHM 5 In-Process SHM DPE-1 DPE-2 Node-1 Node-2 10 6 7 In-Process SHM DPE-3

Heterogeneous deployment algorithm C++-DPE DCHBg In-Memory Data-Grid DCHBc FTOF In-Process SHM Java-DPE Farm Node 11 EC

Streaming data pipeline 12

Performance with 0% frame loss VTP Stream Software Source Ag Based on LMAX disruptor technology Full processing 22 GByte resident memory at 13 core utilization 2500 Data Rate MB/sec EP Ag 2000 1500 1000 15000 17000 19000 21000 23000 Stream Frame Rate Hz 25000 Intel Xeon (R) Gold 5218 @ 2. 3 GHz Ag CPU 600% Resident memory 5. 4 GByte Throughput 5. 3 GB/sec 65536 ns Ag Ep aaabbbbbcc abccc aaaaabc bccccc aacccdddddd ffffddddssss ffddds fffddsssss fdddss ffffddddss aaabbbbbcc ffffddddssss abccc ffddds aaaaabc fffddsssss bccccc fdddss aacccdddddd ffffddddss 13 27000

CLARA: CLAS 12 offline data processing CLARA is a micro-services architecture for data analytics. CLAS 12 processes data using reactive independent actors model since 2011. Event Reconstruction Application MAGF FCAL FTHODO FTEB DCHB FTOFHB EBTB FTOFTB DCTB EBHB LTCC HTCC EC BAND CVT CND Off-line batch processing P 2 Pn Silo EH Physics Analysis Application 14 Data Quality Assurance monitoring P 1

CLAS 12 data processing status and performance 29 26 13, 5 12 RG-A RG-K Fall RG-A Spring 2018 2019 processed data CLAS 12 2020 23 18 9 2, 7 RG-B Fall RG-B Spring 2019 2020 3 RG-F Spring Summer 2020 Raw data 2. 5 PB Physics data 193 TB Total CPU 35 Mhr Trains 28% 193 TB 487 TB DST 72% CLAS 12 Reconstruction Application Vertical Scaling Ration between single-threaded Engine and CLARA deployment performances 129, 6 Amdahl's Law Curve Fit FRAMEWORK SPEEDUP ON THE JLAB FARM 60 Data File : clas_005038. evio. 00130. hipo Node : Intel Xeon E 5 -2697 A v 4 @ 2. 6 GHz Clara : v 4. 3. 11 Plugin 1 : coatjava-6. 3. 1 Plugin 2 : grapes-2. 1 40 80 72 50 P=0. 995 30 50 35 30 25 20 15 10 5 0 32. 2 x 10^9 events DATA PROCESSED 20 10 farm 13 farm 14 farm 16 farm 18 0 farm 19 0 10 20 30 Calculated Speedup 15 40 50 Amdahl's Law Speedup 60 70

JANA 2: Glue. X offline data processing • Modern C++ to get as close to ‘the machine’ as possible, including NUMA awareness • Thread-level parallelism only to minimize communications latency, serialization, and memory usage • Reactive/Dataflow paradigm - Network of queues and arrows, which may be either sequential or parallel Multiple threads may be assigned to each parallel arrow Threads never have to wait on a critical section, only on a queue Users don’t have to manage locks Arrow topology generalizes to support subevents, heterogeneous hardware • Factory pattern to make object creation decoupled, lazy, memorized, and recursive • Plugin architecture to isolate heavyweight dependencies JEvent. Source( sequential) Raw data input file (e. g. EVIO) JFactory (parallel) Queue of unreconstructed events {Hits, …} JEvent. Processor (sequential) Queue of reconstructed events {Tracks, …} 16 Histograms and output files (e. g. ROOT)

Glue. X data processing status and performance Glue. X in 2019 Glue. X R&D and Production (2005 -) BDX R&D (2017 -) EIC R&D (2019 -) CLAS 12 SRO R&D (2020 -) Real data 3. 1 PB MC data 0. 3 PB Total data 3. 4 PB Real data CPU 39. 6 Mhr MC data CPU 8. 0 Mhr Total CPU 47. 5 Mhr Event size 12 -13 k. B JANA microservices Software trigger On-line Monitoring Frontend • • Event builder Aggregators 17 Fast recon Full recon

International collaboration 18

Advantages of the actor model • Artifacts are small, simple and independent －Easier to understand develop －Reduced develop-deploy-debug cycle • Easy to migrate to data • Scales independently • Independent optimizations • Improves fault isolation • Easy to embrace hardware heterogeneity • Eliminates long term commitment to a single technology stack. 19

Summary • Reactive actor model based data-stream processing framework is under development. • Based on mature frameworks: CLARA, JANA 2 and CODA • Performed resource requirement studies , showing feasibility of near real time data processing. • Defined proper interfaces for various data processing actor artifact integration. • Collaborative effort between JLAB Physics and CST division, as well as Tri. DAS group. Thank you. 20

Backup slides 21

Vertical scaling or multi-treading Monoliths vs. Micro-services API Gateway / Proxy Presentation Logic Data Pros • Strong coupling, thus better performance • Full control of your application Cons • No agility for isolating, compartmentalizing and decoupling data processing functionalities, suitable to run on diverse hardware/software infrastructures • No agility for rapid development or scalability Pros • Technology independent • Fast iterations • Small teams • Fault isolation • Scalable Cons • Complexity networking (distributed system) • Requires administration and real-time orchestration 22 g: lin ca Zs mi s ce rvi -se cro Horizo ntal s caling The Art of Scalability. by Martin L. Abbott and Michael T. Fisher. ISBN-13: 978 -0134032801

Event-Driven vs Message-Driven Event Driven Message Driven S 1 S 1 event broadcasting S 2 S 1 message has a clear destination 23

Sub-event level parallelization FCAL CVT FTHODO FTEB CND EBHB DCTB EBTB RICH CTOFTB MAGF DCHB FTOFHB EC LTCC HTCC 24 FTOFTB

Data-quantum size and GPU occupancy C++-DPE DCHBg In-Memory Data-Grid set data-quantum size EC FTOF In-Process SHM Java-DPE Farm Node 25

Data-processing chain per NUMA Start DPE pinned to a NUMA socket. Java-DPE Back pressure control In-Process SHM NUMA 0 Java-DPE In-Process SHM NUMA 1 Farm Node 26