Developments and applications of DAQ framework DABC v

Developments and applications of DAQ framework DABC v 2 http: //dabc. gsi. de Jörn Adamczewski-Musch, Nikolaus Kurz, Sergey Linev GSI / Experiment Electronic CHEP 14. 04. 2015

DABC core functionality Multi-thread, multi-node applications p Zero-copy transport approach p n n p Flexible plugin interface n p MBS, HADAQ, ROOT, Go 4, EPICS, DIM, FESA http for slow control n p full support for Infini. Band VERBS advanced sockets treatment with select() Java. Script ROOT as web GUI In development since 2007 n 14. 4. 2015 v 1. 0 in 2009, now v 2. 7 Sergey Linev, DABC v 2, dabc. gsi. de 2

Local communication Receiver Sender queue File, socket, PCIe, . . . Consumer buffer p Pre-allocated buffer memory pool 14. 4. 2015 memory pool p Reference counter for each buffer p Buffers queue between modules p No memcpy between sender and receiver Sergey Linev, DABC v 2, dabc. gsi. de 3

Remote communication Transport Sender queue Producer buffer buffer Receiver Transport queue memory pool node 1 p Same approach as in local p No memcpy for Infini. Band p Different communication patterns: n 14. 4. 2015 Consumer buffer buffer memory pool node 2 1 1, 1 N, N 1, N x M Sergey Linev, DABC v 2, dabc. gsi. de 4

Results from May 2011 Infini. Band tests with DABC 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 ibswitch 01 ibswitch 02 ibswitch 03 ibswitch 04 ibswitch 05 ibswitch 06 ibspine 01 ibspine 02 ibspine 03 ibspine 04 ibswitch 07 ibswitch 08 ibswitch 09 ibswitch 10 ibswitch 11 ibswitch 12 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 p p p ½ fat tree topology ~800 nodes with QDR IB 12 spine and 34 leaf switches 14. 4. 2015 p p all-to-all traffic pattern 1. 25 TB/s or 80% of available bandwidth Sergey Linev, DABC v 2, dabc. gsi. de 5

PEXOR/KINPEX plug-ins PEXOR (KINPEX): Pci-EXpress Optical Receiver • Lattice (Kintex) FPGA • 4 lane PCIe • 4 high speed SFP connectors various frontends Detector hardware FEBEX (ADC), POLAND (QFW), TAMEX (TDC) GOSIP protocol, optical chains pexor. ko Linux kernel module explodertest: : Device libpexor. so DABC Init. Daq() User. Readout() user library gosipcmd control tool pexorplugin: : Device poland: : Device Init. Daq() User. Readout() Request. Token() Start. Acquisition(). . . Init. Daq() User. Readout() subclasses 14. 4. 2015 Sergey Linev, DABC v 2, dabc. gsi. de 6

http: : Server in DABC p p p Access to different data in DAQ application Java. Script ROOT as user interface One could integrate and access data from different sources: n p p DABC, ROOT, Go 4, MBS, EPICS, DIM, FESA, . . . Same data could be accessed by Go 4 and displayed using native ROOT graphics One also could use command-line tools to access DAQ/analysis data via http or socket 14. 4. 2015 Sergey Linev, DABC v 2, dabc. gsi. de 8

Heterogeneous distributed applications DABC agents DABC DAQ HADAQ MBS DAQ Publisher EPICS shell> mbs -dabc mbs> open file test. lmd mbs> sta acq log cmd TGo 4 Sniffer linev@lxg 0538: ~ $ go 4 analysis -stream localhost GO 4 -*> Reading library: lib. Go 4 User. Analysis GO 4 -*> Close of TUser. Source GO 4 -*> Create default analysis with processor class TFirst. Step. Processor GO 4 -*> Use class TStream. Event as output event GO 4 -*> Welcome to Go 4 Analysis Framework Release v 4. 6. 3 (build 40603) ! GO 4 -*> Analysis: Added analysis step Analysis GO 4 -*> Main: starting analysis in batch mode. . . GO 4 -!> Fail to open Auto. Save file Go 4 Auto. Save. root GO 4 -*> Analysis Load. Objects: Failed to load from file Go 4 Auto. Save. root f_evt_get_open for STREAM: port=6002 timeout=1 GO 4 -*> Factory: Create input event for MBS GO 4 -*> Event Mbs. Event 101 has source localhost class: TGo 4 Mbs. Stream GO 4 -*> Factory: Create event processor Processor GO 4 -*> Create TFirst. Step. Processor GO 4 -*> Executing ROOT script first. C GO 4 -*> Factory: Create output event Output. Event GO 4 -*> Event Output. Event has source Processor class: TFirst. Step. Processor GO 4 -*> Analysis -- Initializing Event. Classes done. Publisher socket mbs> @startup Go 4 analysis socket status mbs plugin socket dabc: : Publisher • collects data from remote • build global hierarchies • version control socket 14. 4. 2015 DABC master http Sergey Linev, DABC v 2, dabc. gsi. de 9

MBS/GOSIP web gui 14. 4. 2015 Sergey Linev, DABC v 2, dabc. gsi. de 10

DABC/PEXOR/GOSIP web gui 14. 4. 2015 Sergey Linev, DABC v 2, dabc. gsi. de 11

POLAND frontend web gui 14. 4. 2015 Sergey Linev, DABC v 2, dabc. gsi. de 12

HADES detector http: //blogs. helmholtz. de http: //www-hades. gsi. de HADES is a versatile detector for a precise spectroscopy of e+e - pairs (di-electrons) and charged hadrons produced in proton, pion and heavy ion induced reactions in a 1÷ 3. 5 Ge. V kinetic beam energy region 14. 4. 2015 Sergey Linev, DABC v 2, dabc. gsi. de 13

HADES DAQ before p TRB Event builder hld /shm TRB IOC p TRB TRB TRB 14. 4. 2015 p p TRB p Event builder hld p ~30 TRB frontend boards ~10 KHz, max 300 MB/s simple round-robin approach C-based, single thread code EPICS IOCs for DAQ slow control file-based quality monitoring /shm IOC small hld UDP Event builder hld /shm HYDRA 2 server IOC Event builder hld client /shm IOC EPICS GUI Sergey Linev, DABC v 2, dabc. gsi. de 14

HADES DAQ now DABC TRB output Combiner TRB /shm TRB output TRB TRB TRB 14. 4. 2015 Combiner /shm UDP stream output Combiner stream HYDRA 2 server output /shm hld http DABC Combiner hld on all event builders p All infrastructure remains as is p Socket data stream to online analysis p Used in production beams 2014 http DABC /shm hld http DABC TRB stream p DABC stream hld client http EPICS GUI Sergey Linev, DABC v 2, dabc. gsi. de 15

HADES DAQ upgrade p New RICH and ECAL electronic n p Use in many places FPGA TDCs n p need for special calibration procedure Potentially free-running readout for RICH n p more than 100 TRBs (30 up to now) need to produce subevents afterwards HLD data format should be preserved 14. 4. 2015 Sergey Linev, DABC v 2, dabc. gsi. de 16

Change networking topology TRB DABC TRB TRB TRB DABC TRB UDP DABC TRB TRB DABC TRB DABC TCP UDP DABC TRB • All data from single TRB always send to same DAQ node • Every DAQ node can perform TDC calibration and event selection immediately • Event building done afterwards with TCP/IP or IB 14. 4. 2015 10 Gb Ethernet TRB Sergey Linev, DABC v 2, dabc. gsi. de 17

Next plans p FPGA TDCs calibrations in DABC n p Event building network (BNet) for HADES n p already done, including web control GUI planned as v 3 release, end of 2015 Accommodate C++11 features n n 14. 4. 2015 native threads, references probably as v 4 in ~2017 Sergey Linev, DABC v 2, dabc. gsi. de 18

Conclusion Flexible framework for different DAQ tasks p Web and JSROOT as user interface p Used in different test beams p Production DAQ for HADES experiment p 14. 4. 2015 Sergey Linev, DABC v 2, dabc. gsi. de 19