EUDAQ 2 0 Ulf Behrens Alan Campbell Francesco

EUDAQ 2. 0 Ulf Behrens, Alan Campbell, Francesco Crescioli, David Cussans, Hendrik Jansen, Moritz Kiehn, Hanno Perrey, Richard Peschke, Igor Rubinskiy, Simon Spannagel

EUDAQ > Distributed DAQ system Software Layout Run. Control TLU Producer Data flow TCP/IP Data Collector Producer 1 § Consists of completely independent parts / programs such as: Run Control Data Collector Producers Producer 2 > Synchronous read out Read Out Device 1 § All devices get a common trigger Device 2 Trig ger / B usy Ha Tr ig § Every device raises a busy signal ge r/B nds h ake > DAQ for MIMOSA telescope us y Trigger Hardware Layout TLU Richard Peschke | EUDAQ 2. 0 | | Page 2

Goals > More flexibility in the data format § Packets > Increasing of the track rate by more than 2 orders of magnitude > Easier combination of different kind of devices § FE-I 4 / Mimosa / Timepix(3) / Slow Control § Asynchronous Data Streams > Resolved scalability issues § Decentralized Data Taking § Data can be stored locally without network overhead > Stay backward compatible to old EUDAQ 1. x Producer and analyze readout chain (EUTelescope) § Only recompiling is needed § Changes are only need to benefit from some new features > Cross platform Richard Peschke | EUDAQ 2. 0 | | Page 3

Motivation MIMOSA Tracks MIMOSA DUT 1 2 3 Information Needed: • Space and time information at the position of the DUT • Space resolution < 3 µs • Separation time* < 30 ns • Time Resolution < 4 ns *Tracks can only be separated if their time difference is larger than the separation time Richard Peschke | EUDAQ 2. 0 | | Page 4

Motivation MIMOSA Tracks MIMOSA DUT 1 2 3 + Intrinsic resolution of ~3. 5 µm precession - Very poor time resolution (115. 2 µs) Richard Peschke | EUDAQ 2. 0 | | Page 5

Motivation MIMOSA Tracks MIMOSA DUT FEI 4 1 2 3 + Intrinsic resolution of ~3. 5 µm precession - Very poor time resolution (115. 2 µs) Solution adding faster device for time stamping the tracks + time resolution 25 ns - Intrinsic resolution 14 x 70 µm Asynchronous data streams Richard Peschke | EUDAQ 2. 0 | | Page 6

Combining of two types of sensors Spacial resolution [µm] 100 Mimosa Telescope FEI 4 10 1 0. 01 Combined Resolution 0. 10 1. 00 10. 00 Integration Time [µs] 100. 00 > By adding the FE-I 4 Detector the time resolution is increased by more than 3 orders of magnitude > Gives the possibility to timestamp the individual Tracks Richard Peschke | EUDAQ 2. 0 | | Page 7

Definition: Read Out Frame (ROF) > Read Out Frame is the finest granularity one gets from a device > It does not mean the granularity with which one reads out the Device. § As an Example: With every read out from the TLU one gets the information from multiple triggers. Since one can disentangle the information from the individual trigger the Read Out Frame is the individual Trigger! § For the FE-I 4 one ROF is one LHC bunch crossing which covers a time of 25 ns § For the Mimosa one ROF is one frame which covers a time of 115 µs Richard Peschke | EUDAQ 2. 0 | | Page 8

Time From Trigger to Timestamps Particles TLU DAQ 1 Readout Frame Contains: Trigger No. Hits … Readout Frame Contains: … DAQ 2 > One trigger per read out frame Trigger No. Hits > Prevents the issuing of triggers for the whole time of the read out > All but the first particle are ignored > Online event building > Slowest device limits the Event rate Marked for write to Disk Not Marked Richard Peschke | EUDAQ 2. 0 | | Page 9

Time From Trigger to Timestamps Particles TLU DAQ 1 DAQ 2 Readout Frame Timestamp 1 Hits Contains: Timestamp 1 Timestamp 2 Hits … Timestamp 2 Hits Readout Frame > Using all particles § Possible rate increase of two orders of magnitude > No online event building > Offline merging Contains: … Marked for write to Disk Not Marked Richard Peschke | EUDAQ 2. 0 | | Page 10

The Building Blocks > Processors § Handle all the data processing § Processors can be chained together > Events § Handles the data § Data are always encapsulated by an Event § Events have IDs to identify individual types § Device specific code is encapsulated in the Data Converter Plugins Richard Peschke | EUDAQ 2. 0 | | Page 11

Raw. Data. Event: the Default Event Type Event Header Event Number Type Run Number Flags: {FLAG_BORE, FLAG_EORE, FLAG_EUDAQ 2, FLAG_PACKET} Tags: Tag Value Tag Name Timestamp(s) Timestamp > Derives header Information from event base class § Type / Subtype pair are used for identifying the event § Tags can be used to store additional information § Vector of Timestamps Block ID Block Binary Data ID ID Binary Data > Adds Vector of “Blocks” § Contain identifier and binary data Richard Peschke | EUDAQ 2. 0 | | Page 12

Data Converter Plugin class Data. Converter. Plugin { virtual void Initialize(const Event &, const Configuration &); virtual unsigned Get. Trigger. ID(const Event &) const; virtual time. Stamp_t Get. Time. Stamp(const Event& ev, size_t index) const; virtual size_t Get. Time. Stamp_size (const Event & ev) const; virtual int Is. Sync. With. TLU(const Event & ev, const Event & tlu. Event) const; virtual bool Get. Standard. Sub. Event (Standard. Event & , const Event & ) const; virtual t_eventid const & Get. Event. Type() const; virtual std: : shared_ptr<Event> Extract. Event. N( shared_ptr<Event> ev, size_t Number. Of. ROF); virtual size_t Get. Number. Of. ROF(const Event& pac); > Handles Device Specific Tasks § Converting Raw Data to hit positions § Extracting Events (ROF) from Packets § Checking for synchronization § (Converting to LCIO) virtual bool is. TLU(const Event&); virtual unsigned get. Unique. Identifier (const Event & ev); … }; Richard Peschke | EUDAQ 2. 0 | | Page 13

Why not Using Inheritance? > Having different CPP classes for different users would end up in a mess > It does not scale well with many different version on different computers § Online Monitor Problem! > Handling the data must not require to understand the content Richard Peschke | EUDAQ 2. 0 | | Page 14

How does it Work? Library Code: int Sync 2 TLU: : Compare. Events(const Event & Current_event, const Event & tlu) { return Plugin. Manager: : Is. Sync. With. TLU(Current_event, tlu); } > The library developer writes the code independent of the user Magic: int Plugin. Manager: : Is. Sync. With. TLU(const Event & ev, const Event & tlu) { return Get. Instance(). Get. Plugin(ev). Is. Sync. With. TLU(ev, tlu); } User’s Code: int Data. Converter. Plugin : : Is. Sync. With. TLU(const Event & ev, const Event & tlu. Event) const { int trigger. ID = ev. Get. Event. Number(); int tlu_trigger. ID = tlu. Event. Get. Event. Number(); return compare. TLU 2 DUT(tlu_trigger. ID, trigger. ID); } Richard Peschke | EUDAQ 2. 0 | | Page 15

Packets Why do we need packets? How should it be: > If the data of the individual ROF is smaller than the overhead of an Event > It should fit in the current scheme of EUDAQ > If the decoding of the individual ROFs takes a long time § It can be postponed to the Offline reconstruction § No additional class needed § No extra overhead for existing classes > Easy to add > Invisible for users who don’t need it Richard Peschke | EUDAQ 2. 0 | | Page 16

Packets Event Packet Header Meta Data ROF ROF ROF Data Converter plugin > Uses the well known mechanism of data converter plugins > Complete flexibility how the users store their data bool Process. Packet(event_sp Ev) { if (!Ev) { return false; } if (Ev->Is. Packet()) { for (size_t i = 0; i < Plugin. Manager: : Get. Number. Of. Events (*Ev); ++i) { if (!Process. Packet(Plugin. Manager: : Extract. Event. N(Ev, i))){ return false; } } return Process. Event(Ev); } Richard Peschke | EUDAQ 2. 0 | | Page 17

Packets Data. Converter for Detector Events using event_sp = std: : shared_ptr<eudaq: : Event>; class Detector. Event. Converter. Plugin : public Data. Converter. Plugin { public: virtual bool Get. Standard. Sub. Event (Standard. Event &, const Event &) const { EUDAQ_THROW("a detector Event can't be converted to standart event" ); } virtual size_t Get. Number. Of. ROF(const eudaq: : Event& pac) { auto det = dynamic_cast<const Detector. Event*>(&pac); return det->Num. Events(); } virtual event_sp Extract. Event. N(event_sp pac, size_t Number. Of. ROF) { auto det = std: : dynamic_pointer_cast <Detector. Event>(pac); if (Number. Of. ROF < det->Num. Events()) { return det->Get. Event. Ptr(Number. Of. ROF); } return nullptr; > A Packet is an Event that contains other Events > The Detector. Events are Packet! > Every plugin handles only its own level of abstraction § No Recursion } private: … } Richard Peschke | EUDAQ 2. 0 | | Page 18

Software Layout with Multiple Data Collector > Run Control: • Central authority • Starts, stops and configures runs • Assigns Data Collectors to Producers • Main user interface > Producer: • Interface with user’s readout system • Receives command from Run Control • Sends its data to the Data Collector > Data Collector: • Receives data from multiple producers • Producer can have their own Data Collector on local machine > Log Collector: • Stores and displays log information • Start time, stop time • Errors and warnings Richard Peschke | EUDAQ 2. 0 | | Page 19

Merging > Every Data Stream is stored separately > The TLU is the central authority for merging > Every event gets compared to the TLU events to find the corresponding TLU event > The Compare algorithm can easily be modified by the users. It is part of the converter plugin > Adding the TLU timestamp to the Event > Processing events from different producers Individually until the tracks are extracted > Merging happens on the level of tracks and not events Richard Peschke | EUDAQ 2. 0 | | Page 20

Information Streams What you Want to say Control Flow Read File Control Flow Select Events Data Flow Save to File Data Flow How it looks in code void Do. Everything() { std: : ifstream file_("test. txt"); std: : ofstream out_file("out. txt"); std: : string buffer; while (std: : getline(file_, buffer)) { if (buffer == "some. Keyword") { out_file << buffer; } else if(buffer == "Some. Other. Key. Word" ) { break; } } } The Native environment for every Task is a loop Control Flow • OK • Skip this Event • Stop • (retry) Error Flow Richard Peschke | EUDAQ 2. 0 | | Page 21

Task & Processors class Processor. Base { public: Processor. Base(); virtual ~Processor. Base() {}; virtual void init() = 0; virtual Return. Param Process. Event(event_sp ev, Connection. Name_ref con) = 0; virtual void end() = 0; virtual void wait() {}. . . }; > Every task is Encapsulated in a Processor > Processors are classes with unified interface > Every Task has 3 transistions § Init § Run § End > Processors can be piped together Richard Peschke | EUDAQ 2. 0 | | Page 22

Processors in EUDAQ Data Source File reader File Handler Data Processing Data Sink Event A MH MH File reader File Handler Event B MH MH File reader File Handler Synchronizer Event D Event C Event B Event A MH MH MH File writer File Handler Event C MH MH Richard Peschke | EUDAQ 2. 0 | | Page 23

How to translate this into source code? User Code Processor_batch; batch>> file. Reader(“run 000213 a. raw“ ) >> file. Reader(“run 000213 b. raw“) >> merger(“detector. Events“) >> Show. Event. NR(10) >> event. Selector(parsenumbers(“ 1 -100“)) >> file. Writer(“text“); batch. init(); batch. run(); batch. end(); Library Code: class select_events : public Processor_Inspector { public: select_events(const vector<unsigned>& events. Of. Intresst , bool do. Bore, bool do. EORE); virtual Return. Param inspect. Event(const Event&ev, Connection. Name_ref con) override; bool m_do_bore = false, m_do_eore = false; vector<unsigned> m_events. Of. Interest ; }; Return. Param select_events: : inspect. Event(const Event& ev, Connection. Name_ref con) { if (ev. Is. BORE() && m_do_bore) return Processor. Base: : sucess; if (ev. Is. EORE() && m_do_eore) return Processor. Base: : sucess; if (m_events. Of. Interest. empty()) return Processor. Base: : sucess; if (ev. Get. Event. Number() > m_events. Of. Interest. back()) return Processor. Base: : stop; if (Event. Of. Intresst(ev. Get. Event. Number())) return Processor. Base: : sucess; return Processor. Base: : skip_event; Richard Peschke | EUDAQ 2. 0 | | Page 24 }

What's about lambdas? Processor_batch; Batch >> file. Reader(“run 000213 a. raw“) >> ; It just works! Batch >> file. Reader(“run 000213 a. raw“ ) >> ADD_LAMBDA_PROZESSOR 1 (ev) { std: : cout << "Event Nr: "<< ev. Get. Event. Number() << std: : endl; }; Why the Macro? ? Because “Expression SFINAE” And “Generic lambdas” Are not yet cross platform Batch >> file. Reader(“run 000213 a. raw“ ) >> [](const eudaq: : Event& ev, Processor. Base: : Connection. Name_ref con) { std: : cout << "Event Nr: "<< ev. Get. Event. Number() << std: : endl; } Richard Peschke | EUDAQ 2. 0 | | Page 25

What is that Good for? > Lets write a network streamer > What should it do? § Read a file § Send the events to another network location § Store data to disc § Handle errors correctly § Command line interface > And it must fit on one slide! On this Slide file. Sender. cxx: int main(int, char ** argv) { Option. Parser op("EUDAQ File Sender" , "1. 0", "", 1); Option<std: : string> dest(op, "d", "destination", "tcp: //127. 0. 0. 1: 44000" , "string", "network Destination" ); try { op. Parse(argv); Processor_batch; batch >> file. Reader(file. Config(op. Get. Arg(0))) >> Show. Event. NR(1000) >> data. Sender(dest. Value()); batch. init(); batch. run(); batch. end(); } catch (. . . ) { return op. Handle. Main. Exception (); } return 0; } file. Reciever. cxx: int main(int, char ** argv) { Option. Parser op("EUDAQ Network Reciever" , "1. 0", "", 0); Option<std: : string> port_(op, "p", "port", "tcp: //44000", "string", "network port"); try { op. Parse(argv); Processor_batch; batch >> data. Reciver(port_. Value()) >> Show. Event. NR(1000) >> wait. For. EORE(); batch. init(); batch. run(); batch. end(); } catch (. . . ) { return op. Handle. Main. Exception (); } return 0; } Richard Peschke | EUDAQ 2. 0 | | Page 26

Done Richard Peschke | EUDAQ 2. 0 | | Page 27

BUT > Is there not a way to do it without inheritance? > Can we have it more generic? Richard Peschke | EUDAQ 2. 0 | | Page 28

Tasks (The Templated Pipeline) class file. Opener { public: file. Opener(const std: : string& name) : inf(&name) {} const std: : string * inf = nullptr; . . . }; template<typename next_t, typename. . . Args> return. Types run. Task(file. Opener& reader_, next_t&& next_ , Args&&. . . args) { std: : ifstream buffer(*reader_. inf); return run. Task(set. Buffer(buffer, next_), args. . . ); } class read. File { public: read. File(std: : ifstream & value) : m_value(&value) {} read. File() : m_value(nullptr) {} ifstream * m_value; }; read. File& set. Buffer(std: : ifstream& in. Buffer, read. File & reader_) { reader_. m_value = &in. Buffer; return reader_; } template<typename next_t, typename. . . Args> return. Types run. Task(read. File& reader_, next_t&& next__, Args&&. . . args) { string buffer; while (std: : getline(*reader_. m_value, buffer)) { run. Task(set. Buffer(buffer, next__), args. . . ); } return_ok; } > Zero Runtime Overhead > Extremely Flexible > Perfect Encapsulation of Tasks BUT: > Non Trivial to read > Exposes some internals to the outside int main(int, char ** argv) { run. Task(file. Opener("d: /DEVICE_1_ASIC_on_Position_5_400 V. txt" ), read. File(), display()); } Richard Peschke | EUDAQ 2. 0 | | Page 29

Summary Accomplished Open Task > Packets: New Data Format for Multiple Readout frames > Track merging in EUTelescope > Multiple Data Collectors: Possibility to Store Data locally to reduce the network overhead > Users need to update their producer converter to take full advantage from EUDAQ 2 > Multiple Data Streams > Merging > Tasks & Processors Richard Peschke | EUDAQ 2. 0 | | Page 30

Richard Peschke | EUDAQ 2. 0 | | Page 31

TLU 1 Trigger In PMT TLU 2 Trigger Out PMT Trigger In PMT Trigger Out Extra: Two TLU Setup Fast Device (FE-I 4) Slow Device (Telescope) > TLU 1 is triggering TLU 2 Richard Peschke | EUDAQ 2. 0 | | Page 32

Time Combination to events Trigger TLU 1 TLU 2 Telescope Particle Trigger TLU 2 Trigger Discarded Trigger > TLU 1 sees all particles > TLU 2 only triggers when Telescope is not busy > TLU 1 has a trigger from TLU 2 for the offline reconstruction > Trigger which where issued after a certain time after the primary trigger will be Richard Peschke | EUDAQ 2. 0 | | Page 33 discarded completely

Test in the Laboratory > Input trigger from pulse generator Output Rate [Hz] 1 E+05 > The main TLU is not limited anymore by the Telescope 1 E+04 > The second TLU is synchronous with the Telescope > Resynchronization of the two data streams via timestamps 1 E+03 § Default approach for EUDAQ 2. 0 1 E+02 1 E+03 1 E+04 Input Rate [Hz] Main TLU rate 1 E+05 > Not tested with beam yet Telescope TLU rate Richard Peschke | EUDAQ 2. 0 | | Page 34

Backup Richard Peschke | EUDAQ 2. 0 | | Page 35

Mimosa Readout > Rolling shutter readout Rows 0 200 400 600 0 > Readout row by row § The readout takes 115 µs 115 Readout 1 Time [µs] Particle Readout 230 345 Readout 3 > Hits that appear at the exact same time can be in two different readout frames It is needed two associate two readout frames two one trigger Richard Peschke | EUDAQ 2. 0 | | Page 36

Trigger Telescope > The information for one Trigger is split up on two ROF Read out Row Time Telescope Type devices > Limitation in the analyze framework prevents the use of references to previous events > Instead of associating only the trigger that happened during the ROF we also Associate the one from the previous ROF to this Event > Trigger get associated to multiple ROF Richard Peschke | EUDAQ 2. 0 | | Page 37