INSTRUMENTATION FOR COLLIDING BEAM PHYSICS Novosibirsk Russia February

INSTRUMENTATION FOR COLLIDING BEAM PHYSICS Novosibirsk, Russia, February 24 - March 1, 2014 n The CMD-3 Data Acquisition System upgrade. A. Ruban*, A. Kozyrev, I. Logashenko, A. Selivanov BINP, Novosibirsk. February 28, 2014

The CMD-3 Data Acquisition System upgrade. VEPP-2000 Collider Layout (before July 2013) n n n VEPP-2 M Historical Siberian Snake 2 E max = 1. 4 Ge. V L(1. 4 Ge. V) = 3 1030 sm-2 s-1 VEPP 2000 Round Beams 2 Ge. V 1031 sm-2 s-1 L(2 Ge. V) =1032 sm-2 s-1 February 28, 2014

The CMD-3 Data Acquisition System upgrade. February 28, 2014 K-500 injection channel will increase e+ production rate, e+/sec from 2*107 to 3*108 e- production rate, e-/sec from 109 to 1011 n n n Now VEPP 2000 is under upgrade. After upgrade of VEPP 2000 Luminosity will increase up to 1032 sm-2 s-1 Background condition will increase also.

The CMD-3 Data Acquisition System upgrade. CMD – 3 Subsystem Layout February 28, 2014 1 – Vacuum Pipe 2 – Drift Chamber 3 – BGO Endcap Calorimeter 4 – Z – Chamber 5 – Superconducting Solenoid CMD-3 6 – LXe Calorimeter 7 – Cs. I Calorimeter 8 – Yoke 9 – Superconducting magnet lenses Mu-System and TOF not showed

The CMD-3 Data Acquisition System upgrade. General view to CMD-3 DAQ upgrade Subsystem Sensor number Time Charge Trigger Scaler DC wires 1260 + ++ + + ZC Sector 48 + + ZC Stripe 512 + + BGO Crystal 680 BGO Grope 48 Cs. I Crystal 1152 Cs. I Grope 80 + + LXe* Stripe 2112 + + LXe Tower 264 + + Mu Counter 48 ++ ++ ++ TOF_2014 Counter 184 ++ ++ ++ FLT Word ~400 + February 28, 2014 + + + + Total up to 15 k channel, Total number of digitizers up to 300 boards. n n n Average speed should be increased from 1 k up to 5 k events per second at Event Builder input. Total number of channel is about 15 k. LXe* Stripe subsystem is under upgrade of digitizing boards now. Legacy Klukwa is totally removed. New digitizer AWF is aligned with front-end inside detector.

The CMD-3 Data Acquisition System upgrade. CMD-3 Data Acquisition Electronics Layout February 28, 2014 n n n Trigger data pipeline synchronization Event queue and time using control Data collection Status control and check Calibration The CMD-3 Time Oriented Measurement & Acquire (TOMA) DAQ utilize distributed architecture based on serial bus called C-Link (Chronopher Link). Data rate at FLT input 10. 4 Gbps, Data rate at C-Link network 7. 5 Gbps, Data rate at Eth network 0. 7 Gbps (input of Event Builder)

The CMD-3 Data Acquisition System upgrade. Modern CMD-3 DAQ typical performance Screenshot of DAQ window Run_4500 till today February 28, 2014

The CMD-3 Data Acquisition System upgrade. Modern CMD-3 DAQ typical performance 2013, Low energy runs February 28, 2014 TOMA DAQ Legacy Klukwa DAQ • 70μs is default power-up value. Digitizers are ready to work with dead time 50μs • All Trigger electronics and new AWF digitizers able to work without dead time

The CMD-3 Data Acquisition System upgrade. Modern CMD-3 DAQ typical performance Screenshots of DAQ diagnostic window ¨ ¨ n n left – Cosmic calibration Run_31760, 2013 -07 -05, 207146 events, 101 events/sec right – Data Taking Run_31957, 2013 -07 -07, 170953 events, 407 events/sec C-link is error-free at this level of statistic Errors is due to raw Ethernet networking packet losses DAQ networking losses level is significantly less then Dead Time losses so it’s remain acceptable up to 1. . 2 kevents/sec At higher luminosity event rate will increase up to 5 kevents/sec and estimated value of losses would be unacceptable. February 28, 2014

The CMD-3 Data Acquisition System upgrade. Modern CMD-3 DAQ typical performance Cosmic muon track February 28, 2014 Some problem with C-Link, Run_26969, 2013 -01 -01, 236551 events

The CMD-3 Data Acquisition System upgrade. The upgrade steps n n Use more Ethernet switches to share data stream Redesign General Interface Board of Data Delivery (GIBDD): To implement interface with more speed ¨ To implement some protocols of TCP stack ¨ To use deeper zero-suppression and data compression ¨ To introduce capability for software 2 -nd level trigger ¨ n *Since we are invited to upgrade DAQ of KEDR and DEUTON detectors, it’s time to make some modification (grinding ; -)) of GIBDD’s firmware and hardware. February 28, 2014

The CMD-3 Data Acquisition System upgrade. Component to be changed General Interface Board for Dada Delivery GIBDD from current to next generation n n New version (right) of GIBDD control board is under assembling now. All elements are prototype tested. It carries FPGA of 16 k LE, SRAM 8 MB, ¨ ¨ ¨ n February 28, 2014 2 x Ethernet 100 ports, USB High Speed port (2. 0, 480 Mbps), dedicated Up-Link port. Band Width (BW) splitting technique is used to guarantee an access of each interface to memory. ¨ BW splitting is intrinsically robust and allows an automatically detect fault condition

The CMD-3 Data Acquisition System upgrade. New component n 1 G Ethernet is, of course, the technology of choice. But: ¨ What kind of implementation is the best? n Our decision is to use PC motherboard of mini. ITX (170 x 20 mm) form factor rather then to develop custom FPGA-based design with full TCP/IP engine. n So, we will equipped each of GIBDD with it’s own computer, thus we will built “mini-ITX farm” It’s not so elegance but for comparable cost it allows fast implementation due to standard OS capability with built-in networking support and wide set of diagnostic features. Also it allows to run commonly adopted C/C++, Python programs for buffering, fitting, zero suppression and packing. Commercially available HS (480 Mb/s) USB-to. Local_Bus mezzanine IC is used to transfer data to motherboard These ICs are supported with standard drivers in virtually any OS. n n February 28, 2014 To be useful at 1 G application modern CPU should be about 10 W TDP.

The CMD-3 Data Acquisition System upgrade. Upgrade scheduler n n n Till to May 2014 complete assembling of new boards and make electronic part of DAQ ready to work with existing software part of DAQ. Targeted speed performance 1. . 2 k events/second. Start software modification to work with UDP network. Till to November 2014 built “mini-ITX farm” and ready to work with UDP network. Targeted performance 2 k events/second. Start DAQ’s software debugging to work with TCP/IP. Till to January 2015 ready to work with “mini-ITX farm”. Targeted performance 5 k events/second average and up to 10 k events/second peak should be reached. If collider background conditions will increase dramatically, one of UDP network can be used to built Software Second Level Trigger. February 28, 2014

The CMD-3 Data Acquisition System upgrade. End n Thank You for Your attention February 28, 2014

The CMD-3 Data Acquisition System upgrade. All synchronization and data transferring in TOMA DAQ is provided by single specially developed tool. While well-known S-link is designed to hide all synchronization detail, our tool is intended to obtain all chronopher functions, thus it is called C-Link. February 28, 2014 Note, please, this is NOT a mezzanine at physical layer, this is part of FPGA design. C-Link modules and it’s interface is designed in flat model approach. Reference Clock speed Data speed Electrical Levels Media Connector Type F bunch_Crossing*2 = 25 МHz 25 Мbps (50 or 100 optional) LVDS, 4 m. A same as IEEE-1394 Double UTP Cat 5 100 Ohm, double side termination USB, type А, both ends

The CMD-3 Data Acquisition System upgrade. Table of synchronization modes used in CMD-3 TOMA DAQ. Layer of Mode of C-Link synchronization specification Target process February 28, 2014 Time resolution, s Signaling Synchronous Time stamp for precision time measure and DSP fast A/D conversion 10 -11 Signaling Mesochronous First Level Trigger data sampling and frame aligning 10 -9 Transaction Plesiochronous Command/Data Transmitting Event Asynchronous Collaboration with common universal tools, collaboration with legacy DAQ 10 -8 Event Isochronous Changing of DAQ status, Data buffering 10 -7 10 -8 Typically modern HEP DAQ hardware has very complicated hierarchical design. The main feature of TOMA DAQ is exchange of hardware hierarchical complexity to synchronization modes hierarchical complexity.

The CMD-3 Data Acquisition System upgrade. CMD-3 Link Stack n n n February 28, 2014 Bidirectional Data Line, No preamble due to dedicated Clock Line Coincidence of Start. Bit Level and Clock Line Transition is Common Stop Messages are transmitted to “Data Delivery” Modules through Links Answer from Digitizer contains Event Number, Board ID, Status Word Command Code is covered with parity, Data is covered with CRC

The CMD-3 Data Acquisition and Control System. Event’s Queue and Time Control n n February 28, 2014 “Event Manager” accepts Requests, and Builds Event’s Queue “Event Controller” serves Queue one-by-one and convert it to Stream of DAQ Messages Trigger’s Event are serviced immediately or skipped other are settled in queue Messages are transmitted to “Data Delivery” Modules through Links

The CMD-3 Data Acquisition and Control System. CMD-3 “Standard Design” for board’s compatibility. n n n February 28, 2014 To obtain compatibility of different digitizers in DAQ single style is required. Specific “Standard Design” was developed. It includes all function required for digitizer to work in DAQ. It supports full independent access from any interface. Supports modular approach for Board specific Controllers Code is written in Altera HDL

n g