Development of PCI Bus Based DAQ Platform for

Development of PCI Bus Based DAQ Platform for Higher Luminosity Experiments T. Higuchi, 1 H. Fujii, 1 M. Ikeno, 1 Y. Igarashi, 1 E. Inoue, 1 R. Itoh, 1 H. Kodama, 1 T. Murakami, 1 Y. Nagasaka, 2 M. Nakao, 1 K. Nakayoshi, 1 M. Saitoh, 1 S. Shimazaki, 1 S. Y. Suzuki, 1 M. Tanaka, 1 K. Tauchi, 1 G. Varner, 3 M. Yamauchi, 1 Y. Yasu, 1 T. Katayama, 4 K. Watanabe, 4 M. Ishizuka, 5 S. Onozawa, 5 and C. J. Li, 5 1 High Energy Accelerator Research Organization (KEK) 2 Hiroshima Institute of Technology 3 University of Hawaii 4 Densan Co. Ltd. 5 Designtech Co. Ltd.

Future High Energy Experiments • Next B-factory (Super KEB) – – Higher luminosity L > 1035 cm-2 s-1 trigger rate > 10 k. Hz. Dedicated Super Belle detector data size ~ 300 k. B / event. KEK-B bird view • Belle detector J-PARC (Japan Proton Accelerator Research Complex) – – Trigger rate > 10 k. Hz. Requirements vary widely depending on experiments. Linac construction 50 Ge. V synchrotron building

Call for a New DAQ Platform The Belle DAQ was implemented without pipeline. It operates with ~10% deadtime at 500 Hz trigger rate. In future experiments, the trigger rate will be > 10 k. Hz. A pipelined DAQ for deadtime-less system is necessary. Further more, the complicated detector system may output > 10 times larger data as current one. High density platform is favored. To catch up with the future setup, we have developed a new DAQ platform along with the following boundary conditions….

Boundary Conditions in the Development • Keep system flexible – • Reduce bandwidth usage – • Put data reduction CPU on the data stream. New Put event buffer on the stream to reduce deadtime. w Make module compact Ne – • New Increase trigger efficiency – • Separate detector dependent parts (ADC, TDC, etc) from the system. Increase module density to be fit in the room. Keep cost lower – Import computer market trends/techniques as possible.

Conceptual Design (Front-end) • Mezzanine-ADC or Mezzanine-TDC Module Detector ADC/TDC L 1 trigger pipeline sampling clock Readout FIFO 0 xaa 55 0 x 0246 0 xf 3 b 7 … trigger signal User-defined ADC/TDC with add-on structure Less dependence of the platform on the detectors. Mezzanine structure Higher platform density.

TDC Design Example Under design • TDC module layout 71 186 mm 2 FPGA to COPPER AMT -2 to COPPER data/24 ch connector busy digitized output clock, trigger AMT-2 = TDC with L 1 buffer designed by ATLAS – Input = 24 ch LVDS, Output = 24 ch. – Resolution = 0. 78 ns/bit (@ 40 MHz clock). – Trigger buffer depth = 8 words. – Enough space to put required functions.

Conceptual Design (Back-end) • PCI bus based platform Readout FIFO 0 xaa 55 0 x 0246 0 xf 3 b 7 … Local bus DMA PCI 9054 deadtime reduction Data compression EB CPU PCI bus Network Trigger PCI provides up to 133 MB/s data transfer. PCI is one of PC market standards. We need less worry about device development. Newest technology can be always purchased. PCI has mezzanine standard (PMC). Platform density can be increased.

detector signals Schematic of the New DAQ Platform mezzanine (add-on) module local bus PCI bus PMC ADC/TDC FIFO Memory ADC/TDC FIFO Network Bridge ADC/TDC FIFO trigger, clock CPU Bus CTRL trigger interrupt

The COPPER ADC/TDC VME-9 U size board PMC VME ADC/TDC PMC ADC/TDC VME CPU/PMC ADC/TDC Rear Board (SPIGOT) COmmon Pipelined Platform for Electronics Readout

COPPER Components • 4 ADC/TDC (FINESSE) slots – ~100 ch (24 ch 4) / COPPER board • 2 MB readout FIFOs – 512 k. B for each FINESSE slot: IDT 72 V 295 (IDT) 2 4 slots. • Data transfer buses – – – A 32/D 32/33 MHz local bus. FPGA for Local bus and FIFO control. Local-PCI bus bridge: PCI 9054 (PLX), which has DMA facility. A 32/D 32/33 MHz PCI bus. PCI-PCI bus bridge: 21152 (Intel) 2. • 5 PMC slots (2 are on the SPIGOT) – 1 for CPU, 1 for link device to the event builder, and 3 for generic use (intended for additional memory. ) • VME interface – PCI and VME are connected via dual-port memory for reset/debug use.

The FINESSE • User defined ADC/TDC module 186 mm Local bus I/F (for control) Up to 16 bit. output ~24 input ch. 71 mm Clock, L 1 trigger Trigger busy Front-end INstrumentation Entity for Sub-detector Specific Electronics

PCI Mezzanine Card Standard - PMC PCI Ethernet card PMC Ethernet card – PMC is 100% compliant with the PCI. – Many applications are available: Ethernet card, Gigabit Ethernet card, memory module, CPU, etc.

PMC Processor • Radi. Sys EPC-6315 Equipped with Intel Pentium. III 800 MHz. – 512 k. B secondary cache. – 256 MB SDRAM with ECC. – Chipset: Radi. Sys 82600. – Bootable from Compact. Flash. Red. Hat Linux 7. 3 is running. – 33 -bit 33/66 MHz PCI bus interface. – RJ-45 Ethernet port (slow control) Compact. Flash socket (boot)

Test Setup Thermostatic oven (humidity = 30%) COPPER Vacant NIC (Tx) FINESSE Serial Prototype FINESSEs SPIGOT Vacant CPU Vacant Ethernet Trigger 10 Hz trigger Ethernet hub FINESSE Ethernet PC Linux PC (Rx)

Stability Test • Check list at Rx PC. – – – Thermal condition was varied during the test. temp. • Is event-header/event-footer marker correct? Are data contents same as pre-defined one? FINESSE local counter == Event tag from trigger system? Are 4 local counters from FINESSE same? Does FINESSE local counters increase by 1 correctly? 40ºC, 24 h 20ºC 1 h • 10ºC, 24 h 50ºC, 24 h 1 h 1 h 77 hours Total data transfer = 3. 4 GB. 1 h 15ºC, 1 h + room-temp. 48 hours time No error!

Performance Study • Data transfer speed – The data transfer speed from the readout FIFOs to the CPU main memory using DMA is measured ~125 MB/s (w/o DMA overhead. ) 94 % of full performance. Performance study is proceeding…

Status and Prospects • COPPER – • The COPPER is under final tuning and the design will be fixed by this May. FINESSE Design of TDC FINESSE will be finalized soon and start production. – Design of ADC FINESSE is being designed. – • Integrated system test – We replace a part of current Belle DAQ system with COPPERs in the next summer for the integrated system test.

Summary • We are constructing a pipelined DAQ system for future HEP experiments by developing a PCI-bus based DAQ platform that has mezzanine structure for density increase. • The platform equips processor module for data reduction. • The platform showed good stability after 125 h test. • • • The data-transfer performance of the platform was measured to be close to full performance (94%) in DMA mode (without DMA overhead. ) The mezzanine-ADC/TDC modules are being designed. We start integrated system test from next summer using current Belle DAQ system.