BESIII DAQ System BESIII Review Meeting IHEP Beijing

BESIII DAQ System BESIII Review Meeting IHEP · Beijing · China Sep. 16 - 17, 2002 Sep. 17, 2002 BESIII Review Meeting

BESIII DAQ Scheme Sep. 17, 2002 BESIII Review Meeting

Readout Crate System Controller – MVME 2431 -1 Ø Ø Ø Ø Sep. 17, 2002 Power. PC 750™ 32 -bit microprocessor 32 MB of on-board ECC SDRAM Up to 1 MB capacity for on-board firmware 8 MB on-board Flash memory One asynchronous serial debug port 10/100 Mbps Ethernet interface 4 -level requester, 7 -level interrupter, and 7 -level interrupt handler for VMEbus BESIII Review Meeting

3 Levels of Event Builder Sep. 17, 2002 BESIII Review Meeting

BESIII DAQ Tasks Ø Event Data Readout from FEE Ø Event Building Ø Event Filtering Ø Event Recording to Persistent Media Ø Run Control System Ø Monitoring (event, histogram display. . . ) Ø Message Reporting Sep. 17, 2002 BESIII Review Meeting

Data Rate Estimation Ø Level-1 Trigger Rate: 4 KHz (2 KHz Good Events) Ø Event Size: 12 KBytes Ø Data Rate to be Processed by DAQ: 48 MByte/sec Ø To be Recorded on Disk Array/Tape: 36 MByte/sec Sep. 17, 2002 BESIII Review Meeting

Technologies Ø Multi-level Buffering Ø Switch Network Ø Parallel Computing Ø Easily Upgrade and Port Ø Easily Scale Ø Reliable / Stable / Error Recovery Sep. 17, 2002 BESIII Review Meeting

Online Data Flow Sep. 17, 2002 BESIII Review Meeting

Power. PC/Vx. Works Based Test Environment Sep. 17, 2002 BESIII Review Meeting

Test Results – Programmed I/O System Controller: MVME 2431 -1 (350 MHz Power. PC 750) Operating System: Vx. Works 5. 4 Writing from CPU DRAM into VMEbus 32 -bit transfers: 400 ns/32 -bit write, 10 MB/s 16 -bit transfers: 400 ns/16 -bit write, 5 MB/s Reading from VMEbus into CPU DRAM 32 -bit transfers: 1, 100 ns/32 -bit read, 3. 6 MB/s 16 -bit transfers: 1, 100 ns/16 -bit read, 1. 8 MB/s Sep. 17, 2002 BESIII Review Meeting

Note: Ø The above VME access speeds were Programmed I/O speeds. Ø The program was written in C language and run in the context of the Vx. Works shell. Ø The tests were conducted on an idle system with no other competing processes nor any concurrent activity on the VMEbus. Sep. 17, 2002 BESIII Review Meeting

Question: Can we use Programmed Read to read out data from ADCs/TDCs modules? Sep. 17, 2002 BESIII Review Meeting

Estimation of One VME Readout Crate (1) Basic Assumptions: 1 crate = 16 modules Ø 1 module = 32 channels Ø Trigger rate: 4000 Hz Ø When one L 1 trigger comes, 10% channels have signals. Ø Each channel has its own Buffer containing 16 -bit data to be read. Ø Each module has a Hit Map (32 -bit) Ø Data readout is in A 24/D 16 mode and 1. 1 microsec/16 bit read Ø Sep. 17, 2002 BESIII Review Meeting

Estimation of One VME Readout Crate (1) (con’t) For each module: Data Ready Flag: Trigger Number: Hit Map: Data: 1 word 2 word 32 × 10% = 3. 2 words For each crate: 16 × ( 1 + 2 + 3. 2 ) = 115. 2 words When trigger rate is 4000 Hz, readout time will be: 4000 × 115. 2 bytes × 1. 1 microsec = 0. 51 sec Sep. 17, 2002 BESIII Review Meeting

Estimation of One VME Readout Crate (2) Basic Assumptions: 1 crate = 16 modules Ø 1 module = 32 channels Ø Trigger rate: 4000 Hz Ø When one L 1 trigger comes, 10% channels have signals. Ø Each module has a Global Buffer containing 16 bit channel number and 16 -bit data to be read. Ø Data readout is in A 24/D 32 mode and 1. 1 microsec/32 bit read Ø Sep. 17, 2002 BESIII Review Meeting

Global Buffer Data Format 31 Trigger Number Channel Number … Sep. 17, 2002 0 Event Data … Event Data Tail … BESIII Review Meeting

Estimation of One VME Readout Crate (2) (con’t) For each module: Data Ready Flag: Header: Data: Tail: 1 DWord 32 × 10% = 3. 2 DWords 1 DWord For each crate: 16 × ( 1 + 3. 2 + 1 ) = 99. 2 DWords When trigger rate is 4000 Hz, readout time will be: 4000 × 99. 2 bytes × 1. 1 microsec = 0. 44 sec Sep. 17, 2002 BESIII Review Meeting

So, the answer is: Ø The system controller does not have enough time to read out data from ADCs/TDCs by using Programmed Read. Ø The way to get faster reads is to use the DMA engine resident on the Universe VME <=> PCI bridge chip. It is designed to copy larger chunks of data from one bus to the other. Sep. 17, 2002 BESIII Review Meeting

MVME 2431 DMA Controller The Universe II has a DMA controller for high performance data transfer between the PCI bus and VMEbus. Ø Transfer size l Any Number of bytes from 1 byte to 16 Mbytes. Ø Transfer Data Width l D 08 (EO) l D 16 BLT l D 32 l D 64 l D 32 BLT l D 64 BLT (MBLT) Ø Sep. 17, 2002 BESIII Review Meeting

Readout Mode: Global Buffer + DMA Ø Block size must be counted somewhere on board. Ø To avoid too many DMA operations (DMA Interrupts), it’s better for each ADC/TDC board to buffer data of multiple triggers (e. g. 64, 128 triggers), and then interrupt the system controller to start a DMA operation. Sep. 17, 2002 BESIII Review Meeting

DMA Transfer Some typical values for DMA transfers: D 32 Read: 20 -- 23 MB/s D 32 Write: 22 -- 25 MB/s D 64 Read: 40 -- 45 MB/s D 64 Write: 45 -- 50 MB/s Note: This is the raw speed of the hardware without any software overhead. Sep. 17, 2002 BESIII Review Meeting

Main Tasks of Readout Ø DMA transfer Ø Calculation Ø Data unpacking & packing Ø Network transmission. Sep. 17, 2002 BESIII Review Meeting

R & D Prototype Sep. 17, 2002 BESIII Review Meeting

File Server Ø DELL Power. Edge 2500 Dual Intel Pentium III 1. 4 GHz Processors with 512 K Cache Ø Integrated 10/100 Ethernet Adapter Ø 1 GBytes SDRAM Ø SCSI RAID Controller Ø Six 18 GB SCSI Hard Drive, 10 K RPM Ø Intel Pro 1000 Fiber Network Adapter Ø Red Hat Linux 7. 3 Kernel 2. 4. 18 -3 smp Sep. 17, 2002 BESIII Review Meeting

Online Farm & Readout PC Nodes Ø DELL GX 260 Intel Pentium 4 1. 8 GHz Processor with 256 K Cache Ø Integrated 10/1000 Ethernet Adapter Ø 512 MBytes SDRAM Ø 40 GB EIDE 7200 RPM Hard Drive Ø Red Hat Linux 7. 3 Kernel 2. 4. 18 -3 Sep. 17, 2002 BESIII Review Meeting

Ethernet Switch Ø Catalyst 2948 G-L 3 Ø 48 Ports of 10/100 -Mbps Fast Ethernet Ø 2 Ports of 1 -Gbps Gigabit Ethernet Sep. 17, 2002 BESIII Review Meeting

Network Performance (Preliminary) Sep. 17, 2002 Block size Throughput (Bytes) (Mbps) 8 0. 47 1024 26. 15 131072 87. 69 16 0. 93 2048 36. 51 262144 88. 77 32 1. 82 4096 52. 58 524288 89. 26 64 3. 54 8192 65. 60 1048576 89. 52 128 6. 54 16384 75. 76 2097152 89. 66 256 11. 81 32768 81. 91 4194304 89. 72 512 18. 61 65536 85. 96 8388608 89. 76 BESIII Review Meeting

More Studies on Prototype Ø Performance Studies Network performance in parallel mode l Raid disk performance l write/read speed vs. config (raid 0/raid 5) Ø Data flow control software Ø Online filter software Ø Distributed histogram processing Ø Single event display Sep. 17, 2002 BESIII Review Meeting

Thanks! Sep. 17, 2002 BESIII Review Meeting