Evaluation of the LDC Computing Platform for Point

Evaluation of the LDC Computing Platform for Point 2 Super. Micro X 7 DBE-X Andrey Shevel CERN PH-AID ALICE DAQ (Internal report) CERN November 2006 4 Dec 2006 Testing the machine (X 7 DBE-X) with 6 D-RORCs 1

Purpose ® Background: ® Test the machine Elonex Supermicro based on motherboard X 7 DBE-X (2 x. Dual Core Xeon 3. 0 GHz) as LDC with 6 D-RORCs ® Evaluation program: Linux installation. DATE V 5 installation. D-RORC throughput with DATE. Perform a range of runs with internal D-RORC generator and receiving through DATE. ® Perform a range of long runs (two days each) in configuration with the test machine as LDC and another machine as GDC. ® Quick comparison with another machine based on motherboard X 6 DHE-XB (Dual Xeon 2. 8 MHz). ® ® 4 Dec 2006 Testing the machine (X 7 DBE-X) with 6 D-RORCs 2

Architecture of motherboard 4 Dec 2006 Testing the machine (X 7 DBE-X) with 6 D-RORCs 3

Supermicro X 7 DBE-X CPU 2 x Dual Core Intel Xeon, cache L 2 – total 4 MB Chipset Intel 5000 P (Black. Ford) FSB 1333 MHz Memory 2 GB (max 32 GB), 533 MHz, DDR 2 3 x 64 -bit 133 MHz PCI-X, I/O 3 x PCI-X (64 -bit 133 MHz) 3 x PCI-X (64 -bit 100 MHz) Network 2 x Gb. E, Intel 631 x. ESC/632 x. ESB Disk 120 GB, SATA Graphics ATI Technologies Inc ES 1000 Periphery 2 x USB front, 2 x USB back Serial, PS/2 Chassis 4 U (tower) pcald 44. cern. ch 4 Dec 2006 Testing the machine (X 7 DBE-X) with 6 D-RORCs 4

Software Installation ® Linux ® SLC 4. 4 with kernel 2. 6. 9 -42. 0. 3. EL. cernsmp no specific problems: USB boot successful, BIOS – everything is OK ® To use D-RORC modules the BIOS setting is needed to be changed: all PCI slots were set to 100 MHz ® DATE 5. 25 installation ® My. Sql ® mysqltcl ® BWidget ® SMI ® DIM ® Drivers: rorc and physmem have been installed as rpm with additional scripts: DATE_basic_Install-SLC 4 x. bash, DATE_User_Install-SLC 4 x. bash, DATE_Drivers. Install. bash ® No specific problems 4 Dec 2006 Testing the machine (X 7 DBE-X) with 6 D-RORCs 5

Common info about measurement � Total main memory volume = 2 GB • • 1 GB for Linux (in /etc/grub. conf) 1 GB for physmem � 6 cards D-RORC rev 4 (internal data generator). � All measurement were done with DATE 5. 25. No received event data were recorded to HD. The data from the database were analyzed with scripts. The pictures were built up with gnuplot. � In total there were performed 390 measurement runs. Because the figures (rates, bandwidth) are fluctuating every run was continued at least 5 minutes. � Two points: random event size with max=10**4 and max=10**6 Bytes were measured during two days each (LDC on test machine, GDC on different machine). � All measurements with randomly distributed event size were performed with event checking in DATE. � D-RORC Page Size was 10**5 Bytes. 4 Dec 2006 Testing the machine (X 7 DBE-X) with 6 D-RORCs 6

Event rate of one card 4 Dec 2006 Testing the machine (X 7 DBE-X) with 6 D-RORCs 7

Bandwidth of readout of one card 4 Dec 2006 Testing the machine (X 7 DBE-X) with 6 D-RORCs 8

Example of edit. Db page 4 Dec 2006 Testing the machine (X 7 DBE-X) with 6 D-RORCs 9

Example of Memory Banks page 4 Dec 2006 Testing the machine (X 7 DBE-X) with 6 D-RORCs 10

Event rate for several cards 4 Dec 2006 Testing the machine (X 7 DBE-X) with 6 D-RORCs 11

Bandwidth for several cards 4 Dec 2006 Testing the machine (X 7 DBE-X) with 6 D-RORCs 12

Event rate: Consistency. Check. Level 4 Dec 2006 Testing the machine (X 7 DBE-X) with 6 D-RORCs 13

Bandwidth: Consistency. Check. Level 4 Dec 2006 Testing the machine (X 7 DBE-X) with 6 D-RORCs 14

Event rate: Number. Of. Cards, Consistency. Check. Level 4 Dec 2006 Testing the machine (X 7 DBE-X) with 6 D-RORCs 15

Bandwidth: Number. Of. Cards, Consistency. Check. Level 4 Dec 2006 Testing the machine (X 7 DBE-X) with 6 D-RORCs 16

Event rate: D-RORC Page Size 4 Dec 2006 Testing the machine (X 7 DBE-X) with 6 D-RORCs 17

Bandwidth: D-RORC Page Size 4 Dec 2006 Testing the machine (X 7 DBE-X) with 6 D-RORCs 18

X 6 DHE-XB vs X 7 DBE-X Parameter name CPU type Chipset FSB Memory 4 Dec 2006 PCI I/O Network SYSBENC X 6 DBE-XB X 7 DBE-X 2 x Xeon 2. 8 GHz, 1 MB 2 x Xeon 5160 dual core, 3. 0 GHz, 4 MB L 2 cache E 7520 Intel 5000 P (Black. Ford) 800 MHz 1333 MHz 2 GB, DDR 333 2 GB, DDR-2 533 MHz 2 x PCI-X (64/133) 4 x PCI-X (64/100) 1 x PCI-e (x 4) 3 x PCI-X (64 -bit 100 MHz) 3 x PCI-X (64 -bit 133 MHz) 2 x Gb. E, BCM 5721 2 x Gb. E, Intel 631 x. ESB/632 x. ESB Testing the machine (X 7 DBE-X) with 6 D-RORCs 19

X 6 DHE-XB vs X 7 DBE-X (continuation) Parameter X 6 DBE-XB X 7 DBE-X name Event rate for 1 DRORC with 88. 9 KHz 87. 2 KHz the size 1 KB Event rate for 6 DRORCs with size 1 KB 4 Dec 2006 Bandwidth of 6 KHz 12. 2 KHz Testing the machine (X 7 DBE-X) with 6 D-RORCs 20

Recommendation and conclusion � Recommendation: if you need to allocate D-RORC cards in the machine and like to have maximum bandwidth for X 7 DBE-X: • it is better to avoid to use PCI slot 4 & PCI slot 5 at the same time for high bandwidth information sources. � Conclusion ® In the test it was shown bandwidth (1. 96 GB/sec) when all 6 D-RORC cards are in operation and event fragment size is more 10**5 bytes when Consistency. Check. Level=0. ® X 7 DBE-X shows increased event rate and bandwidth with 6 D-RORCs, also it has more CPU power in comparison to X 6 DHE-XB. ® The configuration LDC on test machine and GDC on another machine shown about the speed 118 MB/sec with no error during two days test. ® Randomly distributed event fragment size with maximum 10**6 bytes gives total bandwidth 1. 53 GB/sec during two days run. ® The machine on the base X 7 DBE-X is excellent to be used as LDC with 6 D-RORC cards. 4 Dec 2006 Testing the machine (X 7 DBE-X) with 6 D-RORCs 21

Thank you! Klaus Schossmaier, Sylvain Chapeland, Ulrich Fuchs and PH/AID crew 4 Dec 2006 Testing the machine (X 7 DBE-X) with 6 D-RORCs 22