MAP Project T Bowcock A Kinvig I Last
MAP Project T. Bowcock, A. Kinvig, I. Last M. Mc. Cubbin, A. Moreton C. Parkes, G. Patel University of Liverpool
Introduction w Monte Carlo Array Processor n justification w Status n n Hardware Software w COMPASS w Summary Friday, 04 February 2000 University Of Liverpool
Monte Carlo w At LHCb about 1 interaction /25 ns ! n n 4*1014/year if you want to do physics you need to know the backgrounds l n generating just the signals doesn’t work need to generate large MC samples l O(107) n to O(108) events. LHCb needs to do this now! Friday, 04 February 2000 University Of Liverpool
Philosophy w Cheapest possible that works n n No Gbit ethernet until price falls Don’t buy top of range processors No SMP boards No tapes obsolete? l Develop architecture with future in mind] l Friday, 04 February 2000 University Of Liverpool
MAP hardware w 300 processors n n n 400 MHz PII 128 Mbytes memory 3 Gbytes disk D-Link 100 Base. T ethernet +hubs commercial units BUT l custom boxes for packing and cooling Friday, 04 February 2000 University Of Liverpool
MAP Friday, 04 February 2000 University Of Liverpool
MAP cont’d Friday, 04 February 2000 University Of Liverpool
MAP cont’d Friday, 04 February 2000 University Of Liverpool
External Ethernet MAP Architecture MAP Slaves Master Friday, 04 February 2000 Hub 100 Base. T University Of Liverpool
MAP software w Overview n Linux based on Red. Hat 5. 2 l stripped down version l n n Batch System Network Overloading of master ethernet interfaces (300 at once) implied need for total control of data flow Broadcast of control required phased reply Control At the UDP level l Robust Packet Handling l Friday, 04 February 2000 University Of Liverpool
MAP user w Prepare a job w Submit to Batch Queue w Histograms/Ntuples transmitted back at end of job/DST’s w Random Numbers handled automatically Friday, 04 February 2000 University Of Liverpool
MAP Status w In production for about 6 weeks w 300 Processors n n n produced about 240, 000 LHCb events 24/hrs 5 million events produced to date Also produced DELPHI DST’s (500, 000 24 Hrs) w All Processors tested n Further Air-Conditioning installed l fully commissioned 22/11/99 Friday, 04 February 2000 University Of Liverpool
MAP Issues w Packet Loss n n At UDP (or frame level) have to handle with code. Now not a probem(!) Higher performance with shielded cables? w no w Power n Infrastructure for cooling w Power up/down Friday, 04 February 2000 University Of Liverpool
Emergency Power Down w Unplanned power interruption n n Exploding substation! About 4% of PC’s need manual intervention Friday, 04 February 2000 University Of Liverpool
MAP capabilites w Can be used in “throwaway” mode n Also write events as genenerated w MAP possesses 1 Tbyte internal storage n n n 3 Gbytes/machine events stored locally (1 million events) repeatedly analyse QUICKLY w MAP can handle interprocess communication Friday, 04 February 2000 University Of Liverpool
MAP++ Friday, 04 February 2000 University Of Liverpool
COMPASS Computerized Analysis and Storage Server
COMPASS w Purpose n n Will show this in place and working with MAP Model for LHC analysis store events on disks (cheap!) l move JOB to the DATA l NO HSM l Friday, 04 February 2000 University Of Liverpool
Outline w Hardware w Linux Device Drivers w Linux Installation and Limits w Benchmarking Tests w Results w Future Friday, 04 February 2000 University Of Liverpool
Trial Hardware w Dell Power. Edge Server, 450 Mhz Pentium III, 256 Mb RAM with 4 internal SCSI disks. w 4 Power. Vault 1200 Disk Servers each with 8 Ultra Wide SCSI LVD disks. (spindle 7200 rpm) Total > 1 Tb disk space w Adaptec Ultra Wide SCSI cards. Friday, 04 February 2000 University Of Liverpool
ITS w Purchased Rack mounted n n n 1 TByte based on 50 GByte 7200 rpm disks Redundant Power Supplies 15 KGBP/Tbyte including 2 500 MHz PIII w More storage underway Friday, 04 February 2000 University Of Liverpool
Linux Devices Drivers w Linux Device Drivers: n n Devices accessed through special files in /dev directory specifying block or character device and major / minor number pairs. Major number refers to a device driver e. g. 8 is a SCSI disk (see /usr/src/linux/include/linux/major. h) For disks, minor number refers to disk / partition on disk e. g. /dev/sda major: 8 minor: 0 first SCSI disk found on system /dev/sda 1 major: 8 minor: 1 first partition /dev/sda 15 major: 8 minor: 15 last partition on first disk /dev/sdb major: 8 minor: 16 second SCSI disk found on system minor numbers are 8 -bit i. e. only have values in range 0 -255 only 16 disks per disk major number. Friday, 04 February 2000 University Of Liverpool
Linux Installation & Limits w Red. Hat Linux 5. 2: Kernel 2. 0. x n n Used at Liverpool and CERN – problem: only one SCSI major number is defined – maximum of 16 SCSI disks allowed. Kernel “hacking” necessary to register new SCSI major number with system. w Red. Hat Linux 6. 0: Kernel 2. 2. x n n Defines 8 SCSI major numbers : 8, 65 -71 - max. 128 SCSI disks. Have to create some special files in /dev by hand – relatively trivial with mknod w Physical limit of only 4 PCI slots for SCSI cards on motherboard Friday, 04 February 2000 University Of Liverpool
Benchmarking Tests w Use CERN sequential IO tests for read / write / calibration. n n Block sizes from 1024 Bytes to 0. 5 MBytes Calculates average write rate over previous 10 writes Read. . . Calibration: Comment out write statement and run write tests again. w Modified version of above calculates averages over the whole file. Friday, 04 February 2000 University Of Liverpool
Results w All disks accessible w Performance uniform n n n writing about 20 MBytess reading at 50 MBytes (or better) large block-sizes faster Friday, 04 February 2000 University Of Liverpool
Future w Can we find funding for large(r) scale prototype? n n Applications outside of Physics Interdisciplinary funding Friday, 04 February 2000 University Of Liverpool
Summary w MAP yields high performance at low cost w Storage can be cheap w R&D to Enhance performance w Production for LHCb vertex detector Friday, 04 February 2000 University Of Liverpool
- Slides: 27