Maximising job throughput using HyperThreading Alastair Dewhurst Dimitrios

Maximising job throughput using Hyper-Threading Alastair Dewhurst, Dimitrios Zilaskos RAL Tier 1 Setup The RAL TIER 1 batch farm consists of several multicore, hyperthreading capable CPUs. Increases in the amount of memory per node combined with experiences from other sites made hyperthreading an attractive option for increasing job throughput. RAL supports all LHC VOs, with prime users being Atlas, CMS and LHCB, and a 10% of resources is devoted to non-LHC VOs The virtual cores provided by hyperthreading could double the batch farm capacity, however the amount of memory available in the batch nodes did not permit that. LHC jobs require more than 2 GB RAM to run smoothly. Comparison of physical cores, full HT cores, and optimum setup 12000 Physical cores compared to HT cores 4 1200 3. 5 1000 3 800 Physical cores(before) 600 HT Cores RAM (GB) 2. 5 2 RAM per core With all HT cores enabled total job slots capacity could double. In practice memory constrains resulted in an increase of 30% 10000 8000 Cores Available RAM under different setups 1400 RAM per vcore 1. 5 400 6000 4000 RAM per job slot 1 2000 200 0. 5 HT Physical cores(before) (n on 11 HT Cores Job slots(after) -v 10 igl en 20 -v 11 20 0 ) l -d el en igl en -v 10 20 vis st er lu -c igl io n en igl 20 08 20 20 20 08 -v 20 -v 09 igl -st re en 20 20 Method Results • Each generation of batch farm hardware with hyperthreading capability was benchmarked with HEPSPEC, progressively increasing the number of threads up to the total number of virtual cores • Benchmarks at that time were conducted using Scientific Linux 5. Scientific Linux 6 benchmarks were run later as the batch farm was set to be updated. • Scientific Linux 6 performed slightly better than Scientific Linux 5. The overall trend was identical • Power, temperature, disk I/O and batch server performance were closely monitored • The results indicated a nearly linear increase in the hepspec scores, flattening at about 14 threads for 2 CPU 4 core nodes and 20 threads for 2 CPU 6 core nodes • The revealed sweet spots were then configured for use in the batch farm to discover where production VO jobs would perform optimally • Overall, 2000 extra job slots and 9298 extra hepspec were added in the batch farm using already available hardware • Average job time increases as expected, but overall job throughput increased • Network/disk/power/temperature usage did not increase in a way that could negatively affect the overall throughput or require additional maneuvers • Batch server was able to handle the extra job slots • Of critical importance is the sharp drop in job efficiency as job slots approach the upper hyperthreading limit. This means that real world VO jobs would suffer if we went for full batchfarm HEPSPEC performance! Make HEPSPEC results 200 Job Slots per WN Efficiency Average Job Length (mins) Standard Deviation (mins) Number of jobs Dell 12 0. 9715 297 370 19065 Viglen 12 0. 9757 320 390 23864 Dell 14 0. 9719 238 326 6118 Viglen 14 0. 9767 270 341 11249 Dell 16 0. 9859 343 254 6550 Viglen 16 0. 985 304 249 8756 Dell 18 0. 9781 377 390 5014 Viglen 18 0. 9808 350 391 6263 Dell 20 0. 9758 318 346 11339 Viglen 20 0. 9756 260 285 11229 Dell 22 0. 9747 387 315 6317 Viglen 22 0. 9783 305 236 6307 Dell 24 0. 9257 544 373 6650 Viglen 24 0. 9311 372 278 6713 Evolution of Job efficiency as more HT cores are being used 1. 01 0. 99 0. 97 Job efficiency 250 09 am lin e HT (n on 11 -v 11 20 0 ) l -d el en igl en 10 20 st er lu -c -v vis igl io n en igl -v 09 20 10 09 20 -st re am lin e 0 0. 95 dell-2011 viglen-2011 0. 93 0. 91 2009 -streamline 150 0. 89 2009 -viglen 12 HEPSPEC 2010 -clustervision 14 16 18 Number of threads 20 22 24 2010 -viglen 2011 -dell 100 Conclusions • • 50 0 8 9 10 11 12 13 14 15 16 17 18 Number of threads 19 20 21 22 23 24 New procurements now take into account the hyperthreading capabilities For 2012, dual 8 core CPU systems go up to 32 virtual cores Systems were procured with 128 GB RAM in order to exploit full hyperthreading capabilities Dual Gigabit links, in the future single 10 GB as they became more cost effective So far RAID 0 software raid setup has proven sufficient for disk I/O Performance gains so far on par with previous generations By spending a bit extra on RAM, we save more by buying fewer nodes This also saves machine room space, cables, and power HESPEC % increase with full HT 2012 procurements benchmarks 25% 400 Evolution of batch farm size 20% 16000 350 14000 300 12000 250 15% HEPSPEC 10% Job slots 10000 5% Physical cores 8000 HT cores 6000 ocf-2012 200 dell-2012 150 100 0% 2009 -streamline 2009 -viglen 2010 -clustervision 2010 -viglen 2011 -dell 4000 50 2000 0 16 0 pre 2012 batch farm 17 18 19 20 21 22 23 24 25 Number of threads 26 27 28 29 30 31 32 2012 -present farm Acknowledgements: RAL Tier 1 team, especially John Kelly and James Adams