ROOT IO Performance In Multihread Environment Philippe Canal

ROOT I/O Performance In Multihread Environment Philippe Canal Fermilab Philippe CANAL root. cern. ch

Overview • Support for I/O in multi-thread environment • CMS Event • CMS Condition Database • Conclusion Philippe CANAL root. cern. ch 2

Support • With be only for C++11 but both in v 5 and v 6. – Relies on std: : atomics • In v 5, limited to non-interactive sessions – High deadlock risk when starting the command line. • Cost of atomics (and thread_locals) about 5% of streaming time (mitigated by C++11 being 2% faster). root. cern. ch 3

Removal of unnecessary serialization • Update TClass. Ref – From a linked list of ref per TClass object update at each creation/deletion of TClass. Ref – To a single pointer to TClass* per TClass object shared by the TClass. Ref • TClass* is allocated once per TClass and never changes • TClass: : Get. Class – Move code around to reduce length the lock is held. • TClass: : Get/Find. Streamer. Info – Remove use of lock in the common case by caching in an atomic the most recently found for each Tclass • TThread: : Self – Remove linear search doing string comparison by using thread local storage. • Remove locks in TBase. Class by caching information. root. cern. ch 4

Test Machine • Cpu: Intel Xeon X 5570 @ 2. 93 GHz – 2 CPUs for 8 cores (16 including hyperthreading). – Cache size: 8192 KB • 12 GB of DDR 3 at 1333 MHz root. cern. ch 5

CMS Event • • CMS Event file 133 MB, 100 entries. One TFile and TTree per thread. Use TTree. Cache and slightly modified Make. Project lib. Less than 5% serialization 7. 00 6. 00 Strong Scaling 5. 00 Weak Scaling 4. 00 3. 00 2. 00 1 root. cern. ch 2 3 4 5 6 7 8 6

Test Description • 226 distincts CMS condition database objects • 233 MB of data • Run example with no thread enabled then 1 through 8 threads. • Load the data (amount varies) into 1 TBuffer. File per thread. – Small Set: first 3 objects for 393 KB. • Each thread deserialize the content multiple times root. cern. ch 7

CMS Condition Database 8 Random 7 Ordered memcpy Small Set Speed Up 6 Small set memcpy 5 4 3 2 1 1 root. cern. ch 2 3 4 5 Number of Threads 6 7 8 8

Data Bandwidth 64 Total Data Processed in GB/s 32 16 8 4 2 Random Ordered 1 1 0. 5 root. cern. ch 2 3 4 5 6 memcpy 7 Small Set 8 Small set memcpy Number of Threads 9

Data Bandwidth 40 35 Random Total Data Processed in GB/s 30 Ordered memcpy Small Set 25 Small set memcpy 20 15 10 5 0 1 root. cern. ch 2 3 4 5 Number of Threads 6 7 8 10

Observations • Average streaming performance nicely (1 GB/s) • But varies a lot • Condition database objects very varied. – Top 6 objects takes 60% of the time but 15% of the size. – This will limit the amount of possible parallelism. • For example – Ideal. Geometry. Record / PGeometric. Det • 8 MB (3% of total) • 23557 std: : string in an object in a vector • Read @ 325 MB/s … 12% of total – Si. Pixel. Gain. Calibration. Offline • 65 MB (28% of total) … Read @ 4680 MB/s … 6% of total – L 1 Mu. DTPta. Lut • 448 Kb (0. 2% of total). . . Slowest read @ 25 MB/s … 8% of total … • Contains a vector<map<short, short> >. root. cern. ch 11

Worst case. • Class L 1 RPCConfig – Each of those contains 2 bytes! – 22% of time, 3% of space • High serialization • Most time consuming object. • Contains vector of 93160 objects – which contains an array of 6 objects. – Atomics and lock play a role but not enough to explain behavior • But no clear explanation – Maybe try running in Vtune 2. 40 C++03 2. 20 C++11 2. 00 Speed up 1. 80 1. 60 1. 40 1. 20 1. 00 0. 80 0. 60 1 root. cern. ch 2 3 4 5 Number of Threads 6 7 8 12

Conclusion • ROOT I/O is now thread friendly – It works! – Less than 5% serialization on CMS Events TTree. – Less than 15% serialization on all CMS cond db objects. • Some object layout lead to poor performance and poor scalability. • More can be done to optimize – Reduce number of ‘class/version/checksum’ searches. • To reduce the number of atomic and thread local uses. – Change byte swap order (increase memcpy case) – Continue refactoring of the I/O internals • Increase vectorization, reduce branches, etc. root. cern. ch 13