The 23 nd Geant 4 Collaboration Meeting Lund

The 23 nd Geant 4 Collaboration Meeting Lund University, Sweden Aug. 27~31, 2018 Geant 4 profiling for low energy physics Kihyeon Cho and Insung Yeo (KISTI) 1

Contents 1. 2. 3. 4. 5. Introduction Profiling on Version dependency (updated) Profiling on Event dependency Profiling on Machine dependency Summary 2

1. Introduction 3

Profiling system • Current status – High energy physics profiling (Fermilab) • Simpli. Carlo (Sequential) • CMSExp (Multi-Thread) – Low energy physics profiling • Using Brachytherapy code • Reported last collaboration meeting on mesh size(2017. 9) 4

Goal(New) • To check architecture for S/W – Version dependency – Event dependency • To check architecture for H/W – Machine dependency at supercomputer 4 – Supercomputer 4 vs. Supercomputer 5 (plan) ÞWe are using the profiling system of brachytherapy code 5

Bench mark @ current machine • The 4 th supercomputer @ KISTI Rpeak 300 TFlops Nodes Number 3, 176 Processor Intel Xeon X 5570 2. 93 GHz CPU Number 25, 408 Memory 76. 8 TB Storage 1, 061 TB Service date 2010. 11 6

Plan@ new supercomputer • 5 th Supercomputer – Processing: 25. 7 PF • Heterogonous: 25. 3 PF CS 400 w/KNL • CPU: 0. 4 PF CS 500 w/SKL – Storage • 20 PB SPS • 10 PB Archive – Schedule • Beta service (current) • Full service (3 Q, 2018) 7

Machines OS Install Geant 4 tachyon 2 Linux Compile Brachytherapy code tachyon 2 Igprof Run batch jobs Input file outputfile PBS - - Linux PBS - - tachyon 2 Linux - kumac Ig. Prof_iodine. FTFP_INCLXX _HP. 35. 0_ME M_LIVE_1. txt Sprof Run tachyon 2 Linux - kumac g 4 profiling_1 _X. tgz Draw plot hepkisti Linux - Output of igprof and sprof png Web site hepkisti Linux - Output of igprof and sprof html 8

Simpli. Carlo vs. Brachytherapy Simpli. Carlo Brachytherapy Energy More than Te. V Less than Me. V CPU time Long Too short CPU time / event (sec) 0. 01 0. 0002 Snap shot time 30 5 Institute Fermilab KISTI 9

Results • • • CPU time and Memory size (reported) Profiling on Mesh size (reported) Profiling on Version dependency Profiling on Event dependency Profiling on Machine dependency 10

2. Profiling on Version dependency 11

Settings Content Geant 4 Version Profiling 10. 02 Brachy version 10. 03 p 02 10. 03 p 03 10. 04 10. 02 • Trial time: 528 • Macro file – Iodium. Source. Macro. mac (endocavitary brachytherapy) – Iridium. Source. Macro. mac (interstitial brachytherapy) – Leipzig. Source. Macro. mac (superficial brachytherapy) • Physics list 1. QGSP_BIC_EMY 2. QGSP_BIC_EMZ 3. QGSP_BIC_LIV 12

<Trial time : 528> Geant 4 version : 10. 02 10, 000 Geant 4 version : 10. 03. p 02 10, 000

<Trial time : 528> Geant 4 version : 10. 03. p 03 10, 000 Geant 4 version : 10. 04

1. QGSP_BIC_EMY(1/2)

1. QGSP_BIC_EMY(2/2)

2. QGSP_BIC_EMZ(1/2)

2. QGSP_BIC_EMZ(2/2)

3. QGSP_BIC_LIV(1/2)

3. QGSP_BIC_LIV(2/2)

3. Profiling on Event dependency 21

Settings • Geant 4 Version: 10. 04 • Brachytherapy Version: 10. 02 • Macro file 1) Iodium. Source. Macro. mac (endocavitary brachytherapy) 2) Iridium. Source. Macro. mac (interstitial brachytherapy) 3) Leipzig. Source. Macro. mac (superficial brachytherapy) • Physics list – QGSP_BIC_EMY – QGSP_BIC_EMZ – QGSP_BIC_LIV 22

Comparing Profiling • Information – Mesh size: 300 – Trial time: 528 – Number of events: 2, 500~ 15, 000 – Snap shot time: 5 sec 23

1) Iodine QGSP BIC EMY Chi 2 NDf p 0 p 1 = 3. 42857 e-09 = 4 = 0. 00156 +/- 2. 72554 e-05 = 1. 01714 e-07 +/- 2. 79942 e-09 QGSP BIC EMZ Chi 2 NDf p 0 p 1 = 3. 42857 e-09 = 4 = 0. 00156 +/- 2. 72554 e-05 = 1. 01714 e-07 +/- 2. 79942 e-09 QGSP BIC LIV Chi 2 NDf p 0 p 1 = 5. 33333 e-09 = 4 = 0. 00159333 +/- 3. 39935 e-05 = 9. 6 e-08 +/- 3. 49149 e-09

1) Iodine QGSP BIC EMY Chi 2 NDf p 0 p 1 = 3. 87741 e-26 = 4 = 143. 342 +/- 9. 16573 e-14 = -1. 70299 e-17 +/- 9. 41416 e-18 QGSP BIC EMZ Chi 2 NDf p 0 p 1 = 3. 87741 e-26 = 4 = 143. 342 +/- 9. 16573 e-14 = -1. 70299 e-17 +/- 9. 41416 e-18 QGSP BIC LIV Chi 2 NDf p 0 p 1 = 3. 87741 e-26 = 4 = 143. 342 +/- 9. 16573 e-14 = -1. 70299 e-17 +/- 9. 41416 e-18 QGSP BIC EMY Chi 2 NDf p 0 p 1 = 0. 00677762 = 4 = 143. 522 +/- 0. 0383208 = 0. 00248423 +/- 3. 93595 e-06 QGSP BIC EMZ Chi 2 NDf p 0 p 1 = 0. 00609042 = 4 = 143. 522 +/- 0. 0363262 = 0. 00248455 +/- 3. 73108 e-06 QGSP BIC LIV Chi 2 NDf p 0 p 1 = 0. 00653394 = 143. 522 +/- 0. 0376256 = 0. 00248482 +/- 3. 86455 e-06

2) Iridium QGSP BIC EMY Chi 2 NDf p 0 p 1 = 1. 14728 e-35 = 4 = 0. 0016 +/= 7. 6 e-07 +/- 1. 57664 e-18 1. 61937 e-22 QGSP BIC EMZ Chi 2 NDf p 0 p 1 = 1. 14728 e-35 = 4 = 0. 0016 +/= 7. 6 e-07 +/- 1. 57664 e-18 1. 61937 e-22 QGSP BIC LIV Chi 2 NDf p 0 p 1 = 1. 14728 e-35 = 4 = 0. 0016 +/= 7. 6 e-07 +/- 1. 57664 e-18 1. 61937 e-22

2) Iridium QGSP BIC EMY Chi 2 NDf p 0 p 1 = 1. 38133 e-25 = 4 = 174. 411 +/- 1. 72999 e-13 = -3. 40598 e-17 +/- 1. 77688 e-17 QGSP BIC EMZ Chi 2 NDf p 0 p 1 = 1. 38133 e-25 = 4 = 174. 411 +/- 1. 72999 e-13 = -3. 40598 e-17 +/- 1. 77688 e-17 QGSP BIC LIV Chi 2 NDf p 0 p 1 = 1. 38133 e-25 = 4 = 174. 411 +/- 1. 72999 e-13 = -3. 40598 e-17 +/- 1. 77688 e-17 QGSP BIC EMY Chi 2 NDf p 0 p 1 = 0. 00298349 = 4 = 174. 531 +/- 0. 0254248 = 0. 00210887 +/- 2. 6114 e-06 QGSP BIC EMZ Chi 2 NDf p 0 p 1 = 0. 0034471 = 4 = 174. 531 +/- 0. 027329 = 0. 00210875 +/- 2. 80697 e-06 QGSP BIC LIV Chi 2 NDf p 0 p 1 = 0. 00182394 = 174. 534 +/- 0. 0198793 = 0. 00210854 +/- 2. 04182 e-06

3) Leipzig QGSP BIC EMY Chi 2 NDf p 0 p 1 = 3. 42857 e-09 = 4 = 0. 00176 +/- 2. 72554 e-05 = 1. 81714 e-07 +/- 2. 79942 e-09 QGSP BIC EMZ Chi 2 NDf p 0 p 1 = 3. 42857 e-09 = 4 = 0. 00176 +/- 2. 72554 e-05 = 1. 81714 e-07 +/- 2. 79942 e-09 QGSP BIC LIV Chi 2 NDf p 0 p 1 = 3. 42857 e-09 = 4 = 0. 00176 +/- 2. 72554 e-05 = 1. 81714 e-07 +/- 2. 79942 e-09

3) Leipzig QGSP BIC EMY Chi 2 NDf p 0 p 1 = 3. 87741 e-26 = 4 = 184. 646 +/- 9. 16573 e-14 = -1. 70299 e-17 +/- 9. 41416 e-18 QGSP BIC EMZ Chi 2 NDf p 0 p 1 = 3. 87741 e-26 = 4 = 184. 646 +/- 9. 16573 e-14 = -1. 70299 e-17 +/- 9. 41416 e-18 QGSP BIC LIV Chi 2 NDf p 0 p 1 = 3. 87741 e-26 = 4 = 184. 646 +/- 9. 16573 e-14 = -1. 70299 e-17 +/- 9. 41416 e-18 QGSP BIC EMY Chi 2 NDf p 0 p 1 = 0. 00349177 = 4 = 184. 871 +/- 0. 0275055 = 0. 00340387 +/- 2. 8251 e-06 QGSP BIC EMZ Chi 2 NDf p 0 p 1 = 0. 00361328 = 4 = 184. 878 +/- 0. 0279799 = 0. 00340306 +/- 2. 87383 e-06 QGSP BIC LIV Chi 2 NDf p 0 p 1 = 0. 00261968 = 4 = 184. 866 +/- 0. 0238243 = 0. 0034045 +/- 2. 447 e-06

4. Machine dependency 30

Tachyon 2 node dependency Version: Geant 4. 10. 04 Trial time: 528 Events: 10, 000 Macro file 1) Iodium. Source. Macro. mac (endocavitary brachytherapy) 2) Iridium. Source. Macro. mac (interstitial brachytherapy) 3) Leipzig. Source. Macro. mac (superficial brachytherapy) • Physics list – QGSP_BIC_EMY – QGSP_BIC_EMZ – QGSP_BIC_LIV • Machine node (10 nodes) • • # 1 2 3 Tachyon 2 Node# 2883 2899 1808 4 5 6 7 8 9 10 2114 1996 2010 2119 2272 2645 1765

1) Iodine QGSP BIC EMY Chi 2 NDf p 0 = 1. 88079 e-36 = 9 = 0. 0026 +/- 1. 4456 e-19 ******************** QGSP BIC EMZ Chi 2 = 1. 88079 e-36 NDf = 9 p 0 = 0. 0026 +/- 1. 4456 e-19 ******************** QGSP BIC LIV Chi 2 = 1. 88079 e-36 NDf = 9 p 0 = 0. 0026 +/- 1. 4456 e-19

QGSP BIC EMY Chi 2 NDf p 0 = = = 0 9 143. 342 1) Iodine +/- 0. 316228 ******************** QGSP BIC EMZ Chi 2 = 0 NDf = 9 p 0 = 143. 342 +/- 0. 316228 ******************** QGSP BIC LIV Chi 2 = 0 NDf = 9 p 0 = 143. 342 +/- 0. 316228 QGSP BIC EMY Chi 2 NDf p 0 0. 00163299 = = = 0. 00024 9 168. 341 +/- ******************** QGSP BIC EMZ Chi 2 = 0. 00065 NDf = 9 p 0 = 168. 340 +/******************** QGSP BIC LIV Chi 2 = 0. 00024 NDf = 9 p 0 = 168. 339 +/- 0. 00268742 0. 00163299

2) Iridium QGSP BIC EMY Chi 2 NDf p 0 = = = 1. 6 e-08 9 0. 00922 +/- 1. 33333 e-05 ******************** QGSP BIC EMZ Chi 2 = 1. 6 e-08 NDf = 9 p 0 = 0. 00922 +/- 1. 33333 e-05 ******************** QGSP BIC LIV Chi 2 = 1. 6 e-08 NDf = 9 p 0 = 0. 00922 +/- 1. 33333 e-05

QGSP BIC EMY Chi 2 NDf p 0 = = = 0 9 174. 411 2) Iridium +/- 0. 316228 ******************** QGSP BIC EMZ Chi 2 = 0 NDf = 9 p 0 = 174. 411 +/- 0. 316228 ******************** QGSP BIC LIV Chi 2 = 0 NDf = 9 p 0 = 174. 411 +/- 0. 316228 QGSP BIC EMY Chi 2 NDf p 0 = = = 0. 00024 9 195. 605 +/- 0. 00163299 ******************** QGSP BIC EMZ Chi 2 = 0. 00041 NDf = 9 p 0 = 195. 608 +/- 0. 00213437 ******************** QGSP BIC LIV Chi 2 = 0. 00044 NDf = 9 p 0 = 195. 607 +/- 0. 00221108

3) Leipzig QGSP BIC EMY Chi 2 NDf p 0 = 7. 52316 e-36 = 9 = 0. 0036 +/- 2. 89121 e-19 ******************** QGSP BIC EMZ Chi 2 = 7. 52316 e-36 NDf = 9 p 0 = 0. 0036 +/- 2. 89121 e-19 ******************** QGSP BIC LIV Chi 2 = 7. 52316 e-36 NDf = 9 p 0 = 0. 0036 +/- 2. 89121 e-19

QGSP BIC EMY Chi 2 NDf p 0 = 8. 07794 e-27 = 9 = 184. 646 +/- 3) Leipzig 9. 4739 e-15 ******************** QGSP BIC EMZ Chi 2 = 8. 07794 e-27 NDf = 9 p 0 = 184. 646 +/- 9. 4739 e-15 ******************** QGSP BIC LIV Chi 2 = 8. 07794 e-27 NDf = 9 p 0 = 184. 646 +/- 9. 4739 e-15 QGSP BIC EMY Chi 2 NDf p 0 0. 00213437 = = = 0. 00041 9 218. 921 +/- ******************** QGSP BIC EMZ Chi 2 = 0. 00024 NDf = 9 p 0 = 218. 920 +/******************** QGSP BIC LIV Chi 2 = 0. 00021 NDf = 9 p 0 = 218. 921 +/- 0. 00163299 0. 00152753

Average Physics List Iodine Iridium Leipzig EMY 0. 0026± 0. 0000 0. 00922± 0. 00001 0. 0036± 0. 0000 EMZ 0. 0026± 0. 0000 0. 00922± 0. 00001 0. 0036± 0. 0000 LIV 0. 0026± 0. 0000 0. 00922± 0. 00001 0. 0036± 0. 0000 EMY 143. 342± 0. 000* 174. 411± 0. 000* 184. 646± 0. 0000 EMZ 143. 342± 0. 000* 174. 411± 0. 000* 184. 646± 0. 0000 LIV 143. 342± 0. 000* 174. 411± 0. 000* 184. 646± 0. 0000 EMY 168. 341± 0. 002 195. 605± 0. 002 218. 921± 0. 002 EMZ 168. 340± 0. 002 195. 608± 0. 002 218. 921± 0. 002 LIV 168. 339± 0. 002 195. 607± 0. 002 218. 921± 0. 002 CPU Time (sec) Total Memory in First event(MB) Total Memory in Last event(MB) • * chi^2 is zero *χ2 is zero. • The fittings show that χ2/Do. F is very small. ⇒ There is no machine dependence. 38

5. Summary 39

Summary • Done – – Low energy physics profiling code Profiling on version dependence ⇒ updated Profiling on events dependence ⇒ linearity Profiling on machine ⇒ No dependence @ 4 th KISTI supercomputer • Future plan – To use Geant 4 profiling test – To support homepage – To use new architecture • 5 th supercomputer @ KISTI • Scalability • Machine dependence 40

Acknowledgement • Soon Yung Jun • Susanna Guatelli • Hyungil Chun 41

Thank you! 42

Back up

Introduction • Why profiling? – Diversity of physics applications – Evolving Computing Þ S/W development Þ Profiling tools ÞTo draw community interests for collateral effort 44

• There are many efforts to develop HEP software to save CPU Time and Memory. (https: //g 4 cpt. fnal. gov) • However, there are not enough codes for low energy profiling system. ⇒ To use FNAL Geant 4 computing performance profiling protocol and tools as a starting point 45

Goal(New) • To check architecture for S/W – Version dependency • To check architecture for H/W – Machine dependency at supercomputer 4 – Supercomputer 4 vs. Supercomputer 5 ÞWe are using the profiling system of brachytherapy code ÞTitle: A study of Geant 4 tool kit using low energy physics profiling system 46

Brachytherapy • Using Brachytherapy simulation in advanced examples 47

Cf. Goal on Brachytherapy • • Low Energy Physics Profiling CPU/Memory Usage Mesh Size(Done) Version dependency(updated) Events dependency(new) Machine dependency(new) Scalability for new computing architecture (KISTI supercomputer)- plan 48

Methodology 49

Physics List Name Content Energy QGS Quark Gluon String Model > ~20 Ge. V FTF Fritiof string model > ~ 5 Ge. V BIC Binary Cascade < ~ 10 Ge. V BERT Bertini-style cascade < ~10 Ge. V HP High Precision neutron model >20 Me. V P G 4 Precompund model used for de-excitation 50

Scalability to Future Hybrid machine • The 5 th supercomputer @ KISTI Rpeak 25. 7 PFlops Price 54, 700, 000 won ($50 million) Many core Processor Intel Xeon Phi Knight landing processing Multicore CPU Intel Xeon Skylake process Service date 1 Q, 2018 Company Cray • 100 X Scalability ? • Mini component supercomputer? 1 PFlops 51