OCAPIE project Ottimizzazione di CAlcolo Parallelo Intensivo Applicate

OCAPIE project (Ottimizzazione di CAlcolo Parallelo Intensivo Applicate a problematiche in ambito Energetico) W. Borreani (INFN) & G. Ciaccio (DIBRIS-Unige) Outline -Project description and objectives -Codes to be used -(some) KNL architecture -KNL Genova Cluster -Performance tests and scalability OCAPIE collaboration: P. Saracco, W. Borreani, A. Brunengo, V. Calvelli, G. Ciaccio (Unige), M. Corosu, S. Farinon, G. Lomonaco (Unige) R. Musenich, M. Osipenko, M. Ripani, M. Taiuti (Unige) + Poli. To, Poli. Mi, ANN (Ansaldo Nucleare spa), ASG Superconductors CCR Workshop – May 22 -26 2017 - LNGS

Project description and objectives - Financed in 2016 by “Compagnia di San Paolo” (Torino), following a public call and selection - Amount of the project: 200. 000 € (180. 000 financed) - 63. 000 to buy servers, 108. 000 for 2 Ad. R (W. Borreani, V. Calvelli) + movements - Objectives: - - - “the study and the development of parallel computing technologies useful to the description of multiscale/multiphysics phenomena in the broad field of energy production and distribution: complex systems modelization usually is grounded on a variety of physical models, because of the wide differences in the scales at which different physical phenomena occur. ” “The traditional approach to describe such complex systems relies on the sequential use of simulation tools specifically tailored for each subset of the physical problems involved: (…)Realization of a full and reliable model usually requires many iterations of this computational process; then completion of the calculation is computationally expensive and requires large amount of human work…” to develop and optimize an intermediate approach: parallelization and (partial) integration of existing and reliable computational/simulation codes, by using the newest hardware and software technologies. This should reduce both the computational demands and the time needed to the full R&D process, due to (at least partial) use of computational methodologies that are optimized for the different physical processes involved and to the removal of the need of a validation phase, which has been carried on along decades for each of the codes. CCR Workshop – May 22 -26 2017 - LNGS

Codes to be used - Transport codes: mainly MCNP 6 (Open. MP, MPI), but we project to use also GEANT 4, Tripoli, Fluka and Serpent 2 (integrates thermohydraulics, at least partially) - Typical run times for realistic configurations on single CPU ∼ month(s) - Open. FOAM v. 1612+ (MPI) for 3 D thermohydraulics modelization - Typical run times for realistic configurations on single CPU: steady state ∼ day(s), transients ∼ month(s) - Need to iterate transport and thermohydraulics until convergence - Different solvers and calculation meshes - Use of validated codes for industrial design - In-house developed code (Open. MP+MPI), dedicated to quench propagation in superconducting devices (see later on) - Typical run times for realistic configurations on single CPU: days or more CCR Workshop – May 22 -26 2017 - LNGS

KNL package Omni. Path available on some models only CCR Workshop – May 22 -26 2017 - LNGS

KNL internal mesh CCR Workshop – May 22 -26 2017 - LNGS

KNL tile CCR Workshop – May 22 -26 2017 - LNGS

KNL MCDRAM CCR Workshop – May 22 -26 2017 - LNGS

KNL clustering modes CCR Workshop – May 22 -26 2017 - LNGS

KNL clustering modes CCR Workshop – May 22 -26 2017 - LNGS

KNL clustering modes CCR Workshop – May 22 -26 2017 - LNGS

MKL dgemm() @ KNL e Xeon CCR Workshop – May 22 -26 2017 - LNGS

MKL dgemv() @ KNL e Xeon CCR Workshop – May 22 -26 2017 - LNGS

MKL dcsrgemv() @ KNL e Xeon CCR Workshop – May 22 -26 2017 - LNGS

MKL @ KNL: hyperthreading Peak performance with #threads = #cores, no improvement with hyperthreading. Possible reasons: BLAS level 3 (CPU-bound): code so well optimized that one thread is enough to saturate a core BLAS level 2, and sparse (memory-bound): 64 threads are enough to saturate DRAM bandwidth But had no time to investigate this (do we need to? ) CCR Workshop – May 22 -26 2017 - LNGS

MKL @ KNL: notes quad/cache and a 2 a/cache look like best ones, overall. a 2 a/flat-mcdram good too, but limited to small jobs (16 GB max), so not quite practical weird performance of BLAS level 2 and sparse on snc 4/cache; maybe because such a NUMA-like configuration requires suitable data placement in memory Sparse matrix * vector: MKL shows a bare 1. 3 speedup over a handcrafted one written in C (CSR format, Intel compiler) when size is large (>100 M) CCR Workshop – May 22 -26 2017 - LNGS

superconducting magnet An electromagnet made of windings of a cable containing a superconducting wire inside. Zero resistance: huge current (1000 -- 10000 Ampere) can flow indefinitely, no power supply during service, no heat dissipation. Intense magnetic field (orders of 10 Tesla), not feasible with ordinary conductors. Cooling system (liquid He) necessary for keeping the superconducting state. CCR Workshop – May 22 -26 2017 - LNGS

the cable CCR Workshop – May 22 -26 2017 - LNGS

applications: magnetic resonance CCR Workshop – May 22 -26 2017 - LNGS

applications: nuclear physics CCR Workshop – May 22 -26 2017 - LNGS

quench Loss of superconductivity in a region of the wire. Various causes: defective wire, mechanical stress, electric malfunction, cooling failure, nuclear particles swarm. More likely during the initial charging of the magnet, but also possible during normal operation. The resistive region dissipates energy as heat. Heat expands to near regions, triggering further losses of superconductivity. Energy dissipation increases, more heat is generated. . . CCR Workshop – May 22 -26 2017 - LNGS

quench CCR Workshop – May 22 -26 2017 - LNGS

quench Superconducting magnets are expensive toys. . . safety devices must detect and absorb the quench before a permanent damage occurs. The correct dimensioning of the safety devices requires a model of quench behaviour in space/time. Possible use of a simulator. CCR Workshop – May 22 -26 2017 - LNGS

quench simulators Some quench simulators are available, but simplified model of the cable (single material) source code not available; impossible to customize slow execution; acceptable run time achieved at expenses of accuracy, rather than HPC techniques. So there is room for improvement, especially with parallel computing! CCR Workshop – May 22 -26 2017 - LNGS

QUEPS: QUEnch Parallel Simulator A new quench simulator from scratch. work in progress. Start simple but with a look ahead: born as a parallel program: C++, MPI+OMP multiple-material cable solenoids only, no iron (for now) cooling: adiabatic, convection discretization: finite volume solver: Bi. CGSTAB + preconditioner CCR Workshop – May 22 -26 2017 - LNGS

QUEPS @ KNL and Xeon Test with a solenoid from literature 30 layers, 24 turns per layer cable is thin, Nb. Ti + Cu + outer insulation adiabatic simulated time: 1. 2 sec (quench inception at t=0) mesh: 3. 8 millions cells tests on a single KNL node: Open. MP, MPI, hybrid comparison with a dual Xeon E 5 2650 2 x 8 cores @ 2. 6 GHz CCR Workshop – May 22 -26 2017 - LNGS

QUEPS @ KNL and Xeon: MPI MCDRAM configured in cache mode execution times in minutes 64 processes 128 processes 256 processes a 2 a 125 81 86 quad 134 80. 5 85 snc 4 125. 5 78. 5 82 16 processes 32 processes (hyperthreading) 59. 5 77 dual Xeon CCR Workshop – May 22 -26 2017 - LNGS

QUEPS @ KNL and Xeon: Open. MP MCDRAM configured in cache mode execution times in minutes 64 threads 128 threads 256 threads a 2 a 55. 5 39 35. 5 quad 53 39 34 snc 4 58 39 32 16 threads 32 threads (hyperthreading) 65 65 dual Xeon CCR Workshop – May 22 -26 2017 - LNGS

Open. FOAM tests Open. FOAM (MPI) single node tests - ico. Foam CCR Workshop – May 22 -26 2017 - LNGS

Open. FOAM tests Open. FOAM (MPI) single node tests - pimple. Foam [transient, incompressible, LES]_4 Mcells CCR Workshop – May 22 -26 2017 - LNGS

MPCN tests MCNP (MPI+OMP) single node tests only OMP only MPI CCR Workshop – May 22 -26 2017 - LNGS

MPCN tests MCNP (MPI+OMP) 2 nodes tests With 2 x-hypertreading Speed-UP about 1. 54 Better performance with optimal ratio «MPI_processes / OMP_threads» CCR Workshop – May 22 -26 2017 - LNGS