The Pencil Code multipurpose and multiuser maintained Axel

The Pencil Code: multi-purpose and multi-user maintained Axel Brandenburg (Nordita, Copenhagen) Axel Brandenburg: The Pencil-Code

Overview • • • Pencil formulation (advantages, headaches) Structure of code, cvs maintainence High-order schemes, tests Peculiarities on big linux clusters Online data processing Axel Brandenburg: The Pencil-Code 2

Pencil Code • • Started in Sept. 2001 with Wolfgang Dobler High order (6 th order in space, 3 rd order in time) Cache & memory efficient MPI, can run Pacx. MPI (across countries!) Maintained/developed by many people (CVS!) Automatic validation (over night or any time) Max resolution so far 10243 Axel Brandenburg: The Pencil-Code 3

Range of applications • Isotropic turbulence – MHD (Nils), passive scalar, cosmic rays • Stratified layers – Convection, radiative transport (Tobi) • Shearing box • MRI (Nils), Planetesimals (Anders), Interstellar (Tony) • Sphere embedded in box – Fully convective stars (Dobler), geodynamo (Mc. Millan) Axel Brandenburg: The Pencil-Code 4

Pencil formulation • In CRAY days: worked with full chunks f(nx, ny, nz, nvar) – Now, on SGI, nearly 100% cache misses • • Instead work with f(nx, nvar), i. e. one nx-pencil No cache misses, negligible work space, just 2 N Communication before sub-timestep Then evaluate all derivatives, e. g. call curl(f, i. A, B) – Vector potential A=f(: , : , i. Ax: i. Az), B=B(nx, 3) Axel Brandenburg: The Pencil-Code 5

A few headaches • All operations must be combined – Curl(curl), max 5(smooth(divu)) must be in one go • rms and max values for monitoring – call max_name(b 2, i_bmax, lsqrt=. true. ) – call sum_name(b 2, i_brms, lsqrt=. true. ) • Similar routines for toroidal average, etc • Online analysis (spectra, slices, vectors) Axel Brandenburg: The Pencil-Code 6

CVS maintained • pserver (password protected) – Public (check-out only), private (ci/co, 20 people) • • Set of 10 test problems Nightly auto-test (different machines, web) Before check-in: run auto-test yourself Mpi and nompi dummy module for single processor machine (or use lammpi on laptops) Axel Brandenburg: The Pencil-Code 7

Switch modules • • • magnetic or nomagnetic (e. g. just hydro) hydro or nohydro (e. g. kinematic dynamo) density or nodensity (burgulence) entropy or noentropy (e. g. isothermal) radiation or noradiation (see Tobi’s talk) dustvelocity or nodustvelocity (planetesimals) Axel Brandenburg: The Pencil-Code 8

Features, problems • Namelist (can freely introduce new params) • Upgrades forgotten on no-modules (auto-test) • SGI namelist problem (see pencil FAQs) Axel Brandenburg: The Pencil-Code 9

Pencil Code check ins 10

High-order schemes • Alternative to spectral or compact schemes – Efficiently parallelized – No transpose necessary • 6 th order central differences in space • Non-conservative scheme – Allows use of logarithmic density and entropy – Copes well with strong stratification and temperature contrasts Axel Brandenburg: The Pencil-Code 11

High-order spatial schemes Main advantage: low phase errors Axel Brandenburg: The Pencil-Code 12

Wavenumber characteristics Axel Brandenburg: The Pencil-Code 13

Higher order – less viscosity Axel Brandenburg: The Pencil-Code 14

Less viscosity – also in shocks Axel Brandenburg: The Pencil-Code 15

High-order temporal schemes Main advantage: low amplitude errors 2 N-RK 3 scheme (Williamson 1980) 2 nd order 3 rd order 1 st order Axel Brandenburg: The Pencil-Code 16

Shock tube test Axel Brandenburg: The Pencil-Code 17

Hydromagnetic turbulence and subgrid scale models? • Want to shorten diffusive subrange – Waste of resources • Want to prolong inertial range – Focus of essential physics • Reasons to be worried about hyperviscosity – Shallower spectra – Wrong amplitudes of resulting large scale fields Axel Brandenburg: The Pencil-Code 18

Simulations at 512^3 Biskamp & Muller (2000) Normal diffusivity With hyperdiffusivity Axel Brandenburg: The Pencil-Code 19

256 processor run at 10243 Axel Brandenburg: The Pencil-Code 20

MHD equation Magn. Vector potential Induction Equation: Momentum and Continuity eqns Axel Brandenburg: The Pencil-Code 21

Vector potential • B=curl. A, advantage: div. B=0 • J=curl. B=curl(curl. A) =curl 2 A • Not a disadvantage: consider Alfven waves B-formulation A-formulation 2 nd der once is better than 1 st der twice! Axel Brandenburg: The Pencil-Code 22

Wallclock time versus processor # Axel Brandenburg: The Pencil-Code 23

Sensitivity to layout on Linux clusters yprox x zproc 4 x 32 1 (speed) 8 x 16 3 times slower 16 x 8 17 times slower Axel Brandenburg: The Pencil-Code Gigabit uplink 100 Mbit link only 24 procs per hub 24

Why this sensitivity to layout? 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 All processors need to communicate with processors outside to group of 24 25

Use exactly 4 columns Only 2 x 4 = 8 processors need to communicate outside the group of 24 optimal use of speed ratio between 100 Mb ethernet switch and 1 Gb uplink 0 4 8 12 16 20 0 4 8 1 5 9 13 17 21 1 5 9 2 6 10 14 18 22 2 6 10 3 7 11 15 19 23 3 7 11 26

Fragmentation over many switches Axel Brandenburg: The Pencil-Code 27

Animation of uz Axel Brandenburg: The Pencil-Code 28

Animation of B vectors Axel Brandenburg: The Pencil-Code 29

Animation of B vectors Axel Brandenburg: The Pencil-Code 30

Animation of energy spectra Axel Brandenburg: The Pencil-Code 31

Saturation behavior explained by magnetic helicity conservation Steady state, closed box Small scale and large scale current helicity in balance Axel Brandenburg: The Pencil-Code 32

With hyperdiffusivity for ordinary hyperdiffusion Axel Brandenburg: The Pencil-Code 33

Conclusions • Subgrid scale modeling can be unsafe (some problems) – shallower spectra, longer time scales, different saturation amplitudes • High order schemes – Low phase and amplitude errors – Need less viscosity • • 100 MB link close to bandwidth limit Comparable to Origin 2 x faster with GB switch 100 MB switches with GB uplink optimal Axel Brandenburg: The Pencil-Code 34