Titanium A Java Dialect for High Performance Computing

Titanium: A Java Dialect for High Performance Computing Katherine Yelick U. C. Berkeley and LBNL

Motivation: Target Problems u Many modeling problems in astrophysics, biology, material science, and other areas require 0 Enormous range of spatial and temporal scales u To solve interesting problems, one needs: 0 Adaptive methods 0 Large scale parallel machines u Titanium is designed for 0 Structured grids 0 Locally-structured grids (AMR) 0 Unstructured grids (in progress) SC 2003 Source: J. Bell, LBNL 11/16/03 S 09: Programming with the Partitioned Global Address Space Model 2

Titanium Background u Based on Java, a cleaner C++ 0 Classes, automatic memory management, etc. 0 Compiled to C and then machine code, no JVM u Same parallelism model at UPC and CAF 0 SPMD parallelism 0 Dynamic Java threads are not supported u Optimizing compiler 0 Analyzes global synchronization 0 Optimizes pointers, communication, memory SC 2003 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 3

Summary of Features Added to Java u Multidimensional arrays: iterators, subarrays, copying u Immutable (“value”) classes u Templates u Operator overloading u Scalable SPMD parallelism replaces threads u Global address space with local/global reference distinction u Checked global synchronization u Zone-based memory management (regions) u Libraries for collective communication, distributed arrays, bulk I/O, performance profiling SC 2003 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 4

Outline u Titanium Execution Model 0 SPMD 0 Global Synchronization 0 Single u Titanium Memory Model u Support for Serial Programming u Performance and Applications u Compiler/Language Status u Compiler Optimizations & Future work SC 2003 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 5

SPMD Execution Model u Titanium has the same execution model as UPC and CAF u Basic Java programs may be run as Titanium programs, but all processors do all the work. u E. g. , parallel hello world class Hello. World { public static void main (String [] argv) { System. out. println(“Hello from proc “ + Ti. this. Proc() + “ out of “ + Ti. num. Procs()); } } u Global synchronization done using Ti. barrier() SC 2003 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 6

Barriers and Single u Common source of bugs is barriers or other collective operations inside branches or loops barrier, broadcast, reduction, exchange u A “single” method is one called by all procs public single static void all. Step(. . . ) u A “single” variable has same value on all procs int single timestep = 0; u Single annotation on methods is optional, but useful in understanding compiler messages u Compiler proves that all processors call barriers together SC 2003 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 7

Explicit Communication: Broadcast u Broadcast is a one-to-all communication broadcast <value> from <processor> u For example: int count = 0; int all. Count = 0; if (Ti. this. Proc() == 0) count = compute. Count(); all. Count = broadcast count from 0; u The processor number in the broadcast must be single; all constants are single. 0 All processors must agree on the broadcast source. u The all. Count variable could be declared single. 0 All will have the same value after the broadcast. SC 2003 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 8

Example of Data Input u Reading from keyboard, uses Java exceptions int my. Count = 0; int single all. Count = 0; if (Ti. this. Proc() == 0) try { Data. Input. Stream kb = new Data. Input. Stream(System. in); my. Count = Integer. value. Of(kb. read. Line()). int. Value(); } catch (Exception e) { System. err. println("Illegal Input"); } all. Count = broadcast my. Count from 0; SC 2003 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 9

More on Single u Global synchronization needs to be controlled if (this processor owns some data) { compute on it barrier } u Hence the use of “single” variables in Titanium u If a conditional or loop block contains a barrier, all processors must execute it 0 conditions must contain only single variables u Compiler analysis statically enforces freedom from deadlocks due to barrier and other collectives being called non-collectively "Barrier Inference" [Gay & Aiken] SC 2003 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 10

Single Variable Example u Barriers and single in N-body Simulation class Particle. Sim { public static void main (String [] argv) { int single all. Timestep = 0; int single all. End. Time = 100; for (; all. Timestep < all. End. Time; all. Timestep++){ read remote particles, compute forces on mine Ti. barrier(); write to my particles using new forces Ti. barrier(); } } } u Single methods inferred by the compiler SC 2003 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 11

Outline u Titanium Execution Model u Titanium Memory Model 0 Global and Local References 0 Exchange: Building Distributed Data Structures 0 Region-Based Memory Management u Support for Serial Programming u Performance and Applications u Compiler/Language Status u Compiler Optimizations & Future work SC 2003 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 12

Global Address Space u Globally shared address space is partitioned u References (pointers) are either local or global Global address space (meaning possibly remote) x: 1 y: 2 x: 5 y: 6 l: l: g: g: p 0 SC 2003 p 1 x: 7 y: 8 Object heaps are shared Program stacks are private pn S 09: Programming with the Partitioned Global Address Space Model 11/16/03 13

Use of Global / Local u As seen, global references (pointers) may point to remote locations 0 easy to port shared-memory programs u Global pointers are more expensive than local 0 True even when data is on the same processor 0 Costs of global: u space (processor number + memory address) u dereference time (check to see if local) u May declare references as local 0 Compiler will automatically infer local when possible SC 2003 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 14

Global Address Space LEFT OFF HERE Process 0 u Processes allocate locally u References can be passed to Process 1 other processes gv gv class C { public int val; . . . } C gv; // global pointer C local lv; // local pointer if (Ti. this. Proc() == 0) { lv = new C(); } gv = broadcast lv from 0; //data race gv. val = Ti. this. Proc(); lv. val = gv. val; lv lv HEAP 0 val: 0 int winner = gv. val SC 2003 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 15

Shared/Private vs Global/Local u Titanium’s global address space is based on pointers rather than shared variables u There is no distinction between a private and shared heap for storing objects 0 Although recent compiler analysis infers this distinction and uses it for performing optimizations [Liblit et. al 2003] u All objects may be referenced by global pointers or by local ones u There is no direct support for distributed arrays 0 Irregular problems do not map easily to distributed arrays, since each processor will own a set of objects (sub-grids) 0 For regular problems, Titanium uses pointer dereference instead of index calculation 0 Important to have local “views” of data structures SC 2003 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 16

Aside on Titanium Arrays u Titanium adds its own multidimensional array class for performance u Distributed data structures are built using a 1 D Titanium array u Slightly different syntax, since Java arrays still exist in Titanium, e. g. : int [1 d] arr; arr = new int [1: 100]; arr[1] = 4*arr[1]; u Will discuss these more later… SC 2003 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 17

Explicit Communication: Exchange u To create shared data structures 0 each processor builds its own piece 0 pieces are exchanged (for object, just exchange pointers) u Exchange primitive in Titanium int [1 d] single all. Data; all. Data = new int [0: Ti. num. Procs()-1]; all. Data. exchange(Ti. this. Proc()*2); u E. g. , on 4 procs, each will have copy of all. Data: 0 SC 2003 2 4 6 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 18

Building Distributed Structures u Distributed structures are built with exchange: class Boxed { public Boxed (int j) { val = j; } public int val; } Object [1 d] single all. Data; all. Data = new Object [0: Ti. num. Procs()-1]; S 09: Programming with the Partitioned Global Address Space Model all. Data. exchange(new Boxed(Ti. this. Proc()); SC 2003 11/16/03 19

Distributed Data Structures u Building distributed arrays: Particle [1 d] single [1 d] all. Particle = new Particle [0: Ti. num. Procs-1][1 d]; Particle [1 d] my. Particle = new Particle [0: my. Particle. Count-1]; all. Particle. exchange(my. Particle); All to all broadcast u Now each processor has array of pointers, one to each processor’s chunk of particles P 0 SC 2003 P 1 P 2 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 20

Region-Based Memory Management u An advantage of Java over C/C++ is: 0 Automatic memory management u But unfortunately, garbage collection: 0 Has a reputation of slowing serial code 0 Is hard to implement and scale in a distributed environment u Titanium takes the following approach: 0 Memory management is safe – cannot deallocate live data 0 Garbage collection is used by default (most platforms) 0 Higher performance is possible using region-based S 09: Programming with the Partitioned Global Address Space Model 11/16/03 management SC 2003 explicit memory 21

Region-Based Memory Management u Need to organize data structures u Allocate set of objects (safely) u Delete them with a single explicit call (fast) 0 David Gay's Ph. D. thesis Private. Region r = new Private. Region(); for (int j = 0; j < 10; j++) { int[] x = new ( r ) int[j + 1]; work(j, x); } try { r. delete(); } catch (Region. In. Use oops) { System. out. println(“failed to delete”); } SC 2003 } S 09: Programming with the Partitioned Global Address Space Model 11/16/03 22

Outline u Titanium Execution Model u Titanium Memory Model u Support for Serial Programming 0 Immutables 0 Operator overloading 0 Multidimensional arrays 0 Templates u Performance and Applications u Compiler/Language Status u Compiler Optimizations & Future work S 09: Programming with the Partitioned Global Address Space Model SC 2003 11/16/03 23

Java Objects u Primitive scalar types: boolean, double, int, etc. 0 implementations will store these on the program stack 0 access is fast -- comparable to other languages u Objects: user-defined and standard library 0 always allocated dynamically 0 passed by pointer value (object sharing) into functions 0 has level of indirection (pointer to) implicit 0 simple model, but inefficient for small objects 2. 6 r: 7. 1 3 true SC 2003 i: 4. 3 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 24

Java Object Example class Complex { private double real; private double imag; public Complex(double r, double i) { real = r; imag = i; } public Complex add(Complex c) { return new Complex(c. real + real, c. imag + imag); public double get. Real { return real; } public double get. Imag { return imag; } } Complex c = new Complex(7. 1, 4. 3); c = c. add(c); class Vis. Complex extends Complex {. . . } SC 2003 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 25

Immutable Classes in Titanium u For small objects, would sometimes prefer 0 to avoid level of indirection and allocation overhead 0 pass by value (copying of entire object) 0 especially when immutable -- fields never modified u extends the idea of primitive values to user-defined datatypes u Titanium introduces immutable classes 0 all fields are implicitly final (constant) 0 cannot inherit from or be inherited by other classes 0 needs to have 0 -argument constructor u Example uses: 0 Complex numbers, xyz components of a field vector at a grid cell (velocity, force) u Note: considering lang. withextension to. Address allow S 09: Programming the Partitioned Global Spacemutation Model SC 2003 11/16/03 26

Example of Immutable Classes 0 The immutable complex class nearly the same immutable class Complex { Complex () {real=0; imag=0; } new keyword. . . } Zero-argument constructor required Rest unchanged. No assignment to fields outside of constructors. 0 Use of immutable complex values Complex c 1 = new Complex(7. 1, 4. 3); Complex c 2 = new Complex(2. 5, 9. 0); c 1 = c 1. add(c 2); 0 Addresses performance and programmability u Similar to C structs in terms of performance u Allows efficient support of complex types through a general language mechanism SC 2003 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 27

Operator Overloading • For convenience, Titanium provides operator overloading • important for readability in scientific code • Very similar to operator overloading in C++ • Must be used judiciously class Complex { private double real; private double imag; public Complex op+(Complex c) { return new Complex(c. real + real, c. imag + imag); } Complex c 1 = new Complex(7. 1, 4. 3); Complex c 2 = new Complex(5. 4, 3. 9); Complex c 3 = c 1 + S 09: c 2; Programming with the Partitioned Global Address Space Model SC 2003 11/16/03 28

Arrays in Java u Arrays in Java are objects u Only 1 D arrays are directly supported 2 d array u Multidimensional arrays are arrays of arrays u General, but slow - due to memory layout, difficulty of compiler analysis, and bounds checking u Subarrays are important in AMR (e. g. , interior of a grid) 0 Even C and C++ don’t support these well 0 Hand-coding (array libraries) can confuse optimizer SC 2003 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 29

Multidimensional Arrays in Titanium u New multidimensional array added 0 One array may be a subarray of another e. g. , a is interior of b, or a is all even elements of b u can easily refer to rows, columns, slabs or boundary regions as sub-arrays of a larger array u 0 Indexed by Points (tuples of ints) 0 Constructed over a rectangular set of Points, called Rectangular Domains (Rect. Domains) 0 Points, Domains and Rect. Domains are built-in immutable classes, with handy literal syntax u Expressive, flexible and fast u Support for AMR and other grid computations 0 domain operations: intersection, shrink, border 0 bounds-checking can be disabled after debugging phase SC 2003 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 30

Unordered Iteration u Memory hierarchy optimizations are essential u Compilers can sometimes do these, but hard in general u Titanium adds explicitly unordered iteration over domains 0 Helps the compiler with loop & dependency analysis 0 Simplifies bounds-checking 0 Also avoids some indexing details - more concise foreach (p in r) { … A[p] … } 0 p is a Point (tuple of ints) that can be used to index arrays 0 r is a Rect. Domain or Domain u Additional operations on domains to subset and xform u Note: foreach is not a parallelism construct SC 2003 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 31

Point, Rect. Domain, Arrays in General u Points specified by a tuple of ints Point<2> lb = [1, 1]; Point<2> ub = [10, 20]; u Rect. Domains given by 3 points: 0 lower bound, upper bound (and optional stride) Rect. Domain<2> r = [lb : ub]; u Array declared by num dimensions and type double [2 d] a; u Array created by passing Rect. Domain a = new double [r]; SC 2003 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 32

Simple Array Example u Matrix sum in Titanium Point<2> lb = [1, 1]; Point<2> ub = [10, 20]; Rect. Domain<2> r = [lb: ub]; No array allocation here Syntactic sugar double [2 d] a = new double [r]; double [2 d] b = new double [1: 10, 1: 20]; double [2 d] c = new double [lb: ub: [1, 1]]; for (int i = 1; i <= 10; i++) for (int j = 1; j <= 20; j++) c[i, j] = a[i, j] + b[i, j]; Optional stride Equivalent loops foreach(p in c. domain()) { c[p] = a[p] + b[p]; } SC 2003 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 33

Naïve Mat. Mul with Titanium Arrays public static void mat. Mul(double [2 d] a, double [2 d] b, double [2 d] c) { int n = c. domain(). max()[1]; // assumes square for (int i = 0; i < n; i++) { for (int j = 0; j < n; j++) { for (int k = 0; k < n; k++) { c[i, j] += a[i, k] * b[k, j]; } } SC 2003 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 34

Better Mat. Mul with Titanium Arrays public static void mat. Mul(double [2 d] a, double [2 d] b, double [2 d] c) { foreach (ij in c. domain()) { double [1 d] a. Rowi = a. slice(1, ij[1]); double [1 d] b. Colj = b. slice(2, ij[2]); foreach (k in a. Rowi. domain()) { c[ij] += a. Rowi[k] * b. Colj[k]; } } } Current performance: comparable to 3 nested loops in C Recent upgrades: automatic blocking for memory hierarchy (Geoff S 09: Programming with the Partitioned Global Address Space Model 11/16/03 SC 2003 Pike’s Ph. D thesis) 35

Example: Domain u Domains in general are not rectangular r u Built using set operations 0 union, + 0 intersection, * 0 difference, - (0, 0) u Example is red-black algorithm Point<2> lb = Point<2> ub = Rect. Domain<2>. . . Domain<2> red foreach (p in. . . } SC 2003 [0, 0]; [6, 4]; r = [lb : ub : [2, 2]]; (6, 4) r + [1, 1] (7, 5) (1, 1) red = r + (r + [1, 1]); red) { (7, 5) (0, 0) S 09: Programming with the Partitioned Global Address Space Model 11/16/03 36

Example using Domains and foreach u Gauss-Seidel red-black computation in multigrid void gsrb() { boundary (phi); for (Domain<2> d = red; d != null; d = (d = = red ? black : null)) { foreach (q in d) unordered iteration res[q] = ((phi[n(q)] + phi[s(q)] + phi[e(q)] + phi[w(q)])*4 + (phi[ne(q) + phi[nw(q)] + phi[se(q)] + phi[sw(q)]) 20. 0*phi[q] - k*rhs[q]) * 0. 05; foreach (q in d) phi[q] += res[q]; } } SC 2003 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 37

Example: A Distributed Data Structure u Data can be accessed across processor boundaries Proc 0 Proc 1 local_grids all_grids SC 2003 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 38

Example: Setting Boundary Conditions foreach (l in local_grids. domain()) { foreach (a in all_grids. domain()) { local_grids[l]. copy(all_grids[a]); } } SC 2003 "ghost" cells S 09: Programming with the Partitioned Global Address Space Model 11/16/03 39

Templates u Many applications use containers: 0 E. g. , arrays parameterized by dimensions, element types 0 Java supports this kind of parameterization through inheritance u Can only put Object types into containers u Inefficient when used extensively u Titanium provides a template mechanism closer to that of C++ 0 E. g. Can be instantiated with "double" or immutable class 0 Used to build a distributed array package 0 Hides the details of exchange, indirection within Programming with the Partitioned Global Address Space Model 11/16/03 SC 2003 the data. S 09: structure, etc. 40

Example of Templates template <class Element> class Stack {. . . public Element pop() {. . . } public void push( Element arrival ) {. . . } } template Stack<int> list = new template Stack<int>(); list. push( 1 ); Not an object int x = list. pop(); Strongly typed, No dynamic cast u Addresses programmability and performance SC 2003 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 41

Using Templates: Distributed Arrays template <class T, int single arity> public class Dist. Array { Rect. Domain <arity> single rd; T [arity d] sub. Matrices; Rect. Domain <arity> [arity d] single sub. Domains; . . . /* Sets the element at p to value */ public void set (Point <arity> p, T value) { get. Homing. Sub. Matrix (p) [p] = value; } } template Dist. Array <double, 2> single A = new template Dist. Array<double, 2> ( [[0, 0]: [a. Height, a. Width]] ); S 09: Programming with the Partitioned Global Address Space Model SC 2003 11/16/03 42

Outline u Titanium Execution Model u Titanium Memory Model u Support for Serial Programming u Performance and Applications 0 Serial Performance on pure Java (Sci. Mark) 0 Parallel Applications 0 Compiler status & usability results u Compiler/Language Status u Compiler Optimizations & Future work SC 2003 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 43

Sci. Mark Benchmark u Numerical benchmark for Java, C/C++ 0 purely sequential u Five kernels: 0 FFT (complex, 1 D) 0 Successive Over-Relaxation (SOR) 0 Monte Carlo integration (MC) 0 Sparse matrix multiply 0 dense LU factorization u Results are reported in MFlops 0 We ran them through Titanium as 100% pure Java with no extensions u Download and run on your machine from: 0 http: //math. nist. gov/scimark 2 S 09: Pozo, Programming the Partitioned Global Address Space Model Roldan NIST, with http: //math. nist. gov/~Rpozo SC 2003 0 C and Java sources are provided 11/16/03 44

Java Compiled by Titanium Compiler –Sun JDK 1. 4. 1_01 (Hot. Spot(TM) Client VM) for Linux –IBM J 2 SE 1. 4. 0 (Classic VM cxia 32140 -20020917 a, jitc JIT) for 32 -bit Linux –Titaniumc v 2. 87 for Linux, gcc 3. 2 as backend compiler -O 3. no bounds check –gcc 3. 2, -O 3 (ANSI-C version of the Sci. Mark 2 benchmark) SC 2003 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 45

Java Compiled by Titanium Compiler –Sun JDK 1. 4. 1_01 (Hot. Spot(TM) Client VM) for Linux –IBM J 2 SE 1. 4. 0 (Classic VM cxia 32140 -20020917 a, jitc JIT) for 32 -bit Linux –Titaniumc v 2. 87 for Linux, gcc 3. 2 as backend compiler -O 3. no bounds check –gcc 3. 2, -O 3 (ANSI-C version of the Sci. Mark 2 benchmark) SC 2003 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 46

Sequential Performance of Java u State of the art JVM's 0 often very competitive with C performance 0 within 25% in worst case, sometimes better than C u Titanium compiling pure Java 0 On par with best JVM's and C performance 0 This is without leveraging Titanium's lang. extensions u We can try to do even better using a traditional compilation model 0 Berkeley Titanium compiler: u Compiles SC 2003 Java + extensions into C u No JVM, no dynamic class loading, whole program compilation u Do not currently optimize Java array accesses (prototype) S 09: Programming with the Partitioned Global Address Space Model 11/16/03 47

Language Support for Performance u Multidimensional arrays 0 Contiguous storage 0 Support for sub-array operations without copying u Support for small objects 0 E. g. , complex numbers 0 Called “immutables” in Titanium 0 Sometimes called “value” classes u Unordered loop construct 0 Programmer specifies loop iterations independent 0 Eliminates need for dependence analysis (short term solution? ) Same idea used by vectorizing compilers. SC 2003 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 48

Array Performance Issues u Array representation is fast, but access methods can be slow, e. g. , bounds checking, strides u Compiler optimizes these 0 common subexpression elimination 0 eliminate (or hoist) bounds checking 0 strength reduce: e. g. , naïve code has 1 divide per dimension for each array access u Currently +/- 20% of C/Fortran for large loops u Future: small loop and cache tiling optimizations SC 2003 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 49

Applications in Titanium u Benchmarks and Kernels 0 Fluid solvers with Adaptive Mesh Refinement (AMR) 0 Scalable Poisson solver for infinite domains 0 Conjugate Gradient 03 D Multigrid 0 Unstructured mesh kernel: EM 3 D 0 Dense linear algebra: LU, Mat. Mul 0 Tree-structured n-body code 0 Finite element benchmark 0 Sci. Mark serial benchmarks u Larger applications SC 2003 0 Heart and Cochlea simulation 0 Genetics: micro-array selection 0 Ocean modeling with AMR (in progress) S 09: Programming with the Partitioned Global Address Space Model 11/16/03 50

NAS MG in Titanium u Preliminary Performance for MG code on IBM SP 0 Speedups are nearly identical 0 About 25% serial performance difference SC 2003 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 51

Heart Simulation: Immersed Boundary Method u Problem: compute blood flow in the heart 0 Modeled as an elastic structure in an incompressible fluid. u The “immersed boundary method” [Peskin and Mc. Queen]. u 20 years of development in model 0 Many other applications: blood clotting, inner ear, paper making, embryo growth, and more u Can be used for design of prosthetics 0 Artificial heart valves 0 Cochlear implants SC 2003 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 52

Fluid Flow in Biological Systems • Immersed Boundary Method • Material (e. g. , heart muscles, cochlea structure) modeled by grid of material points • Fluid space modeled by a regular lattice • Irregular material points need to interact with regular fluid lattice • Trade-off between load balancing of fibers and minimizing communication • Memory and communication intensive • Includes a Navier-Stokes solver and a 3 -D FFT solver • Heart simulation is complete, Cochlea simulation is close to done • First time that immersed boundary simulation has been done on distributed-memory machines • Working on a Ti library for doing other immersed boundary simulations SC 2003 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 53

MOOSE Application u Problem: Genome Microarray construction 0 Used for genetic experiments 0 Possible medical applications long-term u Microarray Optimal Oligo Selection Engine (MOOSE) 0 A parallel engine for selecting the best oligonucleotide sequences for genetic microarray testing from a sequenced genome (based on uniqueness and various structural and chemical properties) 0 First parallel implementation for solving this problem 0 Uses dynamic load balancing within Titanium 0 Significant S 09: memory and I/O demands for larger Programming with the Partitioned Global Address Space Model 11/16/03 SC 2003 genomes 54

Scalable Parallel Poisson Solver u MLC for Finite-Differences by Balls and Colella u Poisson equation with infinite boundaries 0 arise in astrophysics, some biological systems, etc. u Method is scalable 0 Low communication (<5%) u Performance on 0 SP 2 (shown) and T 3 E 0 scaled speedups 0 nearly ideal (flat) u Currently 2 D and non- adaptive SC 2003 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 55

1. 31 x 10 -9 Error on High-Wavenumber Problem u Charge is 0 1 charge of 0 concentric waves 0 2 star-shaped charges. u Largest error is where the charge is changing rapidly. Note: 0 discretization error 0 faint decomposition error SC 2003 S 09: Programming with the Partitioned Global Address Space Model -6. 47 x 10 -9 u Run on 16 procs 11/16/03 56

AMR Poisson u Poisson Solver [Semenzato, Pike, Colella] 03 D AMR 0 finite domain 0 variable Level 2 Level 1 coefficients 0 multigrid across levels. Level 0 u Performance of Titanium implementation 0 Sequential multigrid performance +/- 20% of Fortran 0 On fixed, well-balanced problem of 8 patches, each 723 S 09: Programming with 5. 5 the Partitioned Address Space Model 0 parallel speedups of on 8 Global processors SC 2003 11/16/03 57

AMR Gas Dynamics u Hyperbolic Solver [Mc. Corquodale and Colella] 0 Implementation of Berger-Colella algorithm 0 Mesh generation algorithm included u 2 D Example (3 D supported) 0 Mach-10 shock on solid surface at oblique angle u Future: Self-gravitating gas dynamics package SC 2003 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 58

Outline u Titanium Execution Model u Titanium Memory Model u Support for Serial Programming u Performance and Applications u Compiler/Language Status u Compiler Optimizations & Future work SC 2003 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 59

Titanium Compiler Status • Titanium compiler runs on almost any machine – Requires a C compiler (and decent C++ to compile translator) – Pthreads for shared memory – Communication layer for distributed memory (or hybrid) • Recently moved to live on GASNet: shared with UPC • Obtained Myrinet, Quadrics, and improved LAPI implementation • Recent language extensions – Indexed array copy (scatter/gather style) – Non-blocking array copy under development • Compiler optimizations – Cache optimizations, for loop optimizations – Communication optimizations for overlap, pipelining, and scatter/gather under development SC 2003 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 60

Implementation Portability Status u Titanium has been tested on: 0 POSIX-compliant workstations & SMPs Automatic portability: 0 Clusters of uniprocessors or SMPs Titanium applications run 0 Cray T 3 E on all of these! 0 IBM SP Very important productivity 0 SGI Origin 2000 feature for debugging & development 0 Compaq Alpha. Server 0 MS Windows/GNU Cygwin 0 and others… u Supports many communication layers 0 High performance networking layers: u IBM/LAPI, Myrinet/GM, Quadrics/Elan, Cray/shmem, Infiniband (soon) 0 Portable communication layers: u MPI-1. 1, TCP/IP (UDP) http: //titanium. cs. berkeley. edu S 09: Programming with the Partitioned Global Address Space Model 11/16/03 61

Programmability u Heart simulation developed in ~1 year 0 Extended to support 2 D structures for Cochlea model in ~1 month u Preliminary code length measures 0 Simple torus model u Serial Fortran torus code is 17045 lines long (2/3 comments) u Parallel Titanium torus version is 3057 lines long. 0 Full heart model u Shared memory Fortran heart code is 8187 lines long u Parallel Titanium version is 4249 lines long. 0 Need to be analyzed more carefully, but not a significant overhead for distributed memory parallelism SC 2003 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 62

Robustness u Robustness is the primary motivation for language “safety” in Java 0 Type-safe, array bounds checked, auto memory management 0 Study on C++ vs. Java from Phipps at Spirus: C++ has 2 -3 x more bugs per line than Java u Java had 30 -200% more lines of code per minute u u Extended in Titanium 0 Checked synchronization avoids barrier/collective deadlocks 0 More abstract array indexing, retains bounds checking u No attempt to quantify benefit of safety for Titanium yet 0 Would like to measure speed of error detection (compile time, runtime exceptions, etc. ) 0 Anecdotal evidence suggests the language safety features are very useful in application debugging and development SC 2003 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 63

Calling Other Languages u We have built interfaces to 0 PETSc : scientific library for finite element applications 0 Metis: graph partitioning library 0 Ke. LP: scientific C++ library u Two issues with cross-language calls 0 accessing Titanium data structures (arrays) from C u possible because Titanium arrays have same format on inside 0 having a common message layer u Titanium SC 2003 is built on lightweight communication S 09: Programming with the Partitioned Global Address Space Model 11/16/03 64

Outline u Titanium Execution Model u Titanium Memory Model u Support for Serial Programming u Performance and Applications u Compiler/Language Status u Compiler Optimizations & Future work 0 Local pointer identification (LQI) 0 Communication optimizations 0 Feedback-directed search-based optimizations SC 2003 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 65

Local Pointer Analysis u Global pointer access is more expensive than local u Compiler analysis can frequently infer that a given global pointer always points locally 0 Replace global pointer with a local one 0 Local Qualification Inference (LQI) [Liblit] 0 Data structures must be well partitioned Same idea can be applied to UPC's pointer-to-shared… SC 2003 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 66

Communication Optimizations • Possible communication optimizations • Communication overlap, aggregation, caching • Effectiveness varies by machine • Generally pays to target low-level network API [Bell, Bonachea et al] at IPDPS'03 SC 2003 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 67

Split-C Experience: Latency Overlap u Titanium borrowed ideas from Split-C 0 global address space 0 SPMD parallelism u But, Split-C had explicit non-blocking accesses built in to tolerate network latency on remote read/write int *global p; x : = *p; /* get */ *p : = 3; /* put */ sync; /* wait for my puts/gets */ u Also one-way communication/* store */ *p : - x; all_store_sync; /* wait globally */ u Conclusion: useful, but complicated SC 2003 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 68

Titanium: Consistency Model u Titanium adopts the Java memory consistency model u Roughly: Access to shared variables that are not synchronized have undefined behavior u Use synchronization to control access to shared variables 0 barriers 0 synchronized methods and blocks u Open question: Can we leverage the relaxed consistency model to automate communication overlap optimizations? 0 difficulty of alias analysis is a significant problem SC 2003 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 69

Sources of Memory/Comm. Overlap u Would like compiler to introduce put/get/store u Hardware also reorders 0 out-of-order execution 0 write buffered with read by-pass 0 non-FIFO write buffers 0 weak memory models in general u Software already reorders too 0 register allocation 0 any code motion u System provides enforcement primitives 0 e. g. , memory fence, volatile, etc. 0 tend to be heavyweight and have unpredictable performance SC 2003 S 09: Programming with the Partitioned Global Address Space Model u Open question: Can the compiler hide all this? 11/16/03 70

Feedback-directed optimization u Use machines, not humans for architecture- specific tuning 0 Code generation + search-based selection u Can adapt to cache size, # registers, network buffering 0 Used in u Signal processing: FFTW, SPIRAL, UHFFT u Dense linear algebra: Atlas, PHi. PAC u Sparse linear algebra: Sparsity u Rectangular grid-based computations: Titanium compiler » Cache tiling optimizations - automated search for best tiling parameters for a given architecture SC 2003 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 71

Current Work & Future Plans u Unified communication layer with UPC: GASNet u Exploring communication overlap optimizations 0 Explicit (programmer-controlled) and automated 0 Optimize regular and irregular communication patterns u Analysis and refinement of cache optimizations 0 along with other sequential optimization improvements u Additional language support for unstructured grids 0 arrays over general domains, with multiple values per grid point u Continued work on existing and new applications http: //titanium. cs. berkeley. edu SC 2003 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 72

Titanium Group (Past and Present) u Susan Graham u Ben Liblit u Katherine Yelick u Peter Mc. Quorquodale (LBNL) u Paul Hilfinger u Sabrina Merchant u Phillip Colella (LBNL) u Carleton Miyamoto u Alex Aiken u Chang Sun Lin u Geoff Pike u Greg Balls u Luigi Semenzato (LBNL) u Andrew Begel u Armando Solar-Lezama u Dan Bonachea u Jimmy Su u Kaushik Datta u Tong Wen (LBNL) u David Gay u Siu Man Yau u Ed Givelberg u and many undergraduate researchers http: //titanium. cs. berkeley. edu u Arvind Krishnamurthy SC 2003 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 73

SPMD Model u All processors start together and execute same code, but not in lock-step u Basic control done using Ti. num. Procs() Ti. this. Proc() => total number of processors => id of executing processor u Bulk-synchronous style read remote particles and compute forces on mine Ti. barrier(); write to my particles using new forces Ti. barrier(); u This is neither message passing nor data-parallel SC 2003 S 09: Programming with the Partitioned Global Address Space Model 11/16/03 74