GAMMA An Efficient Distributed Shared Memory Toolbox for

GAMMA: An Efficient Distributed Shared Memory Toolbox for MATLAB Rajkiran Panuganti 1, Muthu Baskaran 1, Jarek Nieplocha 2, Ashok Krishnamurthy 3, Atanas Rountev 1, P. Sadayappan 1 1 The Ohio State University 2 PNNL 3 Ohio Supercomputer Center

GAMMA: An Efficient Distributed Shared Memory Toolbox for MATLAB Overview • Motivation • GAMMA Programming Model • Implementation Overview • Experimental Evaluation • Conclusions 9/24/2021 2

GAMMA: An Efficient Distributed Shared Memory Toolbox for MATLAB High Productivity Computing • Programmers’ productivity is extremely important • C/Fortran – Good performance but poor productivity • Parallel Programming in C/Fortran even harder • MATLAB, Python etc. – Good programmer productivity • Poor performance and inability to run large scale problems (memory limitations) 9/24/2021 3

GAMMA: An Efficient Distributed Shared Memory Toolbox for MATLAB and High Productivity • Numerous features resulting in High Programmer Productivity: ü ü ü ü Array Based Semantics Copy/Value based semantics Debugging and Profiling Support Integrated Development Environment Numerous Domain Specific libraries (Toolboxes) Visualization And a lot more. . . • Need to retain above features while addressing performance Issues 9/24/2021 4

GAMMA: An Efficient Distributed Shared Memory Toolbox for MATLAB Problem Out-Of. Memory! Performance! 199 sec Out-Of. Memory! 10. 19 s 9/24/2021 5

GAMMA: An Efficient Distributed Shared Memory Toolbox for MATLAB Para. M : - ‘Parallel MATLAB’ USER user DPara. M GAMMA user Library Writers Specialized Libraries mex. MPI Compiler MATLAB GA + MVAPICH 9/24/2021 GA + MVAPICH 6

GAMMA: An Efficient Distributed Shared Memory Toolbox for MATLAB Overview • Motivation • GAMMA Programming Model • Implementation Overview • Experimental Evaluation • Conclusions 9/24/2021 7

GAMMA: An Efficient Distributed Shared Memory Toolbox for MATLAB Programming Model • Global Shared View of the distributed Array (1, 1) P 0 P 2 Physical View Logical View P 1 P 3 (250, 75) (700, 610) (1024, 1024) A = GA([1024, 1024], distr); Block = A(250: 700, 75: 610); 9/24/2021 8

GAMMA: An Efficient Distributed Shared Memory Toolbox for MATLAB Programming Model (Contd. . ) • Get-Compute-Put Computation Model Get() Put() Process 0 Compute Process 1 Compute 9/24/2021 9

GAMMA: An Efficient Distributed Shared Memory Toolbox for MATLAB Other features in the Programming Model enabling Efficiency • Pass-by-reference semantics for distributed arrays ü Intended for Library writers • Management of Data Locality (NUMA) ü Distribution information can be retrieved by the programmer ü Reference based access to the local data • Data replication ü Support for replicating near-neighbor data 9/24/2021 10

GAMMA: An Efficient Distributed Shared Memory Toolbox for MATLAB Other features in the Programming Model enabling Efficiency Contd. . • Asynchronous operations ü Support for Library Writers • Interoperable with ‘Message Passing’ ü Message Passing support using ‘mex. MPI’ • Interoperable with some other ‘Parallel MATLAB’ projects ü Interoperable with p. MATLAB, Mathworks DCT 9/24/2021 11

GAMMA: An Efficient Distributed Shared Memory Toolbox for MATLAB Illustration by Example (FFT 2) – 2 D FFT [rank, nprocs] = Begin(); dims = [N N]; distr = [N N/nprocs]; A = GA(dims, distr); tmp=local(A); % GET() tmp = fft(tmp); % Compute() Put(A, tmp); % PUT() Sync(); ATmp = GA(A); Transpose(A, ATmp); % Collective Ops Tmp = local(ATmp); Put(ATmp, fft(Tmp)); Sync(); Transpose(ATmp, A); GA_End(); 9/24/2021 Transpose 12

GAMMA: An Efficient Distributed Shared Memory Toolbox for MATLAB Software Architecture User MATLAB Front-End GAMMA MATLAB Computation Engine 9/24/2021 GA MPI mex. MPI SCALAPACK 13

GAMMA: An Efficient Distributed Shared Memory Toolbox for MATLAB Overview • Motivation • GAMMA Programming Model • Implementation Overview • Experimental Evaluation • Conclusions 9/24/2021 14

GAMMA: An Efficient Distributed Shared Memory Toolbox for MATLAB Evaluation • OSC Pentium 4 Cluster • Two 2. 4 GHz Intel P 4 processors per node, Linux kernel 2. 6. 6 , 4 GB RAM, • MVAPICH 0. 9. 4 • Infiniband • MATLAB Version 7. 01 • Fully distributed environment • Evaluation using NAS Benchmarks 9/24/2021 15

GAMMA: An Efficient Distributed Shared Memory Toolbox for MATLAB Programmability Moderate Slight Increase in SLOC 9/24/2021 16

GAMMA: An Efficient Distributed Shared Memory Toolbox for MATLAB Performance Analysis 9/24/2021 17

GAMMA: An Efficient Distributed Shared Memory Toolbox for MATLAB Performance Analysis 9/24/2021 18

GAMMA: An Efficient Distributed Shared Memory Toolbox for MATLAB Speedup on Large Problem Sizes 9/24/2021 20

GAMMA: An Efficient Distributed Shared Memory Toolbox for MATLAB Related Work • Early 90’s – MPI & Cluster Programming • 1995 – ‘Why there isn’t a Parallel MATLAB? ’ – Cleve Moler • Embarrassingly Parallel ü Paralize(’ 98); Multi(’ 00); PLab(‘ 00); Parmatlab(‘ 01); • Message Passing ü Multi. Matlab(’ 96); PT(’ 96); DPToolbox(‘ 99); MATmarks(‘ 99); PMI(’ 99); MPITB/PVMTB(‘ 00); CMTM(‘ 01); • Compilation Based ü Conlab(‘ 93); Falcon(’ 95); Par. AL(‘ 95); Otter(‘ 98); Menhir(’ 98); Ma. JIC(’ 98); MATCH(‘ 00); RTExpress(’ 00); • Backend Support ü Matpar(‘ 98); DLab(‘ 99); Netsolve(‘ 01); Paramat(‘ 01); 9/24/2021 21

GAMMA: An Efficient Distributed Shared Memory Toolbox for MATLAB Related Work (Currently Active) • Star-P (’ 97) – MIT • Matlab. MPI(’ 98); p. MATLAB(’ 02) – MIT-LL; ü File-based Message Passing Communication • MATLAB_D (’ 00) – Rice ü Telescoping Compilation + HPF + JIT Compilation • Para. M (’ 04) – OSU & OSC • Mathworks(’ 04) – MDCE/MDCT 9/24/2021 22

GAMMA: An Efficient Distributed Shared Memory Toolbox for MATLAB Conclusions • Discussed an efficient Distributed Shared Memory Toolbox for MATLAB • Programming Model and Efficiency features of the toolbox • Demonstrated efficiency using NAS Benchmarks • Download available upon request 9/24/2021 23

GAMMA: An Efficient Distributed Shared Memory Toolbox for MATLAB Questions ? Contact: panugant@cse. ohio-state. edud 9/24/2021 24

GAMMA: An Efficient Distributed Shared Memory Toolbox for MATLAB Backup • NAS FT – A • NAS EP – A • Implementation Issues 9/24/2021 25

GAMMA: An Efficient Distributed Shared Memory Toolbox for MATLAB Performance Analysis Contd… 9/24/2021 26

GAMMA: An Efficient Distributed Shared Memory Toolbox for MATLAB Implementation Issues • Different Memory managers • Automated Book Keeping • Data layout inconsistencies • In-Place Operations • Data movement between different workspaces • Out-of-order and irregular accesses 9/24/2021 27

GAMMA: An Efficient Distributed Shared Memory Toolbox for MATLAB 9/24/2021 28