Array Operation Synthesis to Optimize Data Parallel Programs

Array Operation Synthesis to Optimize Data Parallel Programs Department of Computer Science, National Tsing-Hua University Student: Gwan-Hwang Advisor: Dr. Jenq Kuen Lee

Array Operation Synthesis to Optimize Data Parallel Programs 國立清華大學 資訊 程系 Student: 黃冠寰 Advisor: 李政崑博士

Array Operation Synthesis on Distributed-memory Machines 國立清華大學 資訊 程學系 黃冠寰, Phd.

Research Interests Key Issues Compiler Optimization for Parallel Computations on Distributed & Shared Memory Machines Concurrent Testing Parallel Object Program Model & Heterogeneous Computing • Array Operation Synthesis for Intrinsic Array Functions (JPDC, ACM PPo. PP’ 95, ICPP’ 96 ) • Automatic Alignment for Data Parallel Languages (LCPC’ 97) • Communication Code for Block-Cyclic Distribution of HPF(IPPS’ 98) • Reachability Testing of Concurrent Program (IJSEKE’ 95, APSEC’ 93) • Java-Based Network Computing Environment • Transparent Parallel Computing Environment (Ongoing)

Outline of Presentation • • • Fortran 90 Intrinsic Array Operations Array Operation Synthesis(AOS) SYNTOOL Apply AOS to Shared-Memory Machines Apply AOS to Distributed-Memory Machines • Conclusion and Future Work

Outline of Presentation • • • Fortran 90 Intrinsic Array Operations Array Operation Synthesis(AOS) SYNTOOL Apply AOS to Shared-Memory Machines Apply AOS to Distributed-Memory Machines Integrate AOS with Automatic Data Alignment • Conclusion and Future Work

Intrinsic Array Operations • Provided by Modern Program Languages. v E. g. Fortran 90, High Performance Fortran(HPF), HPF 2, Fortran 97, APL, MATLAB, MATHEMATICA, NESL, C* • Engineering and Scientific Applications • Facilitate a Compilation Analysis for Optimization • Support Parallel Execution and Portability

Intrinsic Array Operations(Cont’d) • Array Operations Provided by Fortran 90, HPF. CSHIFT, TRANSPOSE, MERGE, EOSHIFT, RESHAPE SPREAD, Section Move, Where Constructs, Reductions. • Examples: B=CSHIFT(A, 1, 1) C=TRANSPOSE(B)

Consecutive Array Expressions • Array Expression C=EOSHIFT(MERGE(RESHAPE(S, /N, N/), A+B, T), 1, 0, 1) • Consecutive Array Operations FXP=CSHIFT(F 1, 1, +1) FXM=CSHIFT(F 1, 1, -1) FXP=CSHIFT(F 1, 2, +1) FYM=CSHIFT(F 1, 2, -1) FDERIV=ZXP*(FXP-F 1)+ZXM*(FXM-F 1)+ ZYP*(FYP-F 1)+ZYM*(FYM-F 1)

Classification of Array Operations • Model Array Operations by Data Access Functions (DAF) Type 4 Type 2 Type 1 Type 3

Data Access Functions • Represent Array Operations by Mathematical Functions • Model Array Operations by Data Access Functions (DAF) v. Single-Source, Multiple-Source v. Single-Clause, multiple-Clause

Type 1: Single-source Singleclause Data Access Function • One Source Array • One Data Access Pattern B=TRANSPOSE(A) Data Access Function is B(I, J)=A(J, I)

Single-source Single-clause Data Access Function • One Source Array • One Data Access Pattern B=TRANSPOSE(A) Data Access Function is B(I, J)=A(J, I)

Type 2: Multiple-source Singleclause Data Access Function • Multiple Source Arrays • One Data Access Pattern R=MERGE(T, F, M) Array T Array F Array M Data Access Function is where Array R

Multiple-source Single-clause Data Access Function • Multiple Source Arrays • One Data Access Pattern R=MERGE(T, F, M) Array T Array F Array M Data Access Function is where Array R

Type 3: Single-source Multipleclause Data Access Function • Single Source Array • Multiple Data Access Patterns B=CSHIFT(A, 1, 1) Array A Array B Data Access Function is : a segmentation descriptor

Single-source Multiple-clause Data Access Function • Single Source Array • Multiple Data Access Patterns B=CSHIFT(A, 1, 1) Array A Array B Data Access Function is : a segmentation descriptor

Type 4: Multiple-source Multipleclause Data Access Function • Multiple Source Arrays • Multiple Data Access Patterns v. No array operation of Fortran 90 belongs to type 4 v. Synthesis of multiple array operations may derive a type 4 data access function.

Multiple-source Multiple-clause Data Access Function • Multiple Source Arrays • Multiple Data Access Patterns v. No array operation of Fortran 90 belongs to this type v. Synthesis of multiple array operations may derive a multiple-source multiple-clause data access function

Straightforward Compilation • Translate each operation into a parallel loop B=CSHIFT((TRANSPOSE(EOSHIFT(A, 1, 0, 1), 1, 1) EOSHIFT TRANSPOSE CSHIFT FORALL (I=1: N: 1; J=1: N: 1) IF (1<=I<=N-1) and (1<=J<=N) THEN T 1(I, J)=A(I+1, J) ELSE T 1(I, J)=0 ENDFORALL (I=1: N: 1; J=1: N: 1) T 2(I, J)=T 1(J, I) ENDFORALL (I=1: N: 1; J=1: N: 1) IF (1<=I<=N-1) and (1<=J<=N) THEN B(I, J)=T 2(I+1, J) ELSE B(I, J)=T 2(I-N, J) ENDFORALL

Array Operation Synthesis B=CSHIFT((TRANSPOSE(EOSHIFT(A, 1, 0, 1), 1, 1) • Construct the Parse Tree of Array Expression • Represent Array Operations by Mathematical Functions (DAF) EOSHIFT TRANSPOSE CSHIFT

Array Operation Synthesis (Cont’d) Synthesis of two functions CSHIFT TRANSPOSE COSHIFT+ TRANSPOSE EOSHIFT

Synthesis of two Data Access Functions • Substitution (Term Rewriting like method) v. Having two Data Access Patterns: where v. The Synthesized Data Access Pattern is: where

Synthesis of two DAFs (Cont’d) • For example, • By the substitution rule

Synthesis of two DAFs (Cont’d) • For example,

Code Generation for Synthesized Data Access Function Code Generation FORALL (I=1: N: 1; J=1: N: 1) IF (/I, J/, /1: N-1, 1: N/) (/J, I+1/, /1: N-1, 1: N/) THEN B(I, J)=A(J+1, I+1) IF (/I, J/, /1: N-1, 1: N/) (/J, I+1/, /N: N THEN B(I, J)=0 , 1: N/) IF (/I, J/, /N: N , 1: N/) (/J, I+1/, /1: N-1, 1: N/) THEN B(I, J)=A(J+1, I-N+1) IF (/I, J/, /N: N , 1: N/) (/J, I+1/, /N: N THEN B(I, J)=0 ENDFORALL , 1: N/)

Code Generation for Synthesized Data Access Function After Optimization 1 1 N-1 N

Optimization • Simplifying the ranges at compilation time instead of runtime • Optimization process: v. Normalize: v. Intersection for each dimension: