Synchronous alltoall pattern continued Iterative Synchronous Pattern In

Synchronous all-to-all pattern continued Iterative Synchronous Pattern In a (fully) synchronous patterns, all processes synchronized at regular points throughout computation, usually to exchange data before proceeding. Already seen the Synchronous all-to-all pattern (Seeds Complete. Sync. Graph pattern) where processes exchange data with each other at synchronization points Some problems of this type require a number of iterations to converge on the solution. ITCS 4/5145 Parallel computing, UNC-Charlotte, B. Wilkinson, June 30, 2012 slides 6 a. ppt 6 a. 1

Synchronous all-to-all pattern (Seeds Complete. Sync. Graph pattern) Example with termination after converging on solution Solving General System of Linear Equations by iteration Suppose equations are of a general form with n equations and n unknowns: where the unknowns are x 0, x 1, x 2, … xn-1 (0 <= i < n). 6 a. 2

By rearranging the ith equation: This equation gives xi in terms of the other unknowns. Can be used as an iteration formula for each of the unknowns to obtain better approximations. Process i computes xi 6 a. 3

Pi needs x’s from all other processes on each iteration P 0 (Excluding Pi) Pn-1 Computes: Pi Needs synchronous all-to-all pattern – Seeds Complete. Sync. Graph pattern. 6 a. 4

Using Seeds Complete. Sync. Graph pattern for the Jacobi iteration method of solving system of linear equations Ful details with code given in “The Complete. Synch. Graph Template Tutorial, ” Jeremy Villalobos and Yawo K. Adibolo, June 18, 2012. at http: //coitweb. uncc. edu/~abw/Pattern. Prog. Group/index. html (to be moved) 6 a. 5

Notes on solution given in Complete. Synch. Graph Template Tutorial Equations in matrix-vector form: AX = B where: Converted into Xk = CXk-1 + D where Xk is solution vector at iteration k , Xk-1 the solution vector at iteration k-1, C a matrix derived from input matrix A and D a vector derived from right hand side vector B. Each slave assigned one or more equations to solve 6 a. 6

For MPI programs can use MPI_Allgather() routine: MPI_All. Gather broadcasts and gather values in one composite construction: int MPI_Allgather(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype, MPI_Comm comm) 6 a. 7

MPI-All. Gather() has the same effect as n MPI_Gather()’s executed, for root = 0 to n-1. MPI_Gather has the same effect as each process executing an MPI_Send() and n MPI_Recv()s Question When does the MPI_All. Gather() return? Answer 6 a. 8

Parallel Code Process Pi could be of the form: x[i] = b[i]; // initialize unknown for (iteration = 0; iteration < limit; iteration++) { sum = -a[i][i] * x[i]; for (j = 0; j < n; j++) // compute summation sum = sum + a[i][j] * x[j]; new_x[i] = (b[i] - sum) / a[i][i]; // compute unknown allgather(&new_x[i]); // bcast/rec values … // barrier to wait for // all procs if needed } 6 a. 9

Jacobi Iteration Name given to a computation that uses the previous iteration value to compute the next values. * All values of x are updated together. Convergence: Can be proven that the Jacobi method will converge if diagonal values of a have an absolute value greater than sum of absolute values of other a’s on row, i. e. if This condition is a sufficient but not a necessary condition. * Other (non-parallel) methods use some of the present iteration values to compute the present values, see later. 6 a. 1

Termination Simple, common approach is compare values computed in one iteration to values obtained from the previous iteration. Terminate computation when all values are within given tolerance; i. e. , when However, this does not guarantee the solution to that accuracy. Why? 6 a. 1

Convergence Rate 6 a. 12

Questions 6 a. 13