Introduction to Parallel Programming with MPI Morris Law

Introduction to Parallel Programming with MPI Morris Law, SCID Apr 25, 2018

Multi-core programming ¡ ¡ ¡ Currently, most CPUs has multiple cores that can be utilized easily by compiling with openmp support Programmers no longer need to rewrite a sequential code but to add directives to instruct the compiler for parallelizing the code with openmp. For reference site: http: //bisqwit. iki. fi/story/howto/openmp/

Openmp example /* * Sample program to test runtime of simple matrix multiply * with and without Open. MP on gcc-4. 3. 3 -tdm 1 (mingw) * compile with gcc –fopenmp * (c) 2009, Rajorshi Biswas */ #include printf("Completed array init. n"); printf("Crunching without OMP. . . "); fflush(stdout); start = omp_get_wtime(); for(i=0; i<n; ++i) { for(j=0; j<n; ++j) { temp = 0; for(k=0; k<n; ++k) { temp += arr 1[i][k] * arr 2[k][j]; } arr 3[i][j] = temp; } } <stdio. h> <stdlib. h> <time. h> <assert. h> #include <omp. h> int main(int argc, char **argv) { int i, j, k; int n; double temp; double start, end, run; printf("Enter dimension ('N' for 'Nx. N' matrix) (100 -2000): "); scanf("%d", &n); assert( n >= 100 && n <= 2000 ); int **arr 1 = malloc( sizeof(int*) * n); int **arr 2 = malloc( sizeof(int*) * n); int **arr 3 = malloc( sizeof(int*) * n); for(i=0; i<n; ++i) { arr 1[i] = malloc( sizeof(int) * n ); arr 2[i] = malloc( sizeof(int) * n ); arr 3[i] = malloc( sizeof(int) * n ); } printf("Populating array with random values. . . n"); srand( time(NULL) ); for(i=0; i<n; ++i) { for(j=0; j<n; ++j) { arr 1[i][j] = (rand() % n); arr 2[i][j] = (rand() % n); } } end = omp_get_wtime(); printf(" took %f seconds. n", end-start); printf("Crunching with OMP. . . "); fflush(stdout); start = omp_get_wtime(); #pragma omp parallel for private(i, j, k, temp) for(i=0; i<n; ++i) { for(j=0; j<n; ++j) { temp = 0; for(k=0; k<n; ++k) { temp += arr 1[i][k] * arr 2[k][j]; } arr 3[i][j] = temp; } } end = omp_get_wtime(); printf(" took %f seconds. n", end-start); } return 0;

Compiling for openmp support ¡ GCC gcc –fopenmp –o foo. c gfortran –fopenmp –o foo. f ¡ Intel Compiler icc -openmp –o foo. c ifort –openmp –o foo. f ¡ PGI Compiler pgcc -mp –o foo. c pgf 90 –mp –o foo. f

What is Message Passing Interface (MPI)? Portable standard for communication ¡ Processes can communicate through messages. ¡ Each process is a separable program ¡ All data is private ¡

What is Message Passing Interface (MPI)? This is a library, not a language!! ¡ Different compilers, but all must use the same libraries, i. e. MPICH, LAM, OPENMPI etc. ¡ Use standard sequential language. Fortran, C, C++, etc. ¡

Basic Idea of Message Passing Interface (MPI) ¡ MPI Environment l ¡ Communication between processes l l ¡ Initialize, manage, and terminate communication among processes Point to point communication, i. e. send, receive, etc. Collective communication, i. e. broadcast, gather, etc. Complicated data structures l l Communicate the data effectively e. g. matrices and memory

Message Passing Model Process P 1 P 2 Serial P 3 time Process 0 P 1 Data exchange via interconnection Process 1 P 2 Process 2 P 3 time Message Passing

General MPI program structure MPI include file variable declarations Initialize MPI environment Do work and make message passing calls Terminate MPI Environment #include <mpi. h> int main (int argc, char *argv[]) { int np, rank, ierr; ierr = MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank) ; MPI_Comm_size(MPI_COMM_WORLD, &np); /* Do Some Works */ printf(“Helloworld, I’m P%d of %dn”, rank, np); ierr = MPI_Finalize(); } Helloworld, I’m P 0 of 3 Helloworld, I’m P 1 of 3 Helloworld, I’m P 2 of 3

When Use MPI? You need a portable parallel program ¡ You are writing a parallel library ¡ You care about performance ¡ You have a problem that can be solved in parallel ways ¡

F 77/F 90, C/C++ MPI library calls ¡ Fortran 77/90 uses subroutines l l l ¡ CALL is used to invoke the library call Nothing is returned, the error code variable is the last argument All variables are passed by reference C/C++ uses functions l l l Just the name is used to invoke the library call The function returns an integer value (an error code) Variables are passed by value, unless otherwise specified

Types of Communication ¡ Point to Point Communication l ¡ communication involving only two processes. Collective Communication l communication that involves a group of processes.

Implementation of MPI

Browse the sample files ¡ ¡ Inside your home directory, the sample zip file, mpi 1. zip has been stored for the laboratory. Please unzip the file unzip mpi-1. zip ¡ There shall be 4 subdirectories inside mpi-1 ls –l mpi-1 total 20 drwxr-xr-x. drwxrwxr-x. ¡ 2 2 2 morris morris dean dean 4096 4096 Nov Nov Nov 27 27 27 09: 55 09: 41 09: 51 09: 49 09: 51 00 openmp 0 -hello array-pi mc-pi series-pi The 4 subdirectories stored sample mpi-1 programs with README files for this laboratory.

First MPI C program: - hello 1. c ¡ Change directory to hello and use an editor, e. g. nano to open hello 1. c cd hello nano hello 1. c #include <stdio. h> #include <mpi. h> int main(int argc, char *argv[]) { int version, subversion; MPI_Init(&argc, &argv); MPI_Get_version(&version, &subversion); printf("Hello world!n"); printf("Your MPI Version is: %d. %dn", version, subversion); MPI_Finalize(); return(0); }

First MPI Fortran program: - hello 1. f ¡ Use an editor to open hello 1. f cd hello nano hello 1. f program main include 'mpif. h' integer ierr, version, subversion call MPI_INIT(ierr) call MPI_GET_VERSION(version, subversion, ierr) print *, 'Hello world!' print *, 'Your MPI Version is: ', version, '. ', subversion call MPI_FINALIZE(ierr) end

Second MPI C program: - hello 2. c ¡ Use an editor to open hello 2. c cd hello nano hello 2. c #include <stdio. h> #include <mpi. h> int main(int argc, char *argv[]) { int rank, size; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &size); printf("Hello world! I am P%d of %dn", rank, size); MPI_Finalize(); return(0); }

Second MPI Fortran program: hello 2. f ¡ Use an editor to open hello 2. f cd hello nano hello 2. f program main include 'mpif. h' integer rank, size, ierr call MPI_INIT(ierr) call MPI_COMM_RANK(MPI_COMM_WORLD, rank, ierr) call MPI_COMM_SIZE(MPI_COMM_WORLD, size, ierr) print *, 'Hello world! I am P', rank, ' of ', size call MPI_FINALIZE(ierr) end

Make all files in hello ‘Makefile’ is written for each example directory. ¡ Run ‘make’ will compile all hello examples ¡ make /usr/lib 64/mpich/bin/mpif 77 -o hello. F 1 hello 1. f /usr/lib 64/mpich/bin/mpif 77 -o hello. F 2 hello 2. f /usr/lib 64/mpich/bin/mpicc -o hello. C 1 hello 1. c /usr/lib 64/mpich/bin/mpicc -o hello. C 2 hello 2. c

mpirun hello examples in foreground ¡ You may run the hello examples in foreground by specifying the no of processors and the machinefile with mpirun. e. g. mpirun –np 4 Hello world! ¡ –machinefile machine. /hello. C 2 I am P 0 of 4 I am P 2 of 4 I am P 3 of 4 I am P 1 of 4 machine is the file storing the hostname you want the programs run.

Exercise ¡ Follow the above hello example, mpirun hello. C 1, hello. F 1 and hello. F 2 in foreground with 4 processors in foreground ¡ Change directory to mc-pi, l l ¡ compile all programs inside using ‘make’ Run mpi-mc-pi using 2, 4, 8 processors. Change directory to series-pi, l l l compile all programs inside using ‘make’ Run series-pi using 2, 4, 6, 8 processors. Note the time difference

Parallelization example: serial-pi. c #include <stdio. h> static long num_steps = 10000000; double step; int main () { int i; double x, pi, sum = 0. 0; step = 1. 0/(double) num_steps; for (i=0; i< num_steps; i++){ x = (i+0. 5)*step; sum = sum + 4. 0/(1. 0+x*x); } pi = step * sum; printf("Est Pi= %fn", pi); } 22

Parallelizing serial-pi. c into mpi-pi. c: Step 1: Adding MPI environment #include "mpi. h" #include <stdio. h> static long num_steps = 10000000; double step; int main (int argc, char *argv[]) { int i; double x, pi, sum = 0. 0; MPI_Init(&argc, &argv); step = 1. 0/(double) num_steps; for (i=0; i< num_steps; i++){ x = (i+0. 5)*step; sum = sum + 4. 0/(1. 0+x*x); } pi = step * sum; printf("Est Pi= %fn", pi); MPI_Finalize(); }

Parallelizing serial-pi. c into mpi-pi. c : Step 2: Adding variables to print ranks #include "mpi. h" #include <stdio. h> static long num_steps = 10000000; double step; int main (int argc, char *argv[]) { int i; double x, pi, sum = 0. 0; int rank, size; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &size); step = 1. 0/(double) num_steps; for (i=0; i< num_steps; i++){ x = (i+0. 5)*step; sum = sum + 4. 0/(1. 0+x*x); } pi = step * sum; printf("Est Pi= %f, Processor %d of %d n", pi, rank, size); } MPI_Finalize();

Parallelizing serial-pi. c into mpi-pi. c : Step 3: divide the workload #include "mpi. h" #include <stdio. h> static long num_steps = 10000000; double step; int main (int argc, char *argv[]) { int i; double x, mypi, sum = 0. 0; int rank, size; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &size); step = 1. 0/(double) num_steps; for (i=rank; i< num_steps; i+=size){ x = (i+0. 5)*step; sum = sum + 4. 0/(1. 0+x*x); } mypi = step * sum; printf("Est Pi= %f, Processor %d of %d n", mypi, rank, size); } MPI_Finalize();

Parallelizing serial-pi. c into mpi-pi. c : Step 4: collect partial results #include "mpi. h" #include <stdio. h> static long num_steps = 10000000; double step; int main (int argc, char *argv[]) { int i; double x, mypi, sum = 0. 0; int rank, size; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &size); step = 1. 0/(double) num_steps; for (i=rank; i< num_steps; i+=size){ x = (i+0. 5)*step; sum = sum + 4. 0/(1. 0+x*x); } mypi = step * sum MPI_Reduce(&mypi, &pi, 1, MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD); if (rank==0) printf("Est Pi= %f, n", pi); } MPI_Finalize();

Compile and run mpi program $ mpicc –o mpi-pi. c $ mpirun -np 4 -machinefile machines mpi-pi

Parallelization example 2: serial-mc-pi. c #include <stdio. h> #include <stdlib. h> #include <time. h> main(int argc, char *argv[]) { long in, i, n; double x, y, q; time_t now; in = 0; srand(time(&now)); printf("Input no of samples : "); scanf("%ld", &n); for (i=0; i<n; i++) { x = rand()/(RAND_MAX+1. 0); y = rand()/(RAND_MAX+1. 0); if ((x*x + y*y) < 1) { in++; } } q = ((double)4. 0)*in/n; } printf("pi = %. 20 lfn", q); printf("rmse = %. 20 lfn", sqrt(( (double) q*(4 -q))/n)); 2 r

Parallelization example 2: mpi-mc-pi. c #include "mpi. h" #include <stdio. h> #include <stdlib. h> #include <time. h> main(int argc, char *argv[]) { long in, i, n; double x, y, q, Q; time_t now; int rank, size; MPI_Init(&argc, &argv); in = 0; MPI_Comm_size(MPI_COMM_WORLD, &size); MPI_Comm_rank(MPI_COMM_WORLD, &rank); srand(time(&now)+rank); if (rank==0) { printf("Input no of samples : "); scanf("%ld", &n); } MPI_Bcast(&n, 1, MPI_LONG, 0, MPI_COMM_WORLD); for (i=0; i<n; i++) { x = rand()/(RAND_MAX+1. 0); y = rand()/(RAND_MAX+1. 0); if ((x*x + y*y) < 1) { in++; } } } q = ((double)4. 0)*in/n; MPI_Reduce(&q, &Q, 1, MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD); Q = Q / size; if (rank==0) { printf("pi = %. 20 lfn", Q); printf("rmse = %. 20 lfn", sqrt(( (double) Q*(4 -Q))/n/size)); } MPI_Finalize(); 2 r

Compile and run mpi-mc-pi $ mpicc –o mpi-mc-pi. c $ mpirun -np 4 -machinefile machines mpi-mc-pi

Collective communication 1 3 5 7 105 scatter distribute your data among the processes reduction (e. g. PROD) information of all processes is used to provide a condensed result by/for one process

MPI_Scatter Distributes data from root to all other tasks in a group int MPI_Scatter (void *sendbuf, int sendcnt, MPI_Datatype sendtype , void *recvbuf, int recvcnt, MPI_Datatype recvtype, int root, MPI_Comm comm ) Input Parameters sendbuf address of send buffer (choice, significant only at root ) sendcnt no. of elements sent to each process (integer, significant only at root) sendtype data type of send buffer elements (significant only at root) (handle) recvcnt number of elements in receive buffer (integer) recvtype data type of receive buffer elements (handle) root rank of sending process (integer) communicator (handle) Output Parameter recvbuf address of receive buffer (choice)

MPI_Scatter ¡ Example: two vectors are distributed in order to prepare a parallel computation of their scalar product Data a[0] a[1] a[2] a[3] m 13 Processor 12 11 2 2 11 1 1 10 0 0 MPI_Scatter 10 10 11 12 13 12 3 3 13 MPI_Scatter(&a, 1, MPI_INT, &m, 1, MPI_INT, 2, MPI_COMM_WORLD);

MPI_Reduces values on all processes to a single value on introot. MPI_Reduce (void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm ) Input Parameters sendbuf address of send buffer (choice) count number of elements in send buffer (integer) datatype data type of elements of send buffer (handle) op reduce operation (handle) root rank of root process (integer) communicator (handle) Output Parameter recvbuf address of receive buffer (choice, significant only at root)

MPI_Reduce ¡ Example: calculation of the global minimum of the variables kept by all processes, calculation of a global sum, etc. Data op: MPI_SUM 2 3 5 3 3 9 MPI_Reduce b 2 2 5 a 1 1 3 d 0 2 c Processor b 0 Processor a Data 9 c d 19 MPI_Reduce(&b, &d, 1, MPI_INT, MPI_SUM, 2, MPI_COMM_WORLD);

MPI Datatype Corr. Datatype in C MPI_CHAR signed char MPI_SHORT signed short int MPI_INT signed int MPI_LONG signed long int MPI_UNSIGNED_CHAR unsigned char MPI_UNSIGNED_SHORT unsigned short int MPI_UNSIGNED unsigned int MPI_UNSIGNED_LONG unsigned long int MPI_FLOAT float MPI_DOUBLE double MPI_LONG_DOUBLE long double

Thanks Please give me some comments https: //goo. gl/forms/880 NY 3 k. Z 9 h 7 ay 7 r 32 Morris Law, SCID (morris@hkbu. edu. hk)