MPI Message Passing Interface Basics Grid Computing B

MPI (Message Passing Interface) Basics Grid Computing, B. Wilkinson, 2004 1

MPI (Message Passing Interface) • Message passing library standard developed by group of academics and industrial partners to foster more widespread use and portability. • Defines routines, not implementation. • Several free implementations exist. Grid Computing, B. Wilkinson, 2004 2

MPI designed: • To address some problems with earlier message-passing system such as PVM. • To provide powerful message-passing mechanism and routines - over 126 routines (although it is said that one can write reasonable MPI programs with just 6 MPI routines). Grid Computing, B. Wilkinson, 2004 3

Unsafe Message Passing Intended behavior: Process 0 Example Process 1 Destination send(…, 1, …); lib() send(…, 1, …); Source recv(…, 0, …); lib() recv(…, 0, …); Grid Computing, B. Wilkinson, 2004 4

Unsafe Message Passing Possible behavior: Process 0 Example Process 1 Destination send(…, 1, …); lib() send(…, 1, …); Source recv(…, 0, …); lib() recv(…, 0, …); Grid Computing, B. Wilkinson, 2004 5

• Message tags not sufficient to deal with such situations -- library routines might use same tags. • MPI introduces concept of a “communicator” - which defines the scope of communication. Grid Computing, B. Wilkinson, 2004 6

MPI Communicators • Defines a communication domain - a set of processes that are allowed to communicate between themselves. • Communication domains of libraries can be separated from that of a user program. • Used in all point-to-point and collective MPI message-passing communications. Grid Computing, B. Wilkinson, 2004 7

Default Communicator MPI_COMM_WORLD • Exists as first communicator for all processes existing in the application. • A set of MPI routines exists forming communicators. • Processes have a “rank” in a communicator. Grid Computing, B. Wilkinson, 2004 8

MPI Process Creation and Execution • Purposely not defined - Will depend upon implementation. • Only static process creation supported in MPI version 1. All processes must be defined prior to execution and started together. • Originally SPMD model of computation. Grid Computing, B. Wilkinson, 2004 9

SPMD Computational Model main (int argc, char *argv[]) { MPI_Init(&argc, &argv); . . MPI_Comm_rank(MPI_COMM_WORLD, &myrank); /*get rank*/ if (myrank == 0) master(); /* routine for master to execute */ else slave(); /* routine for slaves to execute */. . MPI_Finalize(); } Grid Computing, B. Wilkinson, 2004 10

MPI Point-to-Point Communication • Uses send and receive routines with message tags (and communicator) • Wild card message tags available Grid Computing, B. Wilkinson, 2004 11

MPI Blocking Routines • Return when “locally complete” – when location used to hold message can be used again or altered without affecting message being sent. • Blocking send will send message and return – does not mean that message has been received, just that process free to move on without adversely affecting message. Grid Computing, B. Wilkinson, 2004 12

Parameters of blocking send Grid Computing, B. Wilkinson, 2004 13

Parameters of blocking receive Grid Computing, B. Wilkinson, 2004 14

Example To send an integer x from process 0 to process 1, int x; MPI_Comm_rank(MPI_COMM_WORLD, &myrank); /* find rank */ if (myrank == 0) { MPI_Send(&x, 1, MPI_INT, 1, msgtag, MPI_COMM_WORLD); } else if (myrank == 1) { MPI_Recv(&x, 1, MPI_INT, 0, msgtag, MPI_COMM_WORLD, status); } Grid Computing, B. Wilkinson, 2004 15

MPI Nonblocking Routines • Nonblocking send - MPI_Isend() – will return “immediately” even before source location is safe to be altered. • Nonblocking receive - MPI_Irecv() – will return even if no message to accept. Grid Computing, B. Wilkinson, 2004 16

Nonblocking Routine Formats MPI_Isend(buf, count, datatype, dest, tag, comm, request) MPI_Irecv(buf, count, datatype, source, tag, comm, request) Completion detected by MPI_Wait() and MPI_Test(). MPI_Wait() waits until operation completed, returns then. MPI_Test() returns with flag set indicating whether operation completed at that time. Need to know whether particular operation completed. Determined by accessing request parameter. Grid Computing, B. Wilkinson, 2004 17

Example To send an integer x from process 0 to process 1 and allow process 0 to continue, MPI_Comm_rank(MPI_COMM_WORLD, &myrank); /* find rank */ if (myrank == 0) { MPI_Isend(&x, 1, MPI_INT, 1, msgtag, MPI_COMM_WORLD, req 1); compute(); MPI_Wait(req 1, status); } else if (myrank == 1) { MPI_Recv(&x, 1, MPI_INT, 0, msgtag, MPI_COMM_WORLD, status); } Grid Computing, B. Wilkinson, 2004 18

Send Communication Modes • Standard Mode Send - Not assumed that corresponding receive routine has started. Amount of buffering not defined by MPI. If buffering provided, send could complete before receive reached. • Buffered Mode - Send may start and return before a matching receive. Necessary to specify buffer space via routine MPI_Buffer_attach(). • Synchronous Mode - Send and receive can start before each other but can only complete together. • Ready Mode - Send can only start if matching receive already reached, otherwise error. Use with care. Grid Computing, B. Wilkinson, 2004 19

• Each of the four modes can be applied to both blocking and nonblocking send routines. • Only the standard mode is available for the blocking and nonblocking receive routines. • Any type of send routine can be used with any type of receive routine. Grid Computing, B. Wilkinson, 2004 20

Collective Communication Involves set of processes, defined by an “intracommunicator. ” Message tags not present. Principal collective operations: • • MPI_Bcast() - Broadcast from root to all other processes MPI_Gather() - Gather values for group of processes MPI_Scatter() - Scatters buffer in parts to group of processes MPI_Alltoall() - Sends data from all processes to all processes MPI_Reduce() - Combine values on all processes to single value MPI_Reduce_scatter() - Combine values and scatter results MPI_Scan() - Compute prefix reductions of data on processes Grid Computing, B. Wilkinson, 2004 21

Example To gather items from group of processes into process 0, using dynamically allocated memory in root process: int data[10]; /*data to be gathered*/ MPI_Comm_rank(MPI_COMM_WORLD, &myrank); /* find rank */ if (myrank == 0) { MPI_Comm_size(MPI_COMM_WORLD, &grp_size); /*find group size*/ buf = (int *)malloc(grp_size*10*sizeof(int)); /*allocate memory*/ } MPI_Gather(data, 10, MPI_INT, buf, grp_size*10, MPI_INT, 0, MPI_COMM_WORLD); MPI_Gather() gathers from all processes, including root. Grid Computing, B. Wilkinson, 2004 22

Barrier • As in all message-passing systems, MPI provides a means of synchronizing processes by stopping each one until they all have reached a specific “barrier” routine. Grid Computing, B. Wilkinson, 2004 23

Barrier Concept Grid Computing, B. Wilkinson, 2004 24

Using Library Routines Grid Computing, B. Wilkinson, 2004 25

Grid Computing, B. Wilkinson, 2004 26

Measuring Execution Time To measure execution time between point L 1 and point L 2 in the code, might have: . L 1: time(&t 1); /* start timer */. . L 2: time(&t 2); /* stop timer */. elapsed_t = difftime(t 2, t 1); /*t 2 - t 1*/ printf(“Elapsed time = %5. 2 f”, elapsed_t); MPI provides MPI_Wtime() for returning time (in seconds). Grid Computing, B. Wilkinson, 2004 27

Executing MPI programs • MPI version 1 standard does not address implementation and did not specify how programs are to be started and each implementation has its own way. Grid Computing, B. Wilkinson, 2004 28

• Several MPI implementations, such as MPICH and LAM MPI, use command: mpirun -np # prog where # is the number of processes and prog is the program. Additional argument specify computers (see later). Grid Computing, B. Wilkinson, 2004 29

MPI-2 • The MPI standard, version 2 does recommend a command for starting MPI programs, namely: mpiexec -n # prog where # is the number of processes and prog is the program. Grid Computing, B. Wilkinson, 2004 30

Sample MPI Programs Grid Computing, B. Wilkinson, 2004 31

Hello World #include "mpi. h" #include <stdio. h> int main(int argc, char *argv[]) { MPI_Init(&argc, &argv); printf("Hello Worldn"); MPI_Finalize(); return 0; } Grid Computing, B. Wilkinson, 2004 32

Hello World Printing out rank of process #include "mpi. h" #include <stdio. h> int main(int argc, char *argv[]) { int myrank, numprocs; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &myrank); MPI_Comm_size(MPI_COMM_WORLD, &numprocs) printf("Hello World from process %d of %dn", myrank, numprocs); MPI_Finalize(); return 0; } Grid Computing, B. Wilkinson, 2004 33

Question Suppose this program is compiled as helloworld and is executed on a single computer with the command: mpirun -np 4 helloworld What would the output be? Grid Computing, B. Wilkinson, 2004 34

Answer Several possible outputs depending upon order processes are executed. Example Hello World from process 2 of 4 Hello World from process 0 of 4 Hello World form process 1 of 4 Hello World form process 3 of 4 Grid Computing, B. Wilkinson, 2004 35

Adding communication to get process 0 to print all messages: #include "mpi. h" #include <stdio. h> int main(int argc, char *argv[]) { int myrank, numprocs; char greeting[80]; /* message sent from slaves to master */ MPI_Status status; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &myrank); MPI_Comm_size(MPI_COMM_WORLD, &numprocs); sprintf(greeting, "Hello World from process %d of %dn", rank, size); if (myrank == 0 ) { /* I am going print out everything */ printf("sn", greeting); /* print greeting from proc 0 */ for (i = 1; i < numprocs; i++) { /* greetings in order */ MPI_Recv(geeting, sizeof(greeting), MPI_CHAR, i, 1, MPI_COMM_WORLD, &status); printf(%sn", greeting); } } else { MPI_Send(greeting, strlen(greeting)+1, MPI_CHAR, 0, 1, MPI_COMM_WORLD); } MPI_Finalize(); return 0; } Grid Computing, B. Wilkinson, 2004 36

MPI_Get_processor_name() Return name of processor executing code (and length of string). Arguments: MPI_Get_processor_name(char *name, int *resultlen) Example returned in here int namelen; char procname[MPI_MAX_PROCE�SSOR_NAME]; MPI_Get_processor_name(procname, &namelen); Grid Computing, B. Wilkinson, 2004 37

Easy then to add name in greeting with: sprintf(greeting, "Hello World from process %d of %d on $sn", rank, size, procname); Grid Computing, B. Wilkinson, 2004 38

Pinging processes and timing Master-slave structure #include <mpi. h> void master(void); void slave(void); int main(int argc, char **argv){ int myrank; printf("This is my ping programn"); MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &myrank); if (myrank == 0) { master(); } else { slave(); } MPI_Finalize(); return 0; } Grid Computing, B. Wilkinson, 2004 39

Master routine void master(void){ int x = 9; double starttime, endtime; MPI_Status status; printf("I am the master - Send me a message when you receive this number %dn", x); starttime = MPI_Wtime(); MPI_Send(&x, 1, MPI_INT, 1, 1, MPI_COMM_WORLD); MPI_Recv(&x, 1, MPI_INT, 1, 1, MPI_COMM_WORLD, &status); endtime = MPI_Wtime(); printf("I am the master. I got this back %d n", x); printf("That took %f secondsn", endtime - starttime); } Grid Computing, B. Wilkinson, 2004 40

Slave routine void slave(void){ int x; MPI_Status status; printf("I am the slave - workingn"); MPI_Recv(&x, 1, MPI_INT, 0, 1, MPI_COMM_WORLD, &status); printf("I am the slave. I got this %d n", x); MPI_Send(&x, 1, MPI_INT, 0, 1, MPI_COMM_WORLD); } Grid Computing, B. Wilkinson, 2004 41

Example using collective routines MPI_Bcast() MPI_Reduce() Adding numbers in a file. Grid Computing, B. Wilkinson, 2004 42

#include “mpi. h” #include <stdio. h> #include <math. h> #define MAXSIZE 1000 void main(int argc, char *argv){ int myid, numprocs; int data[MAXSIZE], i, x, low, high, myresult, result; char fn[255]; char *fp; MPI_Init(&argc, &argv); MPI_Comm_size(MPI_COMM_WORLD, &numprocs); MPI_Comm_rank(MPI_COMM_WORLD, &myid); if (myid == 0) { /* Open input file and initialize data */ strcpy(fn, getenv(“HOME”)); strcat(fn, ”/MPI/rand_data. txt”); if ((fp = fopen(fn, ”r”)) == NULL) { printf(“Can’t open the input file: %snn”, fn); exit(1); } for(i = 0; i < MAXSIZE; i++) fscanf(fp, ”%d”, &data[i]); } MPI_Bcast(data, MAXSIZE, MPI_INT, 0, MPI_COMM_WORLD); /* broadcast data */ x = n/nproc; /* Add my portion Of data */ low = myid * x; high = low + x; for(i = low; i < high; i++) myresult += data[i]; printf(“I got %d from %dn”, myresult, myid); /* Compute global sum */ MPI_Reduce(&myresult, &result, 1, MPI_INT, MPI_SUM, 0, MPI_COMM_WORLD); if (myid == 0) printf(“The sum is %d. n”, result); MPI_Finalize(); } Grid Computing, B. Wilkinson, 2004 43

Compiling/Executing MPI Programs Preliminaries • Set up paths • Create required directory structure • Create a file listing machines to be used (“hostfile”) Grid Computing, B. Wilkinson, 2004 44

Before starting MPI for the first time, need to create a hostfile Sample hostfile terra #venus //Currently not used, commented out leo 1 leo 2 leo 3 leo 4 leo 5 leo 6 leo 7 leo 8 Grid Computing, B. Wilkinson, 2004 45

Compiling/executing (SPMD) MPI program For LAM MPI version 6. 5. 2. At a command line: To start MPI: First time: lamboot -v hostfile Subsequently: lamboot To compile MPI programs: mpicc -o file. c or mpi. CC -o file. cpp To execute MPI program: mpirun -v -np no_processors file To remove processes for reboot lamclean -v Terminate LAM lamhalt If fails: wipe -v lamhost Grid Computing, B. Wilkinson, 2004 46

Compiling/Executing Multiple MPI Programs Create a file, say called appfile, specifying programs: Example 1 master and 2 slaves, appfile contains n 0 master n 0 -1 slave To execute: mpirun -v appfile Sample output 3292 master running on n 0 (o) 3296 slave running on n 0 (o) 412 slave running on n 1 Grid Computing, B. Wilkinson, 2004 47

Parallel Programming Home Page http: //www. cs. uncc. edu/par_prog Gives step-by-step instructions for compiling and executing programs, and other information. Grid Computing, B. Wilkinson, 2004 48