Message Passing Programming Based on MPI Derived Data

Message Passing Programming Based on MPI Derived Data Types Bora AKAYDIN 15. 06. 2012

Outline Derived Data Types • Derived Datatypes • Packing/Unpacking Datatypes 15. 06. 2012

Derived Datatypes • How to send only the red elements of V in a single communication? V(0) V(1) V(2) V(3) V(4) V(5) V(6) V(7) V(8) V(9) T(0) T(1) T(2) MPI_Send This method requires inefficient copy of noncontiguous data One possibility, copy these elements to a temporary array before sending. Derived Data Types 15. 06. 2012

Non-struct Derived Data Types • There are routines available in MPI library that are more suitable for an array or vector like data structures: • MPI_Type_contiguous • MPI_Type_vector • MPI_Type_indexed • All above functions work with a single data type! Derived Data Types 15. 06. 2012

MPI_Type_contiguous (C) • Constructs a type consisting of the replication of a data type into continuous locations. int MPI_Type_contiguous( int count, MPI_Datatype old_type, MPI_Datatype *newtype) old type x count (4) = new type Data constructor with MPI_Type_contiguous Derived Data Types 15. 06. 2012

mpi_type_contiguous (Fortran) • Constructs a type consisting of the replication of a data type into continuous locations. MPI_TYPE_CONTIGUOUS( count, MPI_Datatype old_type, MPI_Datatype newtype, ierr) old type x count (4) = new type Data constructor with mpi_type_contiguous Derived Data Types 15. 06. 2012

MPI_Type_contiguous • In C, if we create a matrix with static memory allocation, we can say that data of the matrix will be in contiguous memory. double A[3][3]; A (0, 0) A (1, 0) A (2, 0) A A (0, 1) (0, 2) A A (1, 1) (1, 2) A A (2, 1) (2, 2) Derived Data Types in the memory (C) A (0, 0) A (0, 1) A A (0, 2) (1, 0) A (1, 1) A A (1, 2) (2, 0) A (2, 1) A (2, 2) in the memory (fortran) A (0, 0) A A A (1, 0) (2, 0) (0, 1) 15. 06. 2012 A A (1, 1) (2, 1) A (0, 2) A (1, 2) A (2, 2)

MPI_Type_contiguous (C) double A[3][3]; A (0, 0) A (1, 0) A (2, 0) A A (0, 1) (0, 2) A A (1, 1) (1, 2) A A (2, 1) (2, 2) in the memory A (0, 0) A (0, 1) A A (0, 2) (1, 0) A (1, 1) A A (1, 2) (2, 0) count =3 old_type =MPI_DOUBLE new_type =rowtype MPI_Type_contiguous(int count, MPI_Datatype old_type, MPI_Datatype *newtype); Derived Data Types 15. 06. 2012 A (2, 1) A (2, 2)

mpi_type_contiguous (Fortran) A (0, 0) A A (1, 0) (2, 0) A A A (0, 1) (1, 1) (2, 1) A A A (0, 2) (1, 2) (2, 2) in the memory (fortran) A (0, 0) A (0, 1) A A (0, 2) (1, 0) A (1, 1) A A (1, 2) (2, 0) A (2, 1) count =3 old_type =MPI_REAL new_type =columntype call mpi_type_contiguous(count, MPI_Datatype old_type, MPI_Datatype newtype, ierr); Derived Data Types 15. 06. 2012 A (2, 2)

Handling Non-Contiguous Data • How to send only the red elements of V, while avoiding copying non-contiguous data to a temporary array? V(0) V(1) V(2) V(3) V(4) V(5) V(6) V(7) V(8) V(9) • Define a new data type, in this case a vector with stride of two from original. v. Type Derived Data Types 15. 06. 2012

MPI_Type_vector • old type Similar to contiguous, but allows for regular gaps (stride) in the displacements. (MPI_INT) new type blocklength=3 stride=5 count=2 Data constructor with MPI_Type_vector. Derived Data Types 15. 06. 2012

MPI_Type_vector • • In C, if we create a matrix with static memory allocation, we can say that data of the matrix will be in contiguous memory. Suppose that, we want to send columns to the each task, instead of rows. double A[3][3]; A (0, 0) A (1, 0) A (2, 0) A A (0, 1) (0, 2) A A (1, 1) (1, 2) A A (2, 1) (2, 2) Derived Data Types in the memory A (0, 0) A (0, 1) A A (0, 2) (1, 0) A (1, 1) A A (1, 2) (2, 0) A (2, 1) We can use, MPI_Type_vector to create vector (strided) data type. 15. 06. 2012 A (2, 2)

MPI_Type_vector double A[3][3]; A (0, 0) A (1, 0) A (2, 0) A A (0, 1) (0, 2) A A (1, 1) (1, 2) A A (2, 1) (2, 2) in the memory A (0, 0) A (0, 1) A A (0, 2) (1, 0) blocklength=1 stride=3 count=3 Derived Data Types 15. 06. 2012 A (1, 1) A A (1, 2) (2, 0) A (2, 1) A (2, 2)

MPI_Type_vector (C) blocklength=1 stride=3 count=3 MPI_Type_vector(int count, int blocklength, int stride, MPI_Datatype old_type, MPI_Datatype *new_type); Derived Data Types 15. 06. 2012

mpi_type_vector (Fortran) blocklength=1 stride=3 count=3 mpi_type_vector(count, blocklength, stride, MPI_Datatype old_type, MPI_Datatype new_type, ierr); Derived Data Types 15. 06. 2012

MPI_Type_vector (C) V(0) V(1) V(2) V(3) V(4) V(5) V(6) V(7) V(8) V(9) • Sending reds v. Type blocklength = 1 blocklength= ? 1 stride= ? 2 count= ? 3 stride =2 count =3 old_type = MPI_DOUBLE new_type = vtype MPI_Type_vector( count, blocklength, stride, old_type, &v. Type); MPI_Send(&V[2], 1, v. Type, dest, tag, MPI_COMM_WORLD); Derived Data Types 15. 06. 2012

mpi_type_vector (Fortran) V(0) V(1) V(2) V(3) V(4) V(5) V(6) V(7) V(8) V(9) • Sending reds type blocklength = 1 blocklength= ? 1 stride= ? 2 count= ? 3 stride =2 count =3 old_type = MPI_INTEGER new_type = type call mpi_type_vector(count, blocklength, stride, old_type, ierr); call mpi_send(V(2), 1, type, dest, tag, MPI_COMM_WORLD, ierr); Derived Data Types 15. 06. 2012

MPI_Type_indexed • Indexed constructor allows one to specify a noncontiguous data layout where displacements between successive blocks need not be equal. double A[4][4]; Derived Data Types 15. 06. 2012 A (0, 0) A (1, 0) A (2, 0) A (3, 0) A (0, 1) A (1, 1) A (2, 1) A (3, 1) A (0, 2) A (1, 2) A (2, 2) A (3, 2) A (0, 3) A (1, 3) A (2, 3) A (3, 3)

MPI_Type_indexed • Derived Data Types This allows: • Gathering of arbitrary entries from an array and sending them in one message, or • Receiving one message and scattering the received message entries into arbitrary locations in an array. 15. 06. 2012

MPI_Type_indexed (C) int MPI_Type_indexed( int count, int blocklength[], int indices[], MPI_Datatype old_type, MPI_Datatype *newtype ) • • • count : blocklength indices measured as number of blocks : number of elements in each block : displacement for each block, number of elements Derived Data Types 15. 06. 2012

mpi_type_indexed (Fortran) mpi_type_indexed( count, blocklength(), indices(), MPI_Datatype old_type, MPI_Datatype newtype, ierr ) • • • count : blocklength indices measured as number of blocks : number of elements in each block : displacement for each block, number of elements Derived Data Types 15. 06. 2012

MPI_Type_indexed old type blen[0]= 2 blen[1]= 3 blen[2]= 1 new type indices[0]=0 indices[1]=3 count= 3 Derived Data Types 15. 06. 2012 indices[2]=8

MPI_Type_indexed • • Suppose that, we have a matrix A(4 x 4) We want to send upper triangular matrix double A[4][4]; A (0, 0) A (1, 0) A (2, 0) A (3, 0) A (0, 1) A (1, 1) A (2, 1) A (3, 1) A (0, 2) A (1, 2) A (2, 2) A (3, 2) A (0, 3) A (1, 3) A (2, 3) A (3, 3) old type = MPI_DOUBLE new type = upper count = 4 blocklen[ ] = (4, 3, 2, 1) indices[ ] = (0, 5, 10, 15) MPI_Type_indexed(count, blocklen, indices, MPI_DOUBLE, upper ) Derived Data Types 15. 06. 2012

MPI_Type_commit • • Commits new datatype to the system. Required for all user constructed (derived) data types. int MPI_Type_commit( MPI_Datatype *datatype ) MPI_TYPE_COMMIT( MPI_Datatype datatype, ierr ) Derived Data Types 15. 06. 2012

MPI_Type_free • • Deallocates the specified data type object. Use of this routine is especially important to prevent memory exhaustion if many data type objects are created, as in a loop. int MPI_Type_free( MPI_Datatype *datatype ) MPI_TYPE_FREE( MPI_Datatype datatype, ierr ) Derived Data Types 15. 06. 2012

Packing Different Data • Sometimes, users need to send non-contiguous data in a single package. MPI allows them to • explicitly pack data into a contiguous buffer before sending it, and • • unpack it from a contiguous buffer after receiving. Several messages can be successively packed into one packing unit. • Derived Data Types 15. 06. 2012

MPI_Pack (C) data to be buffered int MPI_Pack ( void *packdata, number of input data items datatype of each input data item output buffer start size of buffer, in bytes current position in buffer, in bytes communicator for packed message Derived Data Types int count, MPI_Datatype datatype, void *buffer, int size, int *position, MPI_Comm comm ) 15. 06. 2012

mpi_pack (Fortran) data to be buffered MPI_PACK ( number of input data items datatype of each input data item output buffer start size of buffer, in bytes current position in buffer, in bytes communicator for packed message Derived Data Types packdata, count, MPI_Datatype datatype, buffer, size, position, MPI_Comm comm, ierr ) 15. 06. 2012

Packing Data char: (1 Byte) integer: (4 Byte) char c[25]: T o d a y int date[3]: 18 i s 7 a 2007 wo n d e r f u l integer array with 3 elements buffer Derived Data Types 15. 06. 2012 d a y ! char array with 25 element

Packing Data buffer T o d a y i s a wo n d e r f u l d a y ! 18 7 • At the beginning position=0 MPI_Pack(c, 25, MPI_CHAR, buffer, 37 &position, MPI_Comm comm ) • position value is updated by MPI_Pack as position=25 MPI_Pack(date, 3, MPI_INT, buffer, 37, &position, MPI_Comm comm ) • position value is updated by MPI_Pack as position=37 Derived Data Types 15. 06. 2012 2007

Packing Data buffer T o d a y i s a wo n d e r f u l d a y ! 18 7 2007 • At the beginning position=0 call MPI_PACK(c, 25, MPI_CHARACTER, buffer, 37 &position, MPI_Comm comm, ierr ) • position value is updated by MPI_Pack as position=25 MPI_Pack(date, 3, MPI_INTEGER, buffer, 37, &position, MPI_Comm comm, ierr ) • position value is updated by MPI_Pack as position=37 Derived Data Types 15. 06. 2012

Sending Packed Data Derived Data Types • • MPI_Send function used to send packed data, • • Size of the data must be specified in Bytes. Now, buffer size is 37 Bytes MPI_PACKED type is used as datatype. 15. 06. 2012

Sending Packed Data char: (1 Byte) buffer T o d a y i s a integer: (4 Byte) wo n d e r f u l MPI_Send( buffer, position, MPI_PACKED, dest, tag, MPI_Comm comm); Derived Data Types 15. 06. 2012 d a y ! 18 7 2007

Sending Packed Data char: (1 Byte) buffer T o d a y i s a integer: (4 Byte) wo n d e r f u l d a y ! call MPI_SEND( buffer, position, MPI_PACKED, dest, tag, MPI_Comm comm, ierr) Derived Data Types 15. 06. 2012 18 7 2007

Receiving Packed Data • • • Derived Data Types MPI_Recv function used to receive packed data, MPI_PACKED type is used as datatype. Size of the data must be specified in Bytes. 15. 06. 2012

Receiving Packed Data MPI_Recv( Rbuffer, 37, MPI_PACKED, source, tag, comm, &status); Rbuffer T o d a y Derived Data Types i s a wo n d e r f u l 15. 06. 2012 d a y ! 18 7 2007

Receiving Packed Data MPI_RECV( Rbuffer, 37, MPI_PACKED, source, tag, comm, status, ierr); Rbuffer T o d a y Derived Data Types i s a wo n d e r f u l 15. 06. 2012 d a y ! 18 7 2007

MPI_Unpack (C) input buffer start int MPI_Unpack(void *buffer, size of buffer, in bytes current position in buffer, in bytes output buffer start number of items to be unpacked datatype of each output data item communicator for packed message Derived Data Types int size, int *position, void *packdata, int count, MPI_Datatype datatype, MPI_Comm comm ) 15. 06. 2012

mpi_unpack (Fortran) input buffer start int MPI_UNPACK( buffer, size of buffer, in bytes size, current position in buffer, in bytes position, output buffer start packdata, number of items to be unpacked datatype of each output data item communicator for packed message Derived Data Types count, MPI_Datatype datatype, MPI_Comm comm, ierr ) 15. 06. 2012

MPI_Unpack • • Derived Data Types The output buffer can be any communication buffer allowed in MPI_Recv. The buffer is a contiguous storage area containing size bytes, starting at address buffer. The value of position is the first location in the buffer occupied by the packed message. position is incremented by the size of the packed message, so that the output value of position is the first location in the buffer after the locations occupied by the message that was unpacked. 15. 06. 2012

Unpacking Packed Data (C) MPI_Unpack( buffer, 37, &position, First value position=0 Updated to position=25 &Rc, 25, MPI_CHAR, MPI_Comm comm); char Rc[25]: T o d a y Derived Data Types i s a 15. 06. 2012 wo n d e r f u l d a y !

Unpacking Packed Data (Fortran) MPI_UNPACK( buffer, 37, position, First value position=0 Updated to position=25 Rc, 25, MPI_CHARACTER, MPI_Comm comm, ierr); char Rc[25]: T o d a y Derived Data Types i s a 15. 06. 2012 wo n d e r f u l d a y !

Unpacking Packed Data (C) MPI_Unpack( buffer, 37, &position, First value position=25 Updated to position=37 &Rdate, 3, MPI_INT, MPI_Comm comm); int Rdate[3]: Derived Data Types 18 15. 06. 2012 7 2007

Unpacking Packed Data (Fortran) MPI_UNPACK( buffer, 37, position, First value position=25 Updated to position=37 Rdate, 3, MPI_INTEGER, MPI_Comm comm, ierr); int Rdate[3]: Derived Data Types 18 15. 06. 2012 7 2007

Programming Activities Writing parallel MPI codes using following routines – Contiguous – Indexed – Vector – Pack-unpack Derived Data Types 15. 06. 2012