COSC 6374 Parallel Computation Introduction to MPI IV Derived Data Types Edgar Gabriel Fall 2010 Edgar Gabriel Derived Datatypes Basic idea interface to describe memory layout of user data structures e g a structure in C typedef struct char a int b double c mystruct Memory layout Parallel Computation Edgar Gabriel 1 Derived Datatype examples E g describing a column or a row of a matrix Memory layout in C Memory layout in Fortran Parallel Computation Edgar Gabriel How to describe non contiguous data structures typedef struct char a int b double c mystruct baseaddr gap1 address1 address2 gap2 using a list I O interface e g address size baseaddr sizeof char address1 sizeof int address2 sizeof double or baseaddr sizeof char baseaddr gap1 sizeof int baseaddr gap2 sizeof double Parallel Computation Edgar Gabriel 2 or in MPI terminology a list of address count datatype sequences baseaddr 1 MPI CHAR baseaddr gap1 1 MPI INT baseaddr gap2 1 MPI DOUBLE leading to the following interface MPI Type struct int count int blocklength MPI Aint displacements MPI Datatype datatypes MPI Datatype newtype MPI Type create struct int count int blocklength MPI Aint displacements MPI Datatype datatypes MPI Datatype newtype Parallel Computation Edgar Gabriel MPI Type struct MPI Type create struct MPI Aint Is an MPI Address integer An integer being able to store a memory address Displacements are considered to be relative offsets displacement 0 0 in most cases Displacements are not required to be positive distinct or in increasing order How to determine the address of an element MPI Address void element MPI Aint address MPI Get address void element MPI Aint address Parallel Computation Edgar Gabriel 3 Addresses in MPI Why not use the operator in C ANSI C does NOT require that the value of the pointer returned by is the absolute address of the object Might lead to problems in segmented memory space Usually not a problem In Fortran all data elements passed to a single MPI Type struct call have to be in the same common block Parallel Computation Edgar Gabriel Type map vs Type signature Type signature is the sequence of basic datatypes used in a derived datatype e g typesig mystruct char int double Type map is sequence of basic datatypes sequence of displacements typemap mystruct char 0 int 8 double 16 Type matching rule of MPI type signature of sender and receiver has to match Including the count argument in Send and Recv operation e g unroll the description Receiver must not define overlapping datatypes The message need not fill the whole receive buffer Parallel Computation Edgar Gabriel 4 Committing and freeing a datatype If you want to use a datatype for communication or in an MPI I O operation you have to commit it first MPI Type commit MPI Datatype datatype Need not commit a datatype if just used to create more complex derived datatypes MPI Type free MPI Datatype datatype It is illegal to free any predefined datatypes Parallel Computation Edgar Gabriel Our previous example looks like follows baseaddr address1 address2 mystruct mydata MPI Address mydata baseaddr MPI Address mydata b addr1 MPI Address mydata c addr2 displ 0 0 displ 1 addr1 baseaddr displ 2 addr2 baseaddr dtype 0 MPI CHAR dtype 1 MPI INT dtype 2 MPI DOUBLE blength 0 1 blength 1 1 blength 2 1 MPI Type struct 3 blength displ dtype newtype MPI Type commit newtype Parallel Computation Edgar Gabriel 5 Basically we are done With MPI Type struct we can describe any pattern in the memory Why other MPI datatype constructors Because description of some datatypes can become rather complex For convenience Parallel Computation Edgar Gabriel MPI Type contiguous MPI Type contiguous int count MPI Datatype datatype MPI Datatype newtype count elements of the same datatype forming a contiguous chunk in the memory int myvec 4 MPI Type contiguous 4 MPI INT mybrandnewdatatype MPI Type commit mybrandnewdatatype MPI Send myvec 1 mybrandnewdatatype Input datatype can be a derived datatype End of one element of the derived datatype has to be exactly at the beginning of the next element of the derived datatype Parallel Computation Edgar Gabriel 6 MPI Type vector MPI Type vector int count int blocklength int stride MPI Datatype datatype MPI Datatype newtype count blocks of blocklength elements of the same datatype Between the start of each block there are stride elements of the same datatype stride 3 blocklength 2 count 3 Parallel Computation Edgar Gabriel Example using MPI Type vector Describe a column of a 2 D matrix in C dtype stride blength count MPI DOUBLE 8 1 8 MPI Type vector count blength stride dtype newtype MPI Type commit newtype Which column you are really sending depends on the pointer which you pass to the according MPI Send routine Parallel Computation Edgar Gabriel 7 MPI Type hvector MPI Type hvector int count int blocklength MPI Aint stride MPI Datatype datatype MPI Datatype newtype MPI Type create hvector int count int blocklength MPI Aint stride MPI Datatype datatype MPI Datatype newtype Identical to MPI Type vector except that the stride is given in bytes rather than in number of elements Parallel Computation Edgar Gabriel MPI Type indexed MPI Type indexed int count int blocklengths int displacements MPI Datatype datatype MPI Datatype newtype The number of elements per block do not have to be identical displacements gives the distance from the base to the beginning of the block in multiples of the used datatype count 3 blocklengths 0 2 displacements 0 0 blocklengths 1 1 displacements 1 3 blocklengths 2 4 displacements 2 5 Parallel Computation Edgar Gabriel 8 MPI Type hindexed MPI Type hindexed int count int blocklengths MPI Aint displacements MPI Datatype datatype MPI Datatype newtype MPI Type create hindexed int count int blocklengths MPI Aint displacements MPI Datatype datatype MPI Datatype newtype Identical to MPI Type indexed except that the displacements are given in bytes and not in multiples of the datatypes Parallel Computation Edgar Gabriel Duplicating a datatype MPI Type dup MPI Datatype datatype MPI Datatype newtype Mainly useful for library developers e g datatype ownership The new datatype has the same committed state as the previous datatype If datatype has already been committed newtype is committed as well Parallel Computation Edgar Gabriel 9 MPI Type create subarray MPI Type create subarray int ndims int sizes int subsizes int starts int order MPI Datatype datatype MPI Datatype newtype Define sub matrices of n dimensional data