COSC 6374 Parallel Computation Introduction to MPI III Process Grouping Edgar Gabriel Fall 2010 Edgar Gabriel Terminology I an MPI Group is the object describing the list of processes forming a logical entity a group has a size MPI Group size every process in the group has a unique rank between 0 and size of group 1 MPI Group rank a group is a local object and cannot be used for any communication Parallel Computation Edgar Gabriel 1 Terminology II An MPI Comm unicator is an object containing one or two groups of processes intra or intercommunicators topology information attributes A communicator has an error handler attached to it A communicator can have a name these slides focus on intra communicators i e the list of participating processes can be described by a single group Parallel Computation Edgar Gabriel Predefined communicators MPI COMM WORLD contains all processes started with mpirun mpiexec exist upon exiting MPI Init can not be modified freed etc MPI COMM SELF contains just the local process itself size is always 1 exist upon exiting MPI Init can not be modified freed etc Parallel Computation Edgar Gabriel 2 Creating new communicators All communicators in MPI 1 are derived from MPI COMM WORLD or MPI COMM SELF Creating and freeing a communicator is a collective operation all processes of the original communicator have to call the function with the same arguments Methods to create new communicators splitting the original communicator into n parts creating subgroups of the original communicator re ordering of processes based on topology information spawn new processes connect two applications and merge their communicators Parallel Computation Edgar Gabriel Splitting a communicator MPI Comm split MPI Comm comm int color int key MPI comm newcomm Partition comm into sub communicators all processes having the same color will be in the same subcommunicator order processes with the same color according to the key value if the key value is identical on all processes with the same color the same order for the processes will be used as in comm Parallel Computation Edgar Gabriel 3 Example for MPI Comm split I MPI Comm newcomm int color rank MPI Comm rank MPI COMM WORLD rank color rank 2 MPI Comm split MPI COMM WORLD color rank newcomm MPI Comm size newcomm size MPI Comm rank newcomm rank odd even splitting of processes a process can just be part of one of the generated communicators can not see the other communicators can not see how many communicators have been created Parallel Computation Edgar Gabriel Example for MPI Comm split II rank and size of the new communicator MPI COMM WORLD 0123456 newcomm color 0 size 4 0 newcomm color 1 size 3 1 2 3 0 1 2 Parallel Computation Edgar Gabriel 4 Invalid color in MPI Comm split If a process shall not be part of any of the resulting communicators set color to MPI UNDEFINED newcomm will be MPI COMM NULL MPI COMM NULL is an invalid communicator any function taking a communicator as an argument will return an error or abort if you pass MPI COMM NULL i e even MPI Comm size and MPI Comm rank or MPI Comm free Parallel Computation Edgar Gabriel Modifying the group of processes comm MPI Comm group MPI Group incl excl MPI Comm create newcomm original communicator Extract the group of processes from the original communicator Modify the group Create new communicator based on the modified group new communicator Parallel Computation Edgar Gabriel 5 Extracting the group of processes MPI Comm group MPI Comm comm MPI Group group with comm original communicator group the group object describing the list of participating processes in comm Parallel Computation Edgar Gabriel Modifying groups I MPI Group incl MPI Group MPI Group MPI Group excl MPI Group MPI Group group int cnt int ranks newgroup group int cnt int ranks newgroup with group the original group object containing the list of participating processes ranks array of integers containing the ranks of the processes in group which shall be included in the new group for MPI Group incl excluded from the original group for MPI Group excl newgroup resulting group Parallel Computation Edgar Gabriel 6 Modifying groups II for more group constructors see also MPI Group range incl MPI Group range excl MPI Group difference MPI Group intersection MPI Group union Parallel Computation Edgar Gabriel Creating a new communicator based on a group MPI Comm create MPI Comm comm MPI Group newgroup MPI Comm newcomm with comm original communicator group the group object describing the list of processes for the new communicator newcomm resulting communicator Note newcomm is always a subset of comm you can generate one communicator at a time in contrary to MPI Comm split list of arguments has to be identical on all processes of comm newcomm will be MPI COMM NULL for processes which have been excluded not included in newgroup Parallel Computation Edgar Gabriel 7 Example for MPI Comm create generate a communicator which contains only the first four processes and the last process of the original communicator MPI COMM WORLD 0123456 newcomm size 5 0123 4 Parallel Computation Edgar Gabriel 1st Option using MPI Group incl MPI Comm newcomm MPI Group group newgroup int color size ranks 5 cnt MPI Comm size MPI COMM WORLD size cnt 5 ranks 0 0 ranks 1 1 ranks 2 2 ranks 3 3 ranks 4 size 1 MPI Comm group MPI COMM WORLD group MPI Group incl group cnt ranks newgroup MPI Comm create comm newgroup newcomm if newcomm MPI COMM NULL MPI Comm rank newcomm nrank MPI Comm free newcomm MPI Group free newgroup MPI Group free group Parallel Computation Edgar Gabriel 8 2nd Option using MPI Group excl MPI Comm newcomm MPI Group group newgroup int color size ranks cnt NOTE Assuming that size 5 ranks is large enough etc MPI Comm size MPI COMM WORLD size cnt 0 for i 4 i size 1 i ranks cnt i MPI Comm group MPI COMM WORLD group MPI Group excl group cnt 1 ranks newgroup MPI Comm create comm newgroup newcomm if newcomm MPI COMM NULL MPI Comm rank newcomm nrank MPI Comm free newcomm MPI Group free newgroup Parallel Computation MPI Group free group Edgar Gabriel Freeing groups and communicators MPI Comm free MPI Comm comm MPI Group free MPI Group group return MPI COMM NULL respectively MPI GROUP NULL MPI Comm free is a collective function MPI Group free is a local function Parallel Computation Edgar Gabriel 9 Topology information in communicators Some application scenarios require not only to know who is part of a communicator but also how they are organized Called