COSC 6374 Parallel Computation Message Passing Interface MPI I Introduction Edgar Gabriel Spring 2009 Edgar Gabriel Overview Distributed memory machines Basic principles of the Message Passing Interface MPI addressing startup data exchange process management communication COSC 6374 Parallel Computation Edgar Gabriel 1 Distributed memory machines local disks Compute node Processor Network card 2 Network card 1 Memory message passing network administrative network COSC 6374 Parallel Computation Edgar Gabriel Communication between different machines local disks Network card 2 Memory 1st Process Client Network card 1 Host Name and mypc my university edu Host Address 183 69 14 54 http request Internet Processor local disks Protocol webserver provider com 129 74 11 55 Network card 2 Memory Server Network card 1 2nd Process Processor COSC 6374 Parallel Computation Edgar Gabriel 2 Communication between different machines on the Internet Addressing hostname and or IP Address Communication based on protocols e g http or TCP IP Process start up every process application has to be started separately COSC 6374 Parallel Computation Edgar Gabriel The Message Passing universe Process start up Want to start n processes which shall work on the same problem mechanisms to start n processes provided by MPI library Addressing Every process has a unique identifier The value of the rank is between 0 and n 1 Communication MPI defines interfaces routines how to send data to a process and how to receive data from a process It does not specify a protocol COSC 6374 Parallel Computation Edgar Gabriel 3 History of MPI Until the early 90 s all vendors of parallel hardware had their own message passing library Some public domain message passing libraries available all of them being incompatible to each other High efforts for end users to move code from one architecture to another June 1994 Version 1 0 of MPI presented by the MPI Forum June 1995 Version 1 1 errata of MPI 1 0 1997 MPI 2 0 adding new functionality to MPI 2008 MPI 2 1 2009 MPI 2 2 and 3 0 in progress COSC 6374 Parallel Computation Edgar Gabriel Simple Example I MPI command to start process name of the application to start number of processes to be started Number of processes which have been started Rank of the 2nd process Rank of the 1st process COSC 6374 Parallel Computation Edgar Gabriel 4 Simple example II application t1 with rank 0 Parallel computer e g PC cluster PC of the end user mypc mpirun np 2 t1 application t1 with rank 1 mpirun starts the application t1 two times as specified with the np argument on two currently available processors of the parallel machine telling one process that his rank is 0 and the other that his rank is 1 COSC 6374 Parallel Computation Edgar Gabriel Simple Example III include mpi h int main int argc char argv int rank size MPI Init argc argv MPI Comm rank MPI COMM WORLD rank MPI Comm size MPI COMM WORLD size printf Mpi hi von node d job size d n rank size MPI Finalize return 0 COSC 6374 Parallel Computation Edgar Gabriel 5 MPI basics mpirun starts the required number of processes every process has a unique identifier rank which is between 0 and n 1 no identifiers are duplicate no identifiers are left out all processes which have been started by mpirun are organized in a process group communicator called MPI COMM WORLD MPI COMM WORLD is static number of processes can not change participating processes can not change COSC 6374 Parallel Computation Edgar Gabriel MPI basics II The rank of a process is always related to the process group e g a process is uniquely identified by a tuple rank process group A process can be part of the several groups i e a process has in each group a rank MPI COMM WORLD size 7 0123456 new process group size 5 0123 4 COSC 6374 Parallel Computation Edgar Gabriel 6 Simple Example IV Function returns the rank of a process within a process group Rank of a process within the process group MPI COMM WORLD snip MPI Comm rank MPI COMM WORLD rank MPI Comm size MPI COMM WORLD size snip Default process group containing all processes started by mpirun Number of processes in the process group MPI COMM WORLD Function returns the size of a process group COSC 6374 Parallel Computation Edgar Gabriel Simple Example V Function sets up parallel environment processes set up network connection to each other default process group MPI COMM WORLD is set up should be the first function executed in the application snip MPI Init argc argv snip MPI Finalize snip Function closes the parallel environment should be the last function called in the application might stop all processes COSC 6374 Parallel Computation Edgar Gabriel 7 Second example scalar product of two vectors two vectors are distributed on two processors each process holds half of the original vector Process with rank 0 a 0 1 2 b 0 1 2 Process with rank 1 a 2 b 2 COSC 6374 Parallel Computation Edgar Gabriel Second example II Logical Global view of the data compared to local view of the data a 0 1 2 alocal 0 a 0 alocal 1 a 1 alocal 2 a 2 alocal n a 1 2 Process with rank 1 a 2 alocal 0 a 2 alocal 1 a 1 2 alocal 2 a 2 2 Process with rank 0 alocal n a 1 COSC 6374 Parallel Computation Edgar Gabriel 8 Second example III Scalar product 1 s a i b i i 0 Parallel algorithm 2 1 s 1 a i b i a i b i i 0 2 1 i 2 2 1 alocal i blocal i alocal i blocal i i 0 i 0 144424443 1 44424443 rank 0 rank 1 requires communication between the processes COSC 6374 Parallel Computation Edgar Gabriel Second example IV include mpi h int main int i double double int argc char argv rank size a local N 2 b local N 2 s local s MPI Init argc argv MPI Comm rank MPI COMM WORLD rank MPI Comm size MPI COMM WORLD size s local 0 for i 0 i N 2 i s local s local a local i b local i COSC 6374 Parallel Computation Edgar Gabriel 9 Second example V if rank 0 Send the local result to rank 1 MPI Send s local 1 MPI DOUBLE 1 0 MPI COMM WORLD if rank 1 MPI Recv s 1 MPI DOUBLE 0 0 MPI COMM WORLD status Calculate global result s s s local COSC 6374 Parallel Computation Edgar Gabriel Second example VI Rank 1 holds the global result and sends it now to rank 0 if rank 0 MPI Recv s 1 MPI DOUBLE 1 1 MPI COMM WORLD status if rank 1 MPI Send s 1 MPI DOUBLE 0 1 MPI COMM WORLD Close the parallel environment MPI Finalize return 0 COSC 6374 Parallel Computation Edgar Gabriel 10 Sending Data Data element which shall be send Data Type of the element which shall be send Number of elements which shall be send snip MPI Send s