COSC 6374 Parallel Computation Message Passing Interface MPI II Advanced point to point operations Edgar Gabriel Fall 2011 Edgar Gabriel Overview Point to point taxonomy and available functions What is the status of a message Non blocking operations Parallel Computation Edgar Gabriel 1 What you ve learned so far Six MPI functions are sufficient for programming a distributed system memory machine MPI Init int argc char argv MPI Finalize MPI Comm rank MPI Comm comm int rank MPI Comm size MPI Comm comm int size MPI Send void buf int count MPI Datatype dat int dest int tag MPI Comm comm MPI Recv void buf int count MPI Datatype dat int source int tag MPI Comm comm MPI Status status Parallel Computation Edgar Gabriel Point to point operations Data exchange between two processes both processes are actively participating in the data exchange two sided communication Large set of functions defined in MPI 1 50 Blocking Non blocking Persistent Standard MPI Send MPI Isend MPI Send init Buffered MPI Bsend MPI Ibsend MPI Bsend init Ready MPI Rsend MPI Irsend MPI Rsend init Synchronous MPI Ssend MPI Issend MPI Ssend init Parallel Computation Edgar Gabriel 2 A message contains of the data which is to be sent from the sender to the receiver described by the beginning of the buffer a data type the number of elements of the data type the message header message envelope rank of the sender process rank of the receiver process the communicator a tag Parallel Computation Edgar Gabriel Rules for point to point operations Reliability MPI guarantees that no message gets lost Non overtaking rule MPI guarantees that two messages posted from process A to process B arrive in the same order as posted Message based paradigm MPI specifies that a single message cannot be received with more than one Recv operation in contrary to sockets Message Recv buffer1 Recv buffer2 Message in the Recv buffers if rank 0 MPI Send buf 4 if rank 1 MPI Recv buf 3 MPI Recv buf 3 1 Parallel Computation Edgar Gabriel 3 Message matching I How does the receiver know whether the message which he just received is the message for which he was waiting the sender of the arriving message has to match the sender of the expected message the tag of the arriving message has to match the tag of the expected message the communicator of the arriving message has to match the communicator of the expected messag Parallel Computation Edgar Gabriel Message matching II What happens if the length of the arriving message does not match the length of the expected message the length of the message is not used for matching if the received message is shorter than the expected message no problems the received message is longer than the expected message an error code MPI ERR TRUNC will be returned or your application will be aborted or your application will deadlock or your application writes a core dump Parallel Computation Edgar Gabriel 4 Message matching III Example 1 correct example if rank 0 MPI Send buf 3 MPI INT 1 1 MPI COMM WORLD else if rank 1 MPI Recv buf 5 MPI INT 0 1 MPI COMM WORLD status Message Recv buffer untouched elements in the recv buffer Message in the Recv buffer Parallel Computation Edgar Gabriel Message matching IV Example 2 erroneous example if rank 0 MPI Send buf 5 MPI INT 1 1 MPI COMM WORLD else if rank 1 MPI Recv buf 3 MPI INT 0 1 MPI COMM WORLD status Message Recv buffer potentially writing over the end of the recv buffer Message in the Recv buffer Parallel Computation Edgar Gabriel 5 Deadlock I Question how can two processes safely exchange data at the same time Possibility 1 Process 0 Process 1 MPI Send buf MPI Recv buf MPI Send buf MPI Recv buf can deadlock depending on the message length and the capability of the hardware MPI library to buffer messages Parallel Computation Edgar Gabriel Deadlock II Possibility 2 re order MPI functions on one process Process 0 Process 1 MPI Recv rbuf MPI Send buf MPI Send buf MPI Recv rbuf Other possibilities asynchronous communication use buffered send MPI Bsend use MPI Sendrecv shown later not shown here not shown here Parallel Computation Edgar Gabriel 6 Example Implementation of a ring using Send Recv Rank 0 starts the ring MPI Comm rank comm rank MPI Comm size comm size if rank 0 MPI Send buf 1 MPI INT rank 1 MPI Recv buf 1 MPI INT size 1 else if rank size 1 MPI Recv buf 1 MPI INT rank 1 MPI Send buf 1 MPI INT 0 else MPI Recv buf 1 MPI INT rank 1 MPI Send buf 1 MPI INT rank 1 1 comm 1 comm status 1 comm status 1 comm 1 comm status 1 comm Parallel Computation Edgar Gabriel Wildcards Question can I use wildcards for the arguments in Send Recv Answer for Send for Recv tag source communicator no yes MPI ANY TAG yes MPI ANY SOURCE no Parallel Computation Edgar Gabriel 7 Status of a message I the MPI status contains directly accessible information who sent the message what was the tag what is the error code of the message and indirectly accessible information through function calls how long is the message has the message bin cancelled Parallel Computation Edgar Gabriel Status of a message II usage in C MPI Status status MPI Recv buf cnt MPI INT status directly access source tag and error src status MPI SOURCE tag status MPI TAG err status MPI ERROR determine message length and whether it has been cancelled MPI Get count status MPI INT rcnt MPI Test cancelled status flag Parallel Computation Edgar Gabriel 8 Status of a message IV If you are not interested in the status you can pass MPI STATUS NULL MPI STATUSES NULL to MPI Recv and all other MPI functions which return a status Parallel Computation Edgar Gabriel Non blocking operations I A regular MPI Send returns when the data is safely stored away A regular MPI Recv returns when the data is fully available in the receive buffer Non blocking operations initiate the Send and Receive operations but do not wait for its completion Functions which check or wait for completion of an initiated communication have to be called explicitly Since the functions initiating communication return immediately all MPI functions have an I prefix e g MPI Isend or MPI Irecv Parallel Computation Edgar Gabriel 9 Non blocking operations II MPI Isend void buf int cnt MPI Datatype dat int dest int tag MPI Comm comm MPI Request req MPI Irecv void buf int cnt MPI Datatype dat int src int tag MPI Comm comm MPI Request reqs Parallel Computation Edgar Gabriel Non blocking operations III After initiating a non blocking communication it is not allowed to touch modify the