Parallel Processing (CS 730) Lecture 9: Advanced Point to Point CommunicationIntroductionBroadcast/Reduce RingBi-directional Broadcast RingAllgather RingAllGatherAllgather_ringSlide 8HypercubeBroadcast/ReduceAllgatherSlide 12Allgather_cubeSlide 14Buffering AssumptionSendRecvSendRecvReplaceNonblocking Send/RecvSlide 19Allgather_ring (Overlapped)AlltoAllAlltoAll (2 way)Communication ModesOct. 30, 2002 Parallel Processing 1Parallel Processing (CS 730) Lecture 9: Advanced Point to Point CommunicationJeremy R. Johnson*Parts of this lecture was derived from chapters 13 in PachecoOct. 30, 2002 Parallel Processing 2Introduction•Objective: To further examine message passing communication patterns.•Topics–Implementing Allgather•Ring•Hypercube–Non-blocking send/recv•MPI_Isend•MPI_Wait•MPI_TestOct. 30, 2002 Parallel Processing 3Broadcast/Reduce RingP3 P2P1P0P3 P2P1P0P3 P2P1P0P3 P2P1P0Oct. 30, 2002 Parallel Processing 4Bi-directional Broadcast RingP3 P2P1P0P3 P2P1P0P3 P2P1P0Oct. 30, 2002 Parallel Processing 5Allgather Ringx3 x2x0 x1P3 P2P1P0x2,x3 x1,x2x0,x3 x0,x1P3 P2P1P0x1,x2,x3x0,x2,x3P3 P2P1P0x0,x1,x2,x3P3 P2P1P0x0,x1,x2x0,x1,x3x0,x1,x2,x3x0,x1,x2,x3x0,x1,x2,x3Oct. 30, 2002 Parallel Processing 6AllGather•int MPI_AllGather(• void* send_data /* in */• int send_count /* in */• MPI_Datatype send_type /* in */• void* recv_data /* out */• int recv_count /* in */• MPI_Datatype recv_type /* in */• MPI_Comm communicator /* in */)Process 0Process 1Process 2Process 3x0x1x2x3Oct. 30, 2002 Parallel Processing 7Allgather_ringvoid Allgather_cube(float x[], int blocksize, float y[], MPI_Comm comm) {int i, p, my_rank;int successor, predecessor;int send_offset, recv_offset;MPI_Status status;MPI_Comm_size(comm, &p); MPI_Comm_Rank(comm, &my_rank);for (i=0; i < blocksize; i++) y[i + my_rank*blocksize] = x[i];successor = (my_rank + 1) % p;predecessor = (my_rank – 1 + p) % p;Oct. 30, 2002 Parallel Processing 8Allgather_ringfor (i=0; i < p-1; i++) { send_offset = ((my_rank – i + p) % p)*blocksize; recv_offset = ((my_rank –i – 1+p) % p)*blocksize; MPI_Send(y + send_offset,blocksize,MPI_FLOAT, successor, 0, comm); MPI_Recv(y + rec_offset,blocksize,MPI_FLOAT,predecessor,0, comm,&status); }}Oct. 30, 2002 Parallel Processing 9Hypercube•Graph (recursively defined)•n-dimensional cube has 2n nodes with each node connected to n vertices•Binary labels of adjacent nodes differ in one bit0000011011000100111101110001101101Oct. 30, 2002 Parallel Processing 10000001101100010011110111Broadcast/ReduceOct. 30, 2002 Parallel Processing 11000001101100010011110111AllgatherProcess000 x0001 x1010 x2011 x3100 x4101 x5110 x6111 x7DataProcess000 x0 x4001 x1 x5010 x2 x6011 x3 x7100 x0 x4101 x1 x5110 x2 x6111 x3 x7DataProcess000 x0 x2 x4 x6001 x1 x3 x5 x7010 x0 x2 x4 x6011 x1 x3 x5 x7100 x0 x2 x4 x6101 x1 x3 x5 x7110 x0 x2 X4 x6111 x1 x3 x5 x7DataOct. 30, 2002 Parallel Processing 12Allgather01 2345 6701 2345 6701 2345 6701 2345 67Oct. 30, 2002 Parallel Processing 13Allgather_cubevoid Allgather_cube(float x[], int blocksize, float y[], MPI_Comm comm) {int i, d, p, my_rank;unsigned eor_bit, and_bits;int stage, partner;MPI_Datatype hole_type;int send_offset, recv_offset;MPI_Status status;int log_base2(int p);MPI_Comm_size(comm, &p); MPI_Comm_Rank(comm, &my_rank);for (i=0; i < blocksize; i++) y[i + my_rank*blocksize] = x[i];d = log_base2(p); eor_bit = 1 << (d-1); and_bits = (1 << d) – 1;Oct. 30, 2002 Parallel Processing 14Allgather_cubefor (stage = 0; stage < d; stage++) { partner = my_rank ^ eor_bit; send_offset = (my_rank & and_bits) * blocksize; recv_offset = (partner & and_bits)*blocksize; MPI_Type_vector(1 << stage, blocksize, (1 << (d-stage)*blocksize, MPI_FLOAT,&hold_type); MPI_Type_commit(&hole_type); MPI_Send(y+send_offset,1,hold_type,partner, 0, comm); MPI_Recv(y+recv_offset,1,hold_type,partner, 0, comm,&status); MPI_Type_free(&hole_type); eor_bit = eor_bit >> 1; and_bits = and_bits >> 1;}Oct. 30, 2002 Parallel Processing 15Buffering Assumption•Previous code is not safe since it depends on sufficient system buffers being available so that deadlock does not occur.•SendRecv can be used to guarantee that deadlock does not occur.Oct. 30, 2002 Parallel Processing 16SendRecv•int MPI_Sendrecv(• void* send_buf /* in */,• int send_count /* in */,• MPI_Datatype send_type /* in */,• int dest /* in */,• int send_tag /* in */,• void* recv_buf /* out */,• int recv_count /* in */,• MPI_Datatype recv_type /* in */,• int source /* in */,• int recv_tag /* in */,• MPI_Comm communicator /* in */,• MPI_Status* status /* out */)Oct. 30, 2002 Parallel Processing 17SendRecvReplace•int MPI_Sendrecv_replace(• void* buffer /* in */,• int count /* in */,• MPI_Datatype datatype /* in */,• int dest /* in */,• int send_tag /* in */,• int source /* in */,• int recv_tag /* in */,• MPI_Comm communicator /* in */,• MPI_Status* status /* out */)Oct. 30, 2002 Parallel Processing 18Nonblocking Send/Recv•Allow overlap of communication and computation. Does not wait for buffer to be copied or receive to occur. •The communication is posted and can be tested later for completion•int MPI_Isend( /* Immediate */• void* buffer /* in */,• int count /* in */,• MPI_Datatype datatype /* in */,• int dest /* in */,• int tag /* in */,• MPI_Comm comm /* in */,• MPI_Request* request /* out */)Oct. 30, 2002 Parallel Processing 19Nonblocking Send/Recv•int MPI_Irecv(• void* buffer /* in */,• int count /* in */,• MPI_Datatype datatype /* in */,• int source /* in */,• int tag /* in */,• MPI_Comm comm /* in */,• MPI_Request* request /* out */)•int MPI_Wait(• MPI_Request* request /* in/out