Collective MPI Communication MA471 Lecture 5 9 10 2003 http www unix mcs anl gov mpi www 2 Excellent website for MPI command syntax available at Today When all the processes want to send receive or both 9 10 2003 3 To make things easier you should copy down the list of names and global IDs which I will read out Please write down your global ID making sure you know which is which In return I will give you a global ID For the following exercise I need to know each student s first name Names 9 10 2003 MPI Bcast 4 9 10 2003 5 Alternatively he can start off a chain of communications which can thought of in a tree like sequence He could tell everyone individually But clearly while he communicates with each person in turn everyone else is twiddling their thumbs If Chris is determined to tell everyone else something important then there are a number of ways the information can be disseminated MPI Bcast 9 10 2003 0 0 1 0 2 2 3 0 4 4 5 4 6 6 7 Example tree communication for Bcast 6 9 10 2003 Now I need a volunteer to build a Bcast tree Using this tree we will try a Bcast We will construct a tree for the global group Other tree constructions are possible In this case there are 8 processes so the minimum number of communications is 7 Comments 7 9 10 2003 8 Finalize 7 Return to step 4 6 If you have received message and have no leaf nodes put your left hand up 5 Processes on this level communicate to their right leaf node 4 Advance to next level of tree 3 Process 0 sends message to right leaf node 2 Barrier 1 Init Global Exercise Mimic MPI Bcast 8 Process root 9 10 2003 A2 Data Bcast A2 A2 A2 A2 A2 Diagramatic Description of MPI Bcast Comments Observations 9 choice integer handle integer handle 9 10 2003 http www unix mcs anl gov mpi www www3 MPI Bcast html Input output Parameters buffer starting address of buffer count number of entries in buffer datatype data type of buffer root rank of broadcast root comm communicator 10 Synopsis int MPI Bcast void buffer int count MPI Datatype datatype int root MPI Comm comm Broadcasts a message from the process with rank root to all other processes of the group MPI Bcast 9 10 2003 3 Process root will send the same message to all other process 11 2 If a process does not join in the Bcast then the rest Of the processes will wait 1 all processes must make the call to MPI Bcast I e they all need to know that it is going to happen Note Notes on MPI Bcast 9 10 2003 MPI Allreduce 12 9 10 2003 MPI ALLREDUCE combines values from all processes and distributes the result back to all processes One way is to use MPI ALLREDUCE 13 Then how can you all find out what the average suckiness rating is Imagine you each give the last exercise a grade out of 10 as to how much it sucked 10 real bad MPI ALLREDUCE Process Op A A4 9 10 2003 14 http www ncsa uiuc edu UserInfo Resources Hardware CommonDoc MessPass MPIColl html For example Op A could be Op A A0 A1 A2 A3 A4 Op A Op A A3 A2 Op A A1 Allreduce Op A A0 Data Diagramatic Description of MPI Allreduce 9 10 2003 Output Parameter recvbuf starting address of receive buffer choice Input Parameters sendbuf starting address of send buffer choice count number of elements in send buffer integer datatype data type of elements of send buffer handle op operation handle comm communicator handle 15 Synopsis int MPI Allreduce void sendbuf void recvbuf int count MPI Datatype dataty MPI Op op MPI Comm comm MPI Allreduce Combines values from all processes and distribute the result back to all processes 9 10 2003 MPI MAX MPI MIN MPI SUM MPI PROD returns the maximum returns the minimum returns the sum returns the product Some of the different operations available Syntax MPI ALLREDUCE cont 16 9 10 2003 convert sum to average ratingave ratingsum Nprocs 17 all processes send their rating and receive the sum of all ratings ierr MPI Allreduce rating ratingsum ratinglen MPI INT MPI SUM MPI COMM WORLD find number of procsses in world ierr MPI Comm Size MPI COMM WORLD Nprocs there is only one entry in the rating data ratinglen 1 everyone has their own opinion rating some number int Nprocs rating ratinglen ratingsum ierr double ratingave MPI Allreduce example 9 10 2003 MPI Alltoall 18 9 10 2003 Now suppose that you all have something to say to each other Further you all have the same length message to send We can think of the messages as a matrix of messages MPI Alltoall 19 Process A11 A21 A31 A41 A10 A20 A30 A40 A42 A32 A22 A12 A02 A43 A33 A23 A13 A03 A44 A34 A24 A14 A04 Alltoall A04 A03 A14 A13 A12 A11 A01 A02 A10 A00 A24 A23 A22 A21 A20 Data A34 A33 A32 A31 A30 A44 A43 A42 A41 A40 9 10 2003 20 http www ncsa uiuc edu UserInfo Resources Hardware CommonDoc MessPass MPIColl html A01 A00 Data Diagramatic Description of MPI Alltoall Process DID ROT BOB TOP Proc 2 Proc 3 9 10 2003 DOG RED Proc 1 BAD BIG Proc 0 MAN YOU WAS CAT INPUT to MPI ALLTOALL HAT EAT SLY SAD 21 DID ROT Proc 1 BOB TOP Proc 0 Proc 2 Proc 3 9 10 2003 DOG RED Proc 1 BAD BIG Proc 0 Proc 2 MAN YOU WAS CAT Proc 3 HAT EAT SLY SAD 22 WAS SLY CAT SAD Proc 2 Proc 3 EAT YOU DID BOB HAT MAN ROT TOP 9 10 2003 In essence the Alltoall has transposed the data DOG BAD Proc 1 RED BIG Proc 0 OUTPUT from MPI Alltoall 23 9 10 2003 DO NOT USE COMMUNICATION TREE 1 Init 2 Barrier 3 Recall your global ID 4 Write down Nprocs 3 letter words 5 Send your first word to process 0 6 Receive proc 0 s ID th word 7 Send your second word to process 1 8 Receive proc 1 s ID th word 9 10 Send your last word to process Nprocs 1 11 Receive proc Nprocs 1 ID th word 12 Barrier 13 Finalize Global Exercise mimic Alltoall 24 9 10 2003 25 If one of these is frequently necessary in a computation you should probably reconsider the methods you are using and their appropriateness for parallel computation This is one of the least desirable approaches to parallelism it implies that all processes are tightly coupled and have to share data When everyone has something to say at the same time then communication becomes a real bottle neck Now that should have been really tough Comments Observations choice 9 10 2003 http www unix mcs anl gov mpi www www3 MPI Alltoall html Output Parameter recvbuf address of receive buffer Input …