COSC 6374 Parallel Computation Performance Analysis of MPI applications Edgar Gabriel Fall 2010 Edgar Gabriel Performance Analysis of parallel applications Useful to see Who is communicating with whom and how much How are the messages and the processes synchronized General approach link a tracing library to your application which records per process and per MPI function when the function has been started and how long it took two time stamps Some additional information e g for Send operations source and destination process message length communicator etc Parallel Computation Edgar Gabriel 1 Performance Analysis of parallel applications MPI Profiling Interface MPI defines that each function has to be available not only with the prefix MPI but also with the prefix PMPI e g instead of MPI Send you could also call PMPI Send same functionality just different name Tracing libraries implement the MPI function and call internally the PMPI function for the real operation Parallel Computation Edgar Gabriel Profiling Library int MPI Send void buf int cnt MPI Datatype dat int dest int tag MPI Comm comm t1 PMPI Wtime PMPI Send buf cnt dat dest tag comm t2 PMPI Wtime elem new elem t elem cnt cnt elem datatype dat elem time t2 t1 append elem Parallel Computation Edgar Gabriel 2 Profiling interface Example an application calls MPI Send Usual scenario the function MPI Send is implemented in the MPI library Profiling scenario the function MPI Send is implemented in the tracing library The MPI Send function of the tracing library calls PMPI Send PMPI Send is implemented in the MPI library Parallel Computation Edgar Gabriel Profiling interface II Question How does the application now which MPI Send function to use e g from MPI library or from tracing library Answer You tell the application at link time Example mpicc o test test c is in reality for Open MPI cc o test test c I opt OpenMPI 1 4 1 include L opt OpenMPI 1 4 1 lib lmpi lopen rte lopen pal Parallel Computation Edgar Gabriel 3 Profiling interface III The I option with cc tells the compiler where to look for header files The L option tells the linker where to look for libraries Linker searcher the directories in the order as specified by the user If a function has been found in a library the linker does not continue searching in the following libraries usually Thus cc o test test c I opt OpenMPI 1 4 1 include L opt OpenMPI 1 4 1 lib lmpi lopen rte lopen pal will use MPI Send from lmpi because that s where the linker finds the function first Parallel Computation Edgar Gabriel Profiling interface IV However cc o test test c I opt OpenMPI 1 4 1 include L home gabriel MPE lib llmpe lmpe L opt OpenMPI 1 4 1 lib lmpi lopen rte lopen pal will use MPI Send from llmpe because that s where the linker finds the function first but PMPI Send from lmpi because that s where the linker finds this function first Parallel Computation Edgar Gabriel 4 Available performance analysis tools Commercials among others Vampir requires licence and vampirtrace free Intel Trace Analyzer Public domain among others MPE and jumpshot Paraver and mpilog Scalasca and epilog TAU Parallel Computation Edgar Gabriel Hints when using MPE and jumpshot When using MPE you can use instead of the long commandline a wrapper script called mpecc mpef77 e g mpecc mpilog o test mpe test c Run the application as usually mpirun np 2 test After the run there is file called test mpe clog2 in the same directory The viewer jumpshot unfortunately needs a different file format so you have to convert it with clog2TOslog2 test mpe clog2 This generates a file called test mpe slog2 You can load an slog2 file into jumpshot Parallel Computation Edgar Gabriel 5 Hints when using Vampirtrace If the Open MPI library is compiled accordingly you can simply use mpicc vt o test vt test c run the application resulting trace file can be loaded directly into Vampir Parallel Computation Edgar Gabriel Performance analysis with VAMPIR I Application Molecular dynamics simulation running with 16 processes Code of Dr Cheung Parallel Computation Edgar Gabriel 6 Performance analysis with VAMPIR II Timeline view see which process is at which time in which MPI function Parallel Computation Edgar Gabriel Performance analysis with VAMPIR III Timeline view see which process is at which time in which MPI function Parallel Computation Edgar Gabriel 7 Performance analysis with VAMPIR IV Message statistics view how much data is being sent between processes Parallel Computation Edgar Gabriel Performance analysis with VAMPIR V Message statistics view how many messages is being sent between processes Parallel Computation Edgar Gabriel 8 Performance analysis with VAMPIR VI Length statistics what is the most commonly used message length Parallel Computation Edgar Gabriel Performance analysis with VAMPIR VII Function statistics how often is an MPI function used left and how much time is spent in the function right Parallel Computation Edgar Gabriel 9