COSC 6374 Parallel Computation Parallel Measures and Debugging MPI application Edgar Gabriel Fall 2011 Edgar Gabriel Performance Metrics I Speedup how much faster does a problem run on p processors compared to 1 processor Ttotal 1 Ttotal p Optimal S p p linear speedup S p Parallel Efficiency Speedup normalized by the number of processors S p p Optimal E p 1 0 E p Parallel Computation Edgar Gabriel 1 Performance Metrics II Example Application A takes 35 min on a single processor 27 on two processors and 18 on 4 processors S 2 35 1 29 27 E 2 1 29 0 645 2 S 4 35 1 94 18 E 4 1 94 0 485 4 Parallel Computation Edgar Gabriel Amdahl s Law I Basic idea most applications have a small sequential fraction which limits the speedup Ttotal Tsequential T parallel fTTotal 1 f TTotal f fraction of the code which can only be executed sequentially S p Ttotal 1 1 1 f 1 f f Ttotal 1 f p p Parallel Computation Edgar Gabriel 2 Example for Amdahl s Law 60 00 50 00 40 00 f 0 f 0 05 30 00 f 0 1 f 0 2 20 00 10 00 0 00 1 2 4 8 12 16 20 24 28 32 36 40 44 48 52 56 60 64 Parallel Computation Edgar Gabriel Amdahl s Law II Amdahl s Law assumes that the problem size is constant In most applications the sequential part is independent of the problem size while the part which can be executed in parallel is not Parallel Computation Edgar Gabriel 3 Performance Metrics III Scaleup ratio of the execution time of a problem of size n on 1 processor to the execution time of the same problem of size n p on p processors Sc p Ttotal 1 n Ttotal p n p Optimally execution time remains constant e g Ttotal p n Ttotal 2 p 2n Parallel Computation Edgar Gabriel Timing functions in MPI I Can be done e g by gettimeofday MPI functions provided double MPI Wtime void double MPI Wtick void MPI Wtime returns a floating point number of seconds representing elapsed wall clock time since some time in the past The times returned are local to the node that called them There is no requirement that different nodes return the same time MPI Wtick returns the resolution of MPI WTIME in seconds Parallel Computation Edgar Gabriel 4 Timing functions in MPI II double starttime endtime elapsedtime starttime MPI Wtime do some incredibly complex calculations endtime MPI Wtime elapsedtime endtime starttime Timing rules Make sure you time longer than the clock resolution e g on a regular LINUX box clock resolution is 10ms Rule of thumb 100 times the clock resolution Parallel Computation Edgar Gabriel Debugging sequential applications Several ways how to debug a sequential application printf statements in the source code Works works reliably painful to remove afterwards assert statements check for a certain value of a variable If the expression is false the application aborts only active if the macro NDEBUG is defined Setting in the source define NDEBUG 1 Compiling with the flag DNEDBUG 1 include assert h void open record char record name assert record name NULL Parallel Computation Edgar Gabriel 5 Using a debugger For a source file to be visible in the debugger you have to compile the source code with the g option e g gabriel salmon mpicc g o test test c Avoid using optimization flags such as O3 when you would like to debug the code Two types of debugger Command line debugger such as gdb Graphical debuggers such as ddd which is a GUI to gdb Parallel Computation Edgar Gabriel Load application into the debugger Start app Debugger points to the problem Show source code of app Show the value of a variable when the problem occurred Parallel Computation Edgar Gabriel 12 6 gdb commands Setting breakpoints debugger stops execution at the specified line Example gdb break errexample c 10 gdb break myfunc Stepping through the source code gdb next gdb step skips subroutines functions enters subroutines functions Continue execution not step by step anymore gdb cont Quit debugger gdb quit Parallel Computation Edgar Gabriel Parallel Computation Edgar Gabriel 7 Debugging a parallel application Some debuggers for parallel applications available e g totalview ddt Unfortunately expensive products You can still use printf and assert Output from several processes will be mixed you should put the rank of the process in front of each printf statement gdb or ddd still useable You have to choose which process you would like to debug Please be aware that ddd or gdb can only see processes on the local machine Parallel Computation Edgar Gabriel Debugging a parallel application II Hints for parallel debugging Try to find the lowest number of processes for which the problem still occurs Try to execute the application on a single node If the problem does not show up on a single node you will have to run the application on multiple nodes and login to the node where the problem occurs Introduce a sleep statement in your application to have time to attach with a debugger Parallel Computation Edgar Gabriel 8 Attaching to a process Menu File Bullet attach to processes Choose the PID which you would like to debug Parallel Computation Edgar Gabriel Debugging parallel applications III Some MPI libraries support the startup of a debugger in the mpirun command including Open MPI mpirun np 2 ddd colltest only if all processes are running locally Starts one ddd session per process Not useful for large numbers of processes Parallel Computation Edgar Gabriel 9