1Testing, Profiling and InstrumentationCS 2172Testing, Profiling, and Instrumentation• How do you know if your program is correct? Will it ever core dump? Does it ever produce the wrong answer?– Testing• How do you know what your program is doing? How fast is your program? Why is it slow for one input but not for another? Does it have a memory leak?– Timing– Profiling– Instrumentation3Program Verification• How do you know if your program is correct? Can you prove that it is correct? Can you prove properties of the code?– e.g., it terminatesProgramCheckersymtable.cRight/WrongSpecification?4Program Testing• Convince yourself that your program probably works TestProgramsymtable.cProbablyRight/WrongSpecificationHow do you write a test program?How do you write a test program?5Test Programs• Properties of a good test program Tests boundary conditions Exercise as much code as possible Produce output that is known to be right/wrongHow do you achieve all three properties?How do you achieve all three properties?6Test Boundary Conditions• Most bugs occur at boundary conditions What is the most common boundary condition bug?• What are the boundary conditions of this code?int i;char s[MAXLINE];for ( i=0; (s[i] = getchar()) != ‘\n’&& i < MAXLINE-1; i++ );s[--i] = ‘\0’;• Boundary conditions Input starts with \n End of file7Test Boundary Condition, cont’d• Rewrite the codefor ( i=0; i<MAXLINE-1; i++)if ((s[i] = getchar()) == ‘\n’)break;S[i] = ‘\0’;• Another boundary condition: EOFfor ( i=0; i<MAXLINE-1; i++)if ((s[i] = getchar()) == ‘\n’ || s[i] == EOF)break;S[i] = ‘\0’;• What are other boundary conditions? Nearly full Exactly full Over full8Test As Your Write Code• Recall using “assert” in previous lecture• Check pre- and post-conditions for each function Boundary conditions• Check invariants• Check error returns• What is the typical percentage of code doing error-checking?9Systematic Testing• Test plan Unit tests (for each module) System tests• Design test cases Know what input gives what output, according to the spec Check properties of output Compare independent implementations What legal inputs to test– Boundary conditions and “inductions”– Multi-dimensional inputs and combinations• Assembler: instructions, comments, directives• Numerical program: operations, legal combinations What illegal inputs to test– Common illegal inputs– Possible security holes– Multi-dimensional illegal inputs10A Test Case Example• “De-comment” test Empty comments Test single line comment Test very long line Multiple line comment Test many lines Nested comment String literal in comment Character literal in comment Comment in string literal Comment in character literal Unterminated comment . . . 11Test Automation• Automation can provide better test coverage• Test program Client code to test modules Scripts to test inputs and compare outputs• QA test is an iterative process Initial automated test program or scripts Test simple parts first Unit tests before system tests Add tests as new cases created• Regression test Test all cases to compare the new version with the previous one A bug fix often create new bugs in a large software system• What tests cannot be done automatically?12Stress Tests• Motivations Use computer to generate inputs to test High-volume tests often find bugs• What to generate Very long inputs Random inputs (binary vs. ASCII) Fault injection• How much test Exercise all data paths Test all error conditions13Who Test What• Implementers White-box testing Pros: An implementer knows all data paths Cons: influenced by how code is designed/written • Quality Assurance (QA) engineers Black-box testing Pros: No knowledge about the implementation Cons: Unlikely to test all data paths• Customers Field test Pros: Unexpected ways of using the software, “debug” specs Cons: No enough cases 14Timing, Profiling, and Instrumentation• How do you know what your code is doing? How slow is it?– How long does it take for certain types of inputs? Where is it slow?– Which code is being executed most? Why am I running out of memory?– Where is the memory going?– Are there leaks? Why is it slow?– How imbalanced is my binary tree?15Timing• Most shells provide tool to time program execution e.g., bash “time” commandbash> tail -1000 /usr/lib/dict/words > input.txtbash> time sort5.pixie < input.txt > output.txtreal 0m12.977suser 0m12.860ssys 0m0.010s16Timing• Most operating systems provide a way to get the time e.g., UNIX “gettimeofday” command#include <sys/time.h>struct timeval start_time, end_time; gettimeofday(&start_time, NULL);<execute some code here>gettimeofday(&end_time, NULL);float seconds = end_time.tv_sec - start_time.tv_sec + 1.0E-6F * (end_time.tv_usec - start_time.tv_usec);17Timing• Some CPU provides access to CPU “ticks”unsigned long long int getCPUTicks(void){unsigned long long int x;asm volatile (".byte 0x0f, 0x31":"=A" (x));return x;}18Profiling• Gather statistics about your program’s execution How much time did execution of a function take? How many times was a particular function called? How many times was a particular line of code executed? Which lines of code used the most time? • Most compilers come with profilers e.g., pixie and prof19Profiling with gcc+gprof• Apparently, prof doesn’t work with gcc, must use gprofPROFFLAGS = -Wall -ansi -pedantic -O4 -NDEBUG -pgCFLAGS= ${PROFFLAGS}profile: player testinput-player MIN <testinputgprof player >profileplayer: player.c minimax.c gamestate.c ...gcc ${CFLAGS} player.c minimax.c ...minus sign means “keep going even if errors”20Profiled execution% make profilegcc -Wall -O4 -DNDEBUG -pg -o player . . .player MIN <testinput512 11 10 9 8 7|-------------------------|| 4 4 4 5 5 5 || 0 1 || 4 4 4 4 4 0 ||-------------------------|0 1 2 3 4 5MIN player reports invalid move by MAX playermake: *** [profile] Error 1 (ignored)gprof player >profile21Format of gprof profilecalled/total parents index %time self descendents called+self name indexcalled/total children<spontaneous>[1] 59.7 12.97 0.00
View Full Document
Unlocking...