1Alex Aiken Lecture 6 CS295 1Cooperative Bug IsolationLecture 6CS295Alex Aiken Lecture 6 CS295 2Outline• Something different today . . .• Look at monitoring deployed code– Collecting information from actual user runs– Gives a true picture of reality• Two themes– How do we get the data?– What do we do with it?Alex Aiken Lecture 6 CS295 3Build and MonitorAlex Aiken Lecture 6 CS295 4The Goal: Analyze Reality• Where is the black box for software?– Historically, some efforts, but spotty– Now it’s really happening: crash reporting systems• Actual runs are a vast resource– Number of real runs >> number of testing runs– And the real-world executions are most important• This lecture: post-deployment bug huntingAlex Aiken Lecture 6 CS295 5Engineering Constraints• Big systems– Millions of lines of code– Mix of controlled, uncontrolled code– Threads• Remote monitoring– Limited disk & network bandwidth• Incomplete information– Limit performance overhead– Privacy and securityAlex Aiken Lecture 6 CS295 6The Approach1. Guess “potentially interesting” behaviors– Compile-time instrumentation2. Collect sparse, fair subset of these behaviors– Generic sampling transformation– Feedback profile + outcome label3. Find behavioral changes in good/bad runs– Statistical debugging2Alex Aiken Lecture 6 CS295 7Bug Isolation ArchitectureProgramSourceCompilerSamplerGuessesShippingApplicationProfile& J/LStatisticalDebuggingTop bugs withlikely causesAlex Aiken Lecture 6 CS295 8Our Model of BehaviorWe assume any interesting behavior is expressible as a predicate P on program state at a particular program point.Observation of behavior = observing PAlex Aiken Lecture 6 CS295 9Branches Are Interestingif (p) …else …Alex Aiken Lecture 6 CS295 10Branch Predicate Counts++branch_17[!!p];if (p) …else …• Predicates are folded down into counts• C idiom: !!p ensures subscript is 0 or 1Alex Aiken Lecture 6 CS295 11Return Values Are Interestingn = fprintf(…);Alex Aiken Lecture 6 CS295 12Returned Value Predicate Countsn = fprintf(…);++call_41[(n==0)+(n>=0)];• Track predicates: n < 0, n == 0, n > 03Alex Aiken Lecture 6 CS295 13Scalar Relationshipsint i, j, k;…i = …;The relationship of ito other integer-valued variables in scope after the assignment is potentially interesting . . .Alex Aiken Lecture 6 CS295 14Pair Relationship Predicate Countsint i, j, k;…i = …;++pair_6[(i==j)+(i>=j)];++pair_7[(i==k)+(i>=k)];++pair_8[(i==5)+(i>=5)];Test i against all other constants & variables in scope.Is i < j, i = j, or i > j?Alex Aiken Lecture 6 CS295 15Summarization and Reporting• Instrument the program with predicates– We have a variety of instrumentation schemes• Feedback report is:– Vector of predicate counters– Success/failure outcome label• No time dimension, for good or ill• Still quite a lot to measure– What about performance?0 0 4 0 1 . . .P1 P2 P3 P4 P5 . . .Alex Aiken Lecture 6 CS295 16Sampling• Decide to examine or ignore each site…– Randomly– Independently– Dynamically• Why?– Fairness– We need accurate picture of rare events.Alex Aiken Lecture 6 CS295 17Problematic Approaches• Sample every kth predicate– Violates independence• Use clock interrupt– Not enough context– Not very portable• Toss a coin at each instrumentation site– Too slowAlex Aiken Lecture 6 CS295 18Amortized Coin Tossing• Observation– Samples are rare, say 1/100– Amortize cost by predicting time until next sample• Randomized global countdown– Small countdown  upcoming sample• Selected from geometric distribution– Inter-arrival time for biased coin toss– How many tails before next head?4Alex Aiken Lecture 6 CS295 19Geometric Distribution 11log1,0log1DrandnextD= mean of distribution = expected sample densityAlex Aiken Lecture 6 CS295 20Weighing Acyclic Regions• Acyclic code region has:• a finite number of paths• a finite max number of instrumentation sites executed12 111234Alex Aiken Lecture 6 CS295 21Weighing Acyclic Regions• Clone acyclic regions– “Fast” variant– “Slow” variant• Choose at run time based on countdown to next sample>4?Alex Aiken Lecture 6 CS295 22Summary: Feedback Reports• Subset of dynamic behavior– Counts of true/false predicate observations– Sampling gives low overhead• Often unmeasurable at 1/100– Success/failure label for entire run• Certain of what we did observe– But may have missed some events• Given enough runs, samples ≈ reality– Common events seen most often– Rare events seen at proportionate rateAlex Aiken Lecture 6 CS295 23Bug Isolation ArchitectureProgramSourceCompilerSamplerGuessesShippingApplicationProfile& J/LStatisticalDebuggingTop bugs withlikely causesAlex Aiken Lecture 6 CS295 24Find Causes of Bugs• We gather information about many predicates.– 298,482 for BC• Most of these are not predictive of anything.• How do we find the useful predicates?5Alex Aiken Lecture 6 CS295 25Finding Causes of BugsHow likely is failure when P is observed true?F(P) = # failing runs where P observed trueS(P) = # successful runs where P observed trueFailure(P) =F(P) + S(P)F(P)Alex Aiken Lecture 6 CS295 26Not Enough . . .Failure(f == NULL) = 1.0Failure(x == 0) = 1.0if (f == NULL) {x = 0;*f;}• Predicate x == 0 is an innocent bystander– Program is already doomedAlex Aiken Lecture 6 CS295 27ContextWhat is the background chance of failure, regardless of P’s value?F(P observed) = # failing runs observing P S(P observed) = # successful runs observing PContext(P) =F(P observed) + S(P observed)F(P observed)Alex Aiken Lecture 6 CS295 28A Useful MeasureDoes the predicate being true increase the chance of failure over the background rate?Increase(P) = Failure(P) – Context(P)A form of likelihood ratio testing . . .Alex Aiken Lecture 6 CS295 29if (f == NULL) {x = 0;*f;}Increase() Works . . .Increase(f == NULL) = 1.0Increase(x == 0) = 0.0Alex Aiken Lecture 6 CS295 30A First Algorithm1. Discard predicates having Increase(P) ≤ 0– E.g. dead, invariant, bystander predicates– Exact value is sensitive to small F(P)– Use lower bound of 95% confidence interval2. Sort remaining predicates by Increase(P)– Again, use 95% lower bound– Likely causes