CS 350, slide set 7Testing: basic terms - 1Testing: basic terms - 2Failure Probabilities - 1Failure Probabilities - 2Failure Probability CommentsN-Version SoftwareN-Version Software - 2Failure Probabilities for N-Version SystemOracle Problem - 1Oracle Problem - 2Team Project: Software Reliability EstimatorVersion 1 Design ReviewVersion 2 DesignVersion 2 Design ReviewReadingGroup MeetingsOutlineA development strategySteps – Planning and scheduleSteps – Conceptual designIdentify major risksProductivity comes from reuse (but from your own code!)Strategy scripts: overviewDevelopment Strategy - stepsPossible development strategy ideas:Development plan - 1Development plan - 2Planned-value exampleTSPi Planning OverviewSTRAT formSize summary (SUMS) formTASK formSample SCHEDULE formForm WEEK – for weekly reportsSUMQ form (Quality plan) -1SUMQ form (Quality plan) -2SUMQ form (Quality plan) -3SUMQ form (Quality plan) -4Plan Summary (SUMP)TSPi Development plan scriptPlan steps - 1Plan steps - 2Example formsConceptual designProductsTracking the workReuse commentsCS 350, slide set 7M. OverstreetOld Dominion UniversityFall 2005Testing: basic terms - 1Unit testing: testing of single compilation unit; often requires driver & stubsStress testing: test code under extreme conditionsSystem testing: put all of the pieces together and test everythingTesting: basic terms - 2Beta testing: testing by “special” customers who understand errors are likelyAcceptance testing: testing on customer’s site Reliability testing: testing to measure reliabilityRegression testing: testing ensure that changes did not break old featuresFailure Probabilities - 1In software reliability world, failures are treated as "random" events (even though they aren't)Why can code sometimes behave differently even using the same data?Each programs has "input space":The type of data the program will receive when running, including the fact that it will see some inputs frequently, others rarely.Failure Probabilities - 2Then probability of program failure is based on concept:Randomly pick a data value from the program's input space (where selecting particular values reflects how frequently they will be seen when the program really runs).Observe whether or not the program failsDo this for all values in input space (properly weighted).Pr(failure with a randomly selected input) is estimated by running many (perhaps millions or billions) test casesPr(failure) ≈ (number failures) / (number tests)Failure Probability CommentsSimple concept. But involves several potentially hard problems:What is the distribution of inputs the program will really encounter in use?Input space usually too large to run program with all possible inputsHow do you tell when the program fails (assuming it doesn't crash)?Typical industry interest is how often software will fail when used by their customers. This is a similar idea.Depends on both customer’s frequency of use and their typical input dataN-Version SoftwarePurpose: initially, increase software reliabilityOne idea:Develop one requirements documentGive copies to several different independent development groupsEach group develops complete system as if they were the only developer•No info exchange among groups!!Run all “identical” systems:•Give each version the exactly the same input•Wait for each to produce required output•Compare each outputs & take majority as the correct outputQuestion: does this work?N-Version Software - 2Statistical concept: event independenceKnowing one event occurred doesn’t change probability of another event.Independence  Pr(A^B)=Pr(A)Pr(B)•Often, this is not trueKey Question:Do separately developed versions really fail independently?Or if one fails, does that make it more likely that another will fail also?Failure Probabilities for N-Version SystemAssume a 3-version systemAssume each version has a failure probability of 10-3Assume independent failuresHow can this system fail?If at least two versions get the wrong answer.This happens with probability 10-3 ·10-3Oracle Problem - 1When testing software, how do you detect all erroneous outputs?Unsolved SE problem for many applicationsOracle Problem - 2Federal Aviation Administration (FAA) specifies that the failure rate for commercial aircraft of less than 10-9.How many test cases should you run to determine this?Generally considered a statistical problem: if the software is given a "random" input, what is the likelihood of the program producing incorrect output (or crashing or looping forever)?How do you detect wrong answers?Remember, for 10-9, need to run lots of test casesTeam Project: Software Reliability EstimatorDesign (Version 1):Input desired number of tests: nRepeat n times // typical n: 100,000 to // 1,000,000,000Generate random input from program’s input spaceRun the programCheck the answerIf incorrect, increment error countOutput failure probability estimate: (error count)/nVersion 1 Design ReviewWhat’s wrong with this design?(In your analysis, you should mention the Oracle Problem)Version 2 DesignAssume we have several versions of the same program, all “reasonably” correctInput desired number of tests, say nRepeat n timesGenerate random input from program’s input spaceRun each version with this inputCompare all answersIf disagreement, increment error countOutput failure probability estimate:(error count)/nVersion 2 Design ReviewQuestions:How hard is it to run several programs from another program?What happens if one of the programs being tested fails?•Say it generates a wrong answer?•Say it goes into an infinite loop•Say it generates a run-time error and is terminated by the operating system?How do you generate random inputs?•What if inputs are text?•What if inputs are numeric?•Say it generates a wrong answer?ReadingTSP text, Ch. 4, 5Chapters 11 – 15, depending on your role.Group MeetingsGroup meetings in recitationUsed to meet, set group goals, set up project notebookAgenda:Be prepared to described your background to groupBe prepared to state your goals for group projectOutlineMore formsDevelopment strategyDevelopment planA development strategyLots of different approaches to software developmentDevelop entire systemDevelop system in small