1Software Engineering:Software Testing and VerificationProf. Aiken CS 295 Lecture 1 1Lecture 1CS295Course Staff• Instructor–Alex Aiken•Teaching AssistantProf. Aiken CS 295 Lecture 1 2g– Steven EliaCourse Communication•Allclass materials will be on the web– Lecture notes, handouts, papers to read, etc.• Read the web site for announcementsProf. Aiken CS 295 Lecture 1 3• Ask questions in the newsgroup– Preferred for most questions over emailCourse Structure•Lectures– Course taught from notes– Focus on programmer’s view of software engineering• Technology over business issuesProf. Aiken CS 295 Lecture 1 4• Homeworks– For those taking the class for 3 credits– Use a verification/testing tool, report findings• In-class midterm, take-home finalWhy Take This Class?• Be a better programmer• See the future–This is where the field is headedProf. Aiken CS 295 Lecture 1 5This is where the field is headed• Prepare for research in the area• For the war storiesThis LectureThe course is about improving software quality.Prof. Aiken CS 295 Lecture 1 6Claim: This is central to software engineering.2The Ariane Rocket DisasterProf. Aiken CS 295 Lecture 1 7Post Mortem• Failure due to unhandled floating point exception•CostProf. Aiken CS 295 Lecture 1 8Cost– $100M’s for loss of mission– Multi-year setback to the Ariane programEast Coast USA Prof. Aiken CS 295 Lecture 1 9East Coast USA: 2003 BlackoutProf. Aiken CS 295 Lecture 1 10Post Mortem• Local failure rapidly cascaded through grid• Major contributing cause was unnoticed crash of automated alarm systemsProf. Aiken CS 295 Lecture 1 11of automated alarm systems• 10M’s of people affectedMars Polar LanderProf. Aiken CS 295 Lecture 1 123Post Mortem•A units problem– Caller expected values in inches/feet– Callee assumed values in meters– Essentially, a type errorProf. Aiken CS 295 Lecture 1 13• Total loss of $100M missionSecurity• Often exploit bugs in programs• Widespread problem, getting worse–Code RedCode Red– Titan Rain– Moonlight Maze–Operation Orchard– Stuxnet WormProf. Aiken CS 295 Lecture 1 14The Big QuestionHow do we know the code works? Prof. Aiken CS 295 Lecture 1 15Look first at how software is developed . . .Software Development TodayDecision MakerWhy do we have this structure?Prof. Aiken CS 295 Lecture 1 16Programmer TesterTypical Scenario (1)Decision Maker“OK, calm down. We’ll slip the schedule. Try again.”Prof. Aiken CS 295 Lecture 1 17Programmer Tester“I’m done.”“It doesn’t #$%& compile!”Typical Scenario (2)Decision Maker“Now remember, we’re all in this together. Try again.”Prof. Aiken CS 295 Lecture 1 18Programmer Tester“I’m done.”“It doesn’t install!”4Typical Scenario (3)Decision Maker“Let’s have a meeting to straighten out the spec.”Prof. Aiken CS 295 Lecture 1 19Programmer Tester“I’m done.”“It does the wrong thing in half the tests.”“No, half of your tests are wrong!”Typical Scenario (4)Decision Maker“Try again, but pleasehurry up!”Prof. Aiken CS 295 Lecture 1 20Programmer Tester“I’m done.”“It still fails some tests we agreed on.”Typical Scenario (5)Decision Maker“Oops, the world has changed. Here’s the new spec.”Prof. Aiken CS 295 Lecture 1 21Programmer Tester“I’m done.”“Yes, it’s done!”Software Development TodayDecision MakerWhy do we have this structure?Prof. Aiken CS 295 Lecture 1 22Programmer TesterKey Assumptions• Independent development and testing• Specifications must be explicit• Specifications evolve•Resources are finiteProf. Aiken CS 295 Lecture 1 23Resources are finite• Human organizations need decision makers• Examine each of these separatelyIndependent Testing and Development• Testing is basic to every engineering discipline– Design a drug– Manufacture an airplane–Etc.Prof. Aiken CS 295 Lecture 1 24•Why?– Because our ability to predict how our creations will behave is imperfect– We need to check our work, because we will make mistakes5Independent Testing and Development of Software• In what way is software different?• Folklore:– “Programmers are optimists”–The implication is that programmers make poor testersProf. Aiken CS 295 Lecture 1 25ppgp– Economics: “Programming costs more than testing”– The implication is that programming is a higher-skill profession• How valid is the folklore, and how much is due to the current state of the art in testing?Explicit Specifications• Software involves multiple people– At least a programmer and a user– But usually multiple programmers, testers, etc.Prof. Aiken CS 295 Lecture 1 26• Any team effort requires mutual understanding of the goal– A specification– Otherwise, team members inevitably have different goals in mindSpecifications Change•Why?• Many software systems are truly “new”– Differ from all that went before in some wayProf. Aiken CS 295 Lecture 1 27– Initial specification will change as problems are discovered and solved•The world is changing–What people want– The components you build on (e.g., the OS version)Software Specifications• Software specifications are usually–in prose–imprecise–out of dateProf. Aiken CS 295 Lecture 1 28• Current state of specification is not conducive to automation– Not consumable by tools– Without a spec, there is nothing to checkFinite Resources• Organizations make trade-offs– Not all goals can be achieved– Because resources are finite• $’s express relative costs among goalsProf. Aiken CS 295 Lecture 1 29– Goals that are hard to quantify pose a problem– E.g., correctness, completeness“We have 2 months, 5 programmers, and 2 testers. Here is a priority list of features. A feature is finished when it passes all of the tests for that feature; a programmer does not move on to a new feature until all higher priority features are finished or assigned to other programmers. We start now and ship whatever features are finished in 60 days.”Reality• Many proposals for improving software quality• But the world tests–> 50% of the cost of software developmentProf. Aiken CS 295 Lecture 1 30> 50% of the cost of software development•Conclusion– Testing is important– Take a closer look at testing practice . . .6The Purpose of Testing•Two purposes:• Find bugs– Find important bugsProf. Aiken CS 295