Complexity revisited: learning from failures6.033 in one slideToday: Why do systems fail anyway?Too many objectivesSlide 6Complexity: no hard edgeLearn from failure!Keep digging principleSlide 13United Airlines/UnivacCONFIRMAdvanced Automation SystemLondon Ambulance ServiceIBM Workplace OSMany moreRecurring problemsFighting back: control noveltyFighting back: adopt sweeping simplificationsFighting back: design for iteration, iterate the designFighting back: find bad ideas fastThe design loopFighting back: find flaws fastFighting back: conceptual integritySummaryAdmonitionClose the 6.033 design loopComplexity revisited:learning from failuresFrans Kaashoek and Robert MorrisLec 26 --- Last one!5/13/09Credit: Jerry Saltzer6.033 in one slide•Client/server•RPC•File abstraction•Virtual memory•Threads•Coordination•Protocol layering•Routing protocols•Reliable packet delivery•Names•Atomicity•Transactions•Replication•Sign/Verify•Encrypt/Decrypt•AuthorizationCase studies of successful systems: UNIX, X Windows,MapReduce, Ethernet, Internet, WWW, RAID, DNS, ….Principles: End-to-end argument, Modularity, …Today:Why do systems fail anyway?•Complexity has no hard edge•Learning from failures: common problems•Fighting back: avoiding the problems•Final admonitionToo many objectives•Ease of use•Availability•Scalability•Flexibility•Mobility•Security•Networked•Maintainability•Performance•Cheap•….But no systematic methods to synthesizesystems to meet objectivesMany objectives+Few Methods+High d(technology)/dt=High risk of failureThe tarpit[F. Brooks, Mythical Man Month]Complexity: no hard edge•When is it too much?objectives/features/performancecomplexityLearn from failure!“The concept of failure is central to design process, and it is by thinking in terms of obviating failure that successful designs are achieved…”[Henry Petroski]Keep digging principle•Complex systems systems fail for complex reasons–Find the cause …–Find a second cause …–Keep looking …–Find the mind-set. [Petroski, Design Paradigms]Try 1: Meidum (52 angle)Try 2: Dashur/Bent (52 to 43.5 angle)Try 3: Red pyramid (right angle: 43)Pharaoh Sneferu’s Pyramid projectUnited Airlines/Univac•Automated reservations, ticketing, flight scheduling, fuel delivery, kitchens, and general administration•Started 1966, target 1968, scrapped 1970, spent $50M•Second-system effect (First: SABRE)(Burroughs/TWA repeat)CONFIRM•Hilton, Marriott, Budget, American Airlines•Linked air + car + hotel reservations•Started 1988, scrapped 1992, $125M•Second system•DB integration problems•DB not crash recoverable•Bad-news diode[Communications of the ACM 1994]Advanced Automation System•US Federal Aviation Administration•To replace 1972 computerized system•Real-time nation-wide route planning •Started 1982, scrapped 1994 ($6B)•Big ambitions•Changing ideas about UI•12 years -> evolving requirements, tech•12 years -> culture of not finishing•Big -> congressional meddlingLondon Ambulance Service•Ambulance dispatching•Started 1991, scrapped in 1992–20 lives lost in 2 days•No testing/overlap with old system•Required big changes in procedure•Users not consulted during design•Unrealistic schedule (5 months)•Perhaps first of kind, no experience[Report of the Inquiry Into The London Ambulance Service 1993]IBM Workplace OS•One microkernel O/S for all IBM products–PDAs / desktop / servers / supercomputers–“personalities” for OS/2, AIX, OS/400, Windows –x86, new PowerPC, ARM•Started in 1991, scrapped 1996 ($2B)•factoring out common services too hard•PPC needed new OS, new OS needed PPC–but PPC was late, buggy, and slow•IBM division per personality, bad cooperation[Fleisch HotOS 1997]Many more•Portland, Oregan, Water Bureau, 30M, 2002•Washington D.C., Payroll system, 34M 2002•Southwick air traffic control system $1.6B 2002•Sobey’s grocery inventory, 50M, 2002•King’s County financial mgmt system, 38M, 2000)•Australian submarine control system, 100M, 1999•California lottery system, 52M•Hamburg police computer system, 70M, 1998•Kuala Lumpur total airport management system, $200M, 1998•UK Dept. of Employment tracking, $72M, 1994•Bank of America Masternet accounting system, $83M, 1988,•FBI virtual case, 2004.•FBI Sentinel case management software, 2006.Recurring problems•Excessive generality and ambition•Second-system effect•Bad modularity •Inexperience (or ignoring experienced advice)•Bad-news diode•Mythical Man MonthFighting back: control novelty•Only one big new idea at a time•Re-use existing components•Why it’s hard to say “no”–Second-system effect–Technology is better–Idea worked in isolation–Marketing pressure•Hire strong, knowledgeable managementFighting back: adopt sweeping simplifications•Processor, Memory, Communication•Dedicated servers•Best-effort network•End-to-end error control•Atomic transactions•Authentication, confidentialityFighting back:design for iteration, iterate the design•Get something simple working soon–Find out what the real problems are•Structure project to allow feedback–e.g. deploy in phases•Series of small projects“Every successful complex system is found to have evolved from a successful simple system” – John GallFighting back: find bad ideas fast•Question requirements–“And ferry itself across the Atlantic” [LHX light attack helicoper]•Try ideas out, but don’t hesitate to scrap•Have a design loopThe design loop•Find flaws fast!InitialdesignDraft design coding testing deployedmonthsminhoursdaysweeksFighting back: find flaws fast•Plan and simulate–Boeing 777 CAD, F-16 flight sim•Design reviews, coding reviews, regression tests, daily/hourly builds, performance measurements•Design the feedback system:–Alpha and beta tests–Incentives, not penalties, for reporting errorsFighting back:conceptual integrity•One mind controls the design–Macintosh, Visicalc, UNIX, Linux•Good abstractions/modules reduce O(n2) effects–In human organization as much as software–Small focused teams •Good esthetics yields more successful systems–Parsimonious, Orthogonal, Elegant, Readable, …•Best designers much better than average–Find and exploit themSummary•Principles that help avoid failure–Limit novelty–Adopt sweeping simplifications–Get something simple working
View Full Document
Unlocking...