IT Roles and Responsibilities: How Good is Good Enough?Agenda for TodayKiller Robot SummaryMatt’s Humble OpinionsTherac-25 Accidents (Basis of the Killer Robot Case)Therac-25 Accidents (What Happened?)Therac-25 Accidents (Example of Contributing UI Problems)Therac-25 Accidents (What Were the Problems?)Friendly-Fire Tragedy (Afghanistan)Critical ObservationPrinciple Actors in Software DesignObligations of the Software ProviderObligations of the Software BuyerObligations of the Software UserFinal ObservationConclusionCIF for Formal User Testing ReportsIT Roles and Responsibilities:How Good is Good Enough?IS 485, Professor Matt Thatcher2Agenda for TodayBrief review of the Case of the Killer RobotOverview of the Therac-25 accidentsDiscussion of “How Good Is Good Enough?”–what are our social responsibilities?3Killer Robot SummaryThe general problems–simple programming error–inadequate safety engineering and testing–poor HCI design–lax culture of safety in Silicon TechtronicsWhat would change in you replaced one of the characters with an “ethical” person?–would any of these problems have been solved?4Matt’s Humble OpinionsSource of the problems–economic incentives»time pressuresexclusive focus on meeting unrealistic deadlinesthere was no payoff to the dvmt team based on usability or safety measures valuing stock price over operator safety »cut corners  keep your job, challenge decisions  get fired–company culture»poor communication all along the company hierarchy»lots of unproductive, unresolved, and unaddressed conflict»inability to consider alternatives »choice of waterfall model instead of prototyping model as development methodology–inexperience and lack of critical skills in key positions»Johnson (hrdwr guy), Reynolds (data processing guy), Samuels (no exp in physics)Who is most responsible for:–setting appropriate economic incentives–creating an appropriate culture–putting people with the rights skills into the right jobs5Therac-25 Accidents(Basis of the Killer Robot Case)What was Therac-25?–released in 1983–computerized radiation therapy machine used to treat cancer patientsWho developed it?–Atomic Energy of Canada, Ltd and GCR (French-based company)What were the key advances of it over its predecessors (Therac-6 and Therac-20)?–move to more complete software-based control»faster set-up»safety checks were now controlled by software (instead of mechanical interlocks)6Therac-25 Accidents(What Happened?)Massively overdosed patients at least 6 times (3 died, 3 seriously disabled)–June 1985»Marietta, Ga (Linda Knight, 61)–July 1985»Hamilton, Ont (Donna Gartner, 40)–December 1985»Yakima, Wash (Janis Tilman)–March 1986»Tyler, Tx (Isaac Dahl, 33)–April 1986»Tyler, Tx (Daniel McCarthy)–January 1987»Yakima, Wash (Anders Engman)7Therac-25 Accidents(Example of Contributing UI Problems)The technician got the patient set up on the table, and went down the hall to start the treatment. She sat down at the terminal:»hit “x” to start the process she realized she made a mistake, since she needed to treat the patient with the electron beam, not the X-ray beam»hit the “Up” arrow, »selected the “Edit” command, »hit “e” for electron beam, and»hit “enter” (signifying she was ready to start treatment)»the system showed a “beam ready” prompt»she hit “b” to turn the beam therapy on»the system gave her an error message (Malfunction 54) »she overrode the error messageIt turns out that the UI showed that it was in electron mode but it was actually in a “hybrid” mode  delivered more than 125 times the normal dose to the patient8Therac-25 Accidents(What Were the Problems?)The Problems–simple programming errors–inadequate safety engineering»ignored the software risks (almost no unit or integration testing at all)»operators were told it was impossible to overdose a patient –poor HCI design–lax culture of safety in the manufacturing co.–problems were not reported quickly to manufacturer or FDA»prompted a 1990 federal law9Friendly-Fire Tragedy(Afghanistan)10Critical ObservationIf it is true that:–information technology affects societyAND–some choices in computing design are not completely constrained by mathematics, physics, chemistry, etc.THEN–designers, implementers, teachers, and managers of technology make choices that affect society11Principle Actors in Software DesignSoftware Provider–person or organization that creates the softwareSoftware Buyer–person or organization responsible for obtaining the software for its intended useSoftware User–person actually using the softwareSociety –people, other than providers, buyer, or users who can be affected by the software12Obligations of the Software ProviderThe Provider (itself)–profit and good reputation The Buyer–help buyer make an informed decision–set testing goals and meet them–provide warnings about untested areas of the software–inform user about testing results and shortcomings–provide a reasonable warranty on functionality and safetyThe User–education/training about use and limitations of software–provide technical support and maintenance–provide reasonable protections–provide clear instructions and user manualsBiggest Responsibility–answer the question “How Good is Good Enough”13Obligations of theSoftware BuyerThe Provider –respect copyrights and don’t steal–pay a fair price for the product–use the product for the correct purpose–don’t blame the provider for incorrect use –make users available for observations, testing, and evaluationThe Buyer (itself)–learn limitations of the software–perform systems acceptance testing (and audit testing)The User–ensure proper education/training is provided–ensure proper technical support and maintenance is provided–represent the user during development (be a communication link between user and developer)–make sure users are included in the design process–provide reasonable protections (and a safe working environment)Biggest Responsibility–make sure software is used for intended purpose (think about biometrics)14Obligations of theSoftware UserThe Provider –respect copyrights and don’t steal–read documentationThe Buyer–communicate problems and provide feedback (about training, UI, functionality, etc.)–ensure