Johns Hopkins EN 600 445 - The Therac 25 A case study in safety failure

Unformatted text preview:

The Therac 25A case study in safety failure• Radiation therapy machine• “The most serious computer-related accidents to date”• People were killed • Reference:Nancy Leveson and Clark Turner, “The Investigation of the Therac-25 Accidents”, Computer, 26, 7 (July 1993) pp 18-41.Therac 25 Background• Medical linear accelerator developed by Atomic Energy of Canada, Ltd. in mid-1970s• Delivers 25 MeV photons or electrons of various energies• Controlled by PDP-11 minicomputer• Software responsible for safety• Software adapted from earlier Therac-6 & Therac20 systems, which had hardware interlocks for safetyThe Therac 25Therac 25 TurntableTherac 25 Turntable• Electron mode• 5-25 MEV• Magnets spread beam• Ion chamber monitor• X-ray mode• 25 MEV electrons hit target• “Beam flattener” attenuates• 100x beam current• Ion chamber monitor• Field-light mode• No current• Mirror & light used to check alignment• No ion chamber (since not treating)Therac 25 Turntable• Computer adjusts turntable position• Microswitches detect turntable setting• 3-bit binary code used to encode turntable setting• Software checks replace hardware interlocksTherac 25 Software Development• Evolved from Therac 6 system (1972-1976)• Incorporated some Therac 20 code, as well• Written in PDP-11 assembler• Custom operating system• Little documentation during development• Minimal unit and software testing• Q/A testing was 2700 hours of use as integrated system• Programmer left AECL in 1986, little information available about his backgroundTherac 25 Software Functions• Monitors machine status• Sets up machine for treatment• Turns beam on and off in response to operator• Monitors interlocks• If fault, either prevents treatment start or causes a pause/suspendTherac 25 Software Structure• Critical tasks:– Treatment monitor– Servo– Housekeeping• Non-critical tasks:– Checksum– Keyboard– Calibration– etc.• Concurrent access to shared memory, “test” and “set” of variables not indivisible, race conditionsOperator Procedures• Position patient on table• Manually set treatment field size and gantry rotation; attach accessories• Leave room• Use VT-100 console to enter patient data, dose data, etc.• (System compares manual settings with system values)• If “verified”, operator can start machine• Else must re-enter dataOperator Screen LayoutOperator Procedures• Complaint– Re-entering all that data manually is very tedious• Response– Set things up so that “carriage return” copies previous data for entry– Series of carriage returns effectively permits fast re-entry of unchanged parts of dataOperator Procedures• Error Conditions– “Treatment suspend” requires complete machine reset– “Treatment pause” can be resumed if operator types “P” at console– Machine insists on reset after 5 “P”s– Malfunction messages fairly common & usually do not affect safety• Error Messages– Cryptic– Some were of the form “Malfunction NN”FDA Comment on ManualAccident History• 11 Therac 25’s installed (5 US, 6 Canada)• Six accidents involving massive overdoses between 1985 and 1987• Machines recalled in 1987• Related problems in Therac 20 discovered later but hardware interlocks prevented injuriesE.g., East Texas, March 1986• History of 500 patients treated successfully• Prescribed: 22MeV electrons, 180 rads• Operator selected x-rays by mistake, used cursor keys to change to electrons• Machine tripped with “Malfunction 54”– Documentation explains this is “dose input 2” error• Operator proceeded; machine tripped againE.g., East Texas, March 1986• Patient felt something wrong on first jolt, tried to get up• Video/audio links to room not functioning• Patient felt jolt on arm while getting up, pounded on door• Treatment cancelled for day• Calibration checks seemed normal• Later found patient had gotten 16,500-25,000rads over 1 cm square• Patient eventually died after 5 monthsE.g., East Texas, March 1986• AECL engineers could not replicate a Malfunction 54• AECL home office engineer said machine could not overdose patient• AECL suggested patient got an electric shock• No grounding problems found• Machine returned to service April 7, 1986East Texas/ April 11,1986• Prescription 10 MeV, area 7 x 10 cm• Operator used cursor keys to change x-rays to electrons, saw “beam ready”, and turned machine on• Loud noise, shutdown, malfunction 54• Patient in great pain• Patient died three weeks laterEast Texas/ April 11,1986• Machine taken out of service• ETCC eventually reproduced malfunction 54– Data entry speed critical factor– Observed 4000 rad dose• AECL later measured 25,000 rads• In lawsuit, earlier “cursor up” problems reported, which AECL believed to have been fixedYakima Valley, January 1987• Plan: 2 film verification exposures (3 & 4rads) + 79 rad photon treatment• Performed two film exposures• Operator used hand controls to rotate table to field-light position & check alignment• Operator set machine but forgot to remove film• Operator turned beam on, machine showed no dose & displayed fleeting message• Operator proceeded from pause• After another machine pause, operator reentered room.• Patient complained of burning sensation• Patient developed severe striped burns• Patient died in April• Hospital obtained similar pattern on film by running machine with turntable in field light position Yakima Valley, January 1987Responses• Voluntary Class II recall 8/1/85• AECL accident report April 15, 1986• First version of corrective action plan 6/13/86• Second Yakima overdose 1/17/87• Fifth (final) corrective action plan 7/21/87• Interim safety analysis report 1/29/88• Final safety analysis report 11/3/88Tyler Accident Race ConditionOperator Screen LayoutYakima Accident Race ConditionTherac 25 TurntableCorrective Action Plan• Numerous hardware and software changes• All interruptions related to dosimetry notcontinuable• independent hardware & software shutdowns• potentiometer on turntable• hardware interlocks• “dead man switch” motion enable• Fix documentation, messages, & user manuals• etcLessons ( Leveson & Turner)• Complacency• Assumption that problem was understood without adequate evidence (“the last bug” fallacy).• Sole reliance on software for safety•


View Full Document

Johns Hopkins EN 600 445 - The Therac 25 A case study in safety failure

Download The Therac 25 A case study in safety failure
Our administrator received your request to download this document. We will send you the file to your email shortly.
Loading Unlocking...
Login

Join to view The Therac 25 A case study in safety failure and access 3M+ class-specific study document.

or
We will never post anything without your permission.
Don't have an account?
Sign Up

Join to view The Therac 25 A case study in safety failure 2 2 and access 3M+ class-specific study document.

or

By creating an account you agree to our Privacy Policy and Terms Of Use

Already a member?