A Software Layer for Disk Fault InjectionOutlineOverview - 1Overview - 2Motivation – Why purposely cause disk failures?MotivationSlide 7ChallengesSlide 9Slide 10Related WorkSlide 12Slide 13Slide 14ImplementationImplementation – User-level ConsoleImplementation – IDE Driver ModificationImplementation – Kernel ModuleImplementation – System CallsImplementationIDE Driver (2.4.26 Linux Kernel)Slide 22Slide 23Failure ModelFailure Model – Fault TypesSlide 26Slide 27Slide 28Slide 29Verification of Faults (?)Verification - sectorfailVerification - sectorroVerification - sectorwrongVerification - transdataVerification - transaddrVerification - failstopEvaluationSlide 38SummaryA Software Layer for Disk Fault InjectionJake AdriaensDan GibsonCS 736 Spring 2005Instructor: Remzi Arpaci-DusseauOutline1. Introduction, Motivation, & Challenges2. Related Work3. Implementation Details & IDE Driver4. Fault Model5. Methods & Evaluation6. SummaryOverview - 1Software system for modeling IDE disk faults in an x86/Linux-based computerModification to IDE driver for read/write event interceptionOverview - 2Disks faults described at a high levelFaults passed to kernel-level moduleOn read/write event:–IDE driver calls kernel module to perform request modification–Before write event, module may modify data to-be-written–After read event, module may modify data read from diskMotivation – Why purposely cause disk failures?Commodity HW (and SW!) fails, usually at unexpected times–Causing failures at expected times can help improve fault tolerance measuresCan be used to determine fault tolerance of systems–Various flavors of RAID need fault injectionMotivationFaults can happen at the worst time–In the middle of a PowerPoint presentation…ChallengesDrivers are typically written with reliability in mind–May have error detection / correction measuresShould these be removed? Fooled? Applauded?Low-level drivers critically affect performance and stability of the system–Disk faults need not be “stable,” but shouldn’t have unusual “side effects”ChallengesFailure models difficult to justify–Disk manufacturers don’t offer details on how/why their disks failFailstop model is widely used: models complete, detected disk failureOther models must be chosen generally to account for many different disks, controllers, etc.Outline1. Introduction, Motivation, & Challenges2. Related Work3. Implementation Details & IDE Driver4. Fault Model5. Methods & Evaluation6. SummaryRelated WorkSoftware fault injection–Huang et. al. (and many others) use software fault injection for modifying cached web pages (ACM/ProcWWW)–Jarboui et. al. inject software faults into the Linux kernel and observe system behavior–Nagaraja et. al. inject faults into cluster-based systemsRelated WorkDisk Faults, Modeling, Detection–Kaaniche et. al. inject disk faults to study RAID behavior–Kari et. al. presents fault detection and diagnosis techniques (separate studies)–Various other RAID and/or FS papers use some form of fault injection to model failuresRelated WorkHardware Fault InjectionOutline1. Introduction, Motivation, & Challenges2. Related Work3. Implementation Details & IDE Driver4. Fault Model5. Methods & Evaluation6. SummaryImplementationCore components–User-level parser–In-kernel injection module–In-driver upcalls–System callsAdded ~20 lines to IDE driver codeKernel module is demand-loaded, ~250 lines in size2 System calls, inject_fault and getdrivesize, ~ 120 linesImplementation – User-level ConsoleUsed for fault definition–Console interface for fault definition–Processes batch files–Checks faults for validitySector ranges, probability, etc. (more later)–Passes faults to kernel moduleImplementation – IDE Driver ModificationAdded “upcalls” to injection module–Pass I/O requests to module for modification–Provide callback service on I/O completionAdded special-purpose code for certain fault models–Failstop model requires in-driver actionsImplementation – Kernel ModuleReceives fault lists from user-level consoleCalled by IDE driver to perform insertion when:–LBA sector (SCSI-like) becomes known – sector may be modified–Write is initiated – data to be written may be modified–Read completes – data may be modified before returning control to I/O initiatorImplementation – System CallsAdded two system calls–inject_faults()Used to pass fault definitions to kernel module from user space–getsectors()Used to determine raw sector ranges of IDE devices by name (there are other ways to do this)ImplementationFaults DefinedFaults InjectedDisk RequestI/O InitiatedUpcallModified RequestBus TrafficI/O ReturnsControl ReturnsIDE Driver (2.4.26 Linux Kernel)Important structures–struct requestInformation about an IDE request–READ / WRITE–Number of sectors–Etc–struct ide_drive_s (_t)Information about a drive–Drive name (eg. “hdc”)–Sizing/addressing information–EtcIDE Driver (2.4.26 Linux Kernel)Functions–ide_do_rw_disk (3 versions)Common choke-point for reads & writesMany other similar functions, only this one in useTwo versions, swapped by preprocessor directives (one for DMA, one for PIO)Outline1. Introduction, Motivation, & Challenges2. Related Work3. Implementation Details4. Fault Model5. Methods & Evaluation6. SummaryFailure ModelModels selected to represent “generic IDE” disk–No modeling of specific failure (i.e. Western Digital’s “classic” servo malfunction)–Models based on ranges of affected logical sectors (ala SCSI)Failure Model – Fault Typessectorfail–Models inability of a given sector (block) or sector range to store data reliably–Excited on read of sector:Data read is permuted in some way:–Randomized –Set to specific value –Added to offset –Shifted by one or more bytesFailure Model – Fault Typessectorro–Writes to block have no effect on stored value–Excited on writes to sector:Write requests ignoredsectorwrong–Traffic to a given block is directed to a different block–Excited on reads & writesAddress permuted, similarly to dataFailure Model – Fault Typestransaddr–Sector number wrong for first fault excitation, but right for all others–Excited on reads & writesSector permuted as in sectorwrongtransdata–Data is wrong for first fault excitationData permuted as in