Checksumming RAID Brian Kroth bpkroth cs wisc edu Abstract Storage systems exhibit silent data corruptions that go unnoticed until too late potenially resulting in whole trees of lost data To deal with this we ve integrated a checksumming mechanism into Linux s Multi Device Software RAID layer so that we are able to detect and correct these silent data corruptions The analysis of our naive implementation shows that this can be done with a reasonable performance overhead 1 Introduction Storage systems perhaps more than other computer system components exhibit silent partial failures These failures can occur anywhere along the storage stack from the operating system arranging and scheduling requests to the device drivers through the communication channel to the backing device medium itself and all of its internal firmware and mechanical underpinnings The failures themselves can be partial or complete temprorary or permanent and despite some hardware support for it detected or not They occur in the form of bit flips along the path bit rot on the medium over time misdirected writes to the medium and phantom writes that never actually make it to the medium At the same time though perhaps unwise most software in a system presumes a stop failure environment in which all the underlying components either completely work or fail in their entirety For example if a region in memory is damaged it is likely that either the program will crash or in the case of virtual memory perhaps the machine will crash These particular problems are probably not the end of the world though since the program or the machine can simply be restarted to restore a clean state However when this happens in a storage component the resulting state can remain with the system across reboots The effect of this could be as simple as a corrupted block within a file In some cases the application format can deal with this however in others it renders the file including the rest of its intact data useless It is also possible the corrupted block occurs in some metadata for the filesystem living on top of the storage component such Suli Yang suli cs wisc edu as a directory In this case whole subtrees of data may become inaccessible The results of this can be anywhere from lost data to an inoperable machine Given that the storing and processing of data is the primary purpose of most computers and that the data people store represents an investment in time and energy these failures and their ensuing loses can be very costly Thus we propse a storage system that is not only capable of detecting these silent corruptions but also correcting them before it s too late To that end we ve extended the Software RAID layer in Linux s Multi Device driver which already includes some redundancy information to be able to recover from some stop failures scenarios to include an integrity checking component in the form of checksums Using this we show that our solution is capable of detecting and correcting single block corruptions within a RAID stripe at a reasonable performance overhead The remainder of this paper is structured as follows In Section 2 we ll discuss some background and related work Section 3 will discuss our implementation details followed by our evaluation and results in Section 4 and finally our conclusions in Section 5 2 Background In this section we review the problem of silent corruptions and discuss a number of other approaches to dealing with it 2 1 The Problem The issue of silent data corruption within storage systems and beyond has been well understood and studied The IRON Filesystems paper 9 presents a good overview of the possible sources of corruption along the storage stack Studies at CERN in 2007 5 and statistics quoted by 4 showed that error rates occurred in only 1 in 1012 to 1014 bits 8 to 12TB However other studies 9 note that this rate will only increase as disks become more complex and new technologies such as flash take hold even as market pressures force the cost and quality of these components downwards Indeed these statistics already increase when we take into account the other components in the system such as memory and I O controllers Given that this problem is well understood it is natural to ask what existing solutions there are which we now discuss 2 2 RAID One common technique used to deal with failures in storage infrastructures is RAID 6 RAID typically provides data redundancy to protect against drive failure by replicating whole or calculated parity data to a separate disk in the array However this mechanism only protects against whole disk failures since the parity block isn t usually read or more importantly compared when a data block is read Indeed in a mostly read oriented array a silent data corruption could occur in a seldomly read parity block resulting in the eventual restoration of bad data The integrity of a RAID system can be verified periodically by a scrubbing process However this process can take a very long time so is seldom done In contrast our solution incorporates an active check of data and parity block integrity at the time of each read or write operation to the array This allows us to detect and repair individual active stripes much sooner For idle data that is not being actively read it may still be wise to run periodic scrubbing processes in order to detect corruptions before we can t repair them However this process could possibly be optimized to only examine the inactive data believe that DIX is a complimentary addition to our solution 2 4 Other software based solutions such ZFS 10 also aim to provide more complete OS awareness of integrity checking In ZFS the device volume and filesystems layers have been integrated to provide a well integrated checksumming storage infrastructure However accomplishing this at such levels often requires significant upheaval and is not universally applicable to other filesystems For example we originally looked at adding checksum support to Linux s ext4 filesystem 1 Because ext4 is an extent based journalling filesystem 3 and ZFS is copy on write providing complete and consistent checksumming protection to both the file and meta data would have required creating a new journalling mode and significantly altering the on disk layout allocation policies page cache system and more In contrast our system adds data integrity checks at a layer that can hopefully do something useful when a mismatch is detected e g repair it as opposed to simply