Scalability in the XFS File SystemAdam Sweeney, Doug Doucette, Wei Hu, Curtis Anderson, MikeNishimoto, and GeoffPeckSilicon Graphics, Inc.AbstractIn this paper we describe the architecture and designof a newfile system, XFS, for Silicon Graphics’ IRIXoperating system. It is a general purpose file systemfor use on both workstations and servers. The focus ofthe paper is on the mechanisms used by XFS to scalecapacity and performance in supporting very large filesystems. The large file system support includes mech-anisms for managing large files, large numbers offiles, large directories, and very high performance I/O.In discussing the mechanisms used for scalability weinclude both descriptions of the XFS on-disk datastructures and analyses of whytheywere chosen. Wediscuss in detail our use of B+ trees in place of manyof the more traditional linear file system structures.XFS has been shipping to customers since Decemberof 1994 in a version of IRIX 5.3, and we are continu-ing to improve its performance and add features inupcoming releases. Weinclude performance resultsfrom running on the latest version of XFS to demon-strate the viability of our design.1. IntroductionXFS is the next generation local file system for Sili-con Graphics’ workstations and servers. It is a generalpurpose Unix file system that runs on workstationswith 16 megabytes of memory and a single disk driveand also on large SMP network servers with gigabytesof memory and terabytes of disk capacity.Inthispaper we describe the XFS file system with a focus onthe mechanisms it uses to manage large file systemson large computer systems.The most notable mechanism used by XFS to increasethe scalability of the file system is the pervasive use ofB+ trees [Comer79]. B+ trees are used for trackingfree extents in the file system rather than bitmaps. B+trees are used to indexdirectory entries rather thanusing linear lookup structures. B+ trees are used tomanage file extent maps that overflowthe number ofdirect pointers kept in the inodes. Finally,B+trees areused to keep track of dynamically allocated inodesscattered throughout the file system. In addition, XFSuses an asynchronous write ahead logging scheme forprotecting complexmetadata updates and allowingfast file system recovery after a crash. Wealso supportvery high throughput file I/O using large, parallel I/Orequests and DMA to transfer data directly betweenuser buffers and the underlying disk drives. Thesemechanisms allowustorecoverevenvery large filesystems after a crash in typically less than 15 seconds,to manage very large file systems efficiently,and toperform file I/O at hardware speeds that can exceed300 MB/sec.XFS has been shipping to customers since Decemberof 1994 in a version of IRIX 5.3, and XFS will be thedefault file system installed on all SGI systems start-ing with the release of IRIX 6.2 in early 1996. The filesystem is stable and is being used on productionservers throughout Silicon Graphics and at manyofour customers’ sites.In the rest of this paper we describe whywechose tofocus on scalability in the design of XFS and themechanisms that are the result of that focus. Westartwith an explanation of whywechose to start fromscratch rather than enhancing the old IRIX file sys-tem. Wenextdescribe the overall architecture of XFS,followed by the specific mechanisms of XFS whichallowittoscale in both capacity and performance.Finally,wepresent performance results from runningon real systems to demonstrate the success of the XFSdesign.2. WhyaNew File System?The file system literature beganpredicting the comingof the "I/O bottleneck" years ago [Ousterhout90], andwe experienced it first hand at SGI. The problem wasnot the I/O performance of our hardware, but the limi-tations imposed by the old IRIX file system, EFS[SGI92]. EFS is similar to the BerkeleyFast File Sys-tem [McKusick84] in structure, but it uses extentsrather than individual blocks for file space allocationand I/O. EFS could not support file systems greaterthan 8 gigabytes in size, files greater than 2 gigabytesin size, or give applications access to the full I/Obandwidth of the hardware on which theywererunning. EFS was not designed with large systems andfile systems in mind, and it was faltering under thepressure coming from the needs of newapplicationsand the capabilities of newhardware. While we con-sidered enhancing EFS to meet these newdemands,the required changes were so great that we decided itwould be better to replace EFS with an entirely newfile system designed with these newdemands in mind.One example of a newapplication that places newdemands on the file system is the storage and retrievalof uncompressed video. This requires approximately30 MB/sec of I/O throughput for a single stream, andjust one hour of such video requires 108 gigabytes ofdisk storage. While most people work with com-pressed video to makethe I/O throughput and capac-ity requirements easier to manage, manycustomers,for example those involved in professional video edit-ing, come to SGI looking to work with uncompressedvideo. Another video storage example that influencedthe design of XFS is a video on demand server,suchas the one being deployed by Time Warner inOrlando. These servers store thousands of compressedmovies. One thousand typical movies takeuparound2.7 terabytes of disk space. Playing twohundred highquality 0.5 MB/sec MPEG streams concurrently uses100 MB/sec of I/O bandwidth. Applications with sim-ilar requirements are appearing in database and scien-tific computing, where file system scalability and per-formance is sometimes more important than CPU per-formance. The requirements we derivedfrom theseapplications were support for terabytes of disk space,huge files, and hundreds of megabytes per second ofI/O bandwidth.We also needed to ensure that the file system couldprovide access to the full capabilities and capacity ofour hardware, and to do so with a minimal amount ofoverhead. This meant supporting systems with multi-ple terabytes of disk capacity.With today’s9gigabytedisk drivesitonly takes 112 disk drivestosurpass 1terabyte of storage capacity.These requirements alsomeant providing access to the large amount of diskbandwidth that is available in such high capacity sys-tems. Today,SGI’shigh end systems have demon-strated sustained disk bandwidths in excess of 500megabytes per second. Weneeded to makethat band-width available to applications using the file system.Finally,these requirements meant doing
View Full Document
Unlocking...