Performance Measurements of Networked Files Systems Winfred Byrd and John Moser wbyrd cs wisc edu jmoser erc wisc edu SOS Please Conference cs736 Advanced Topics in Operating Systems Fall 2000 Professor Remzi Arpaci Dusseau December 18 2000 Abstract The inability of common benchmarks to gracefully scale to evaluate systems of various size and configuration contributes greatly to the incapacity usually shown in comparing varied systems under a common metric This paper looks at an older self scaling I O benchmark whose results can be normalized using techniques of predicted performance for comparative evaluation of a diverse range of stand alone computer systems We assert that there is no reason these comparisons need stop at local disk workstations and show the comparison to produce a valid evaluation of network file system enabled computers using SUN s NFS protocol 1 Introduction The history of performance evaluation is littered with benchmarks unable to stress the larger and more powerful computers that evolved in their aftermath This inability to scale greatly contributes to inaccuracies when evaluating and comparing varied systems under a common metric In the discussion that follows we examine what characteristics a benchmark need have so that it may be used to realistically evaluate modern and future computer systems We look at what we gain from each of these characteristics and why earlier proposed benchmarks lacking in them have grown obsolete We then turn to a proposed benchmark that meets all of our stated criteria Following this we discuss the changes that were necessary to convert this benchmark for running over a distributed file system The remainder of this paper is divided into the following sections section 2 sets the standards for an ideal benchmark section 3 looks at related work section 4 addresses our testing procedure section 5 presents our results section 6 considers future work and in section 7 conclusions are drawn from the results 2 Ideal Benchmarks When attempting to evaluate systems several questions naturally present themselves What are common shortcomings among existing benchmarks Is there a benchmark that is not limited by these shortcomings or more generally what would be our ideal benchmark Fortunately the literature is rich with attempts to answer that question One such claim obliges that an ideal I O benchmark should Increase understanding of the system This includes helping designers isolate points of poor performance as well as instructing users in optimal machine usage under different workloads Be I O limited Intuitively the facet of the system we are attempting to evaluate should be the bottleneck of our system Scale gracefully We cannot expect comparative information between diverse systems to be useful if our evaluation program is ineffective at measuring systems in the performance range of one or more of the platforms in question Further we would like our evaluation to remain useful with the next generation of computers Hence scaling to stress more powerful systems is a must Allow a fair comparison across machines Relevant to many applications We would like to be able to comparatively evaluate systems under all types of workloads CAD office automation software development and so on Though design decisions are arguably much easier when we target a specific application there is obvious value in very general applicability Tightly Specified For benchmarks to have meaning they must be well defined reproducible and reported with rigor What optimizations such as caching are allowed How is the physical machine environment configured If we are going to hold our results as absolutely reliable these questions become very important 2 With this more thorough specification of the required characteristics we search for our ideal benchmark 3 Related Work In the past the benchmarking of various components of computers has invariably followed a trend towards one or two standards With NFS performance measurement this hasn t changed The computing industry has seen two distinct methods for judging NFS performance in two pieces of software NFS NHFSStone and LADDIS 3 1 NFS NHFSStone Barry Shein initially published NFSStone at the 1989 USENIX conference in a paper entitled NFSSTONE A Network File Server Performance Benchmark Legato Systems created NHFSStone which was incorporated as a NFS load generating program Both share many features And while both may have been used in the past as a popular means of describing NFS performance both also have many deficiencies We will speak in general of NHFSStone though the references apply to NFSStone as well unless otherwise noted 7 First both benchmarks utilized the synthetic duplication of an average user workload on top of an average NFS resource utilization level While this may give us an indication of what might be the performance of our system under a median level workload it does not indicate any sort of performance measurement to the extremities Additionally NHFSStone is limited in the number of clients it supports one Any attempt to document the performance of a large scale network wouldn t find any help within NHFSStone Without any network contention the main bottleneck to NFS couldn t be accurately measured that being the network infrastructure itself Even if control coordination software were implemented to maintain several client copies of NHFSStone the results reported would lose reliability as additional levels of influence were introduced Second both NHFSStone and NFSStone are sensitive to the differences in NFS client implementation The generated NFS command and request sequence produced for any given test could vary sometimes greatly amongst clients depending on how their manufacturer decided to implement the NFS protocol As the client workload increases more inconsistencies are introduced into the results of the benchmark NHFSStone attempted to work around these inconsistencies by implementing various algorithms depending on what client was used As can be expected the effectiveness varied greatly and the results produced weren t reliable In fact the only operating system that the benchmark truly worked well on since it was written originally for that operating system was SunOS The benchmark utilized several operating specific algorithms along with a high level access to kernel data structures file name information attributes caching policies etc While this definitely simplifies use of the benchmark on a