Proceedings of the 16th Symposium on Operating Systems Principles SOSP Saint Malo France October 1997 Disco Running Commodity Operating Systems on Scalable Multiprocessors Edouard Bugnion Scott Devine and Mendel Rosenblum Computer Systems Laboratory Stanford University Stanford CA 94305 bugnion devine mendel cs stanford edu http www flash stanford edu Disco In this paper we examine the problem of extending modern operating systems to run efficiently on large scale shared memory multiprocessors without a large implementation effort Our approach brings back an idea popular in the 1970s virtual machine monitors We use virtual machines to run multiple commodity operating systems on a scalable multiprocessor This solution addresses many of the challenges facing the system software for these machines We demonstrate our approach with a prototype called Disco that can run multiple copies of Silicon Graphics IRIX operating system on a multiprocessor Our experience shows that the overheads of the monitor are small and that the approach provides scalability as well as the ability to deal with the non uniform memory access time of these systems To reduce the memory overheads associated with running multiple operating systems we have developed techniques where the virtual machines transparently share major data structures such as the program code and the file system buffer cache We use the distributed system support of modern operating systems to export a partial single system image to the users The overall solution achieves most of the benefits of operating systems customized for scalable multiprocessors yet it can be achieved with a significantly smaller implementation effort 1 Introduction Scalable computers have moved from the research lab to the marketplace Multiple vendors are now shipping scalable systems with configurations in the tens or even hundreds of processors Unfortunately the system software for these machines has often trailed hardware in reaching the functionality and reliability expected by modern computer users Operating systems developers shoulder much of the blame for the inability to deliver on the promises of these machines Extensive modifications to the operating system are required to efficiently support scalable machines The size and complexity of modern operating systems have made these modifications a resource intensive undertaking In this paper we present an alternative approach for constructing the system software for these large computers Rather than making extensive changes to existing operating systems we insert an additional layer of software between the hardware and operating system This layer acts like a virtual machine monitor in that multiple copies of commodity operating systems can be run on a single scalable computer The monitor also allows these commodity operating systems to efficiently cooperate and share resources with each other The resulting system contains most of the features of custom scalable operating systems developed specifically for these machines at only a fraction of their complexity and implementation SOSP 16 c ACM 1997 cost The use of commodity operating systems leads to systems that are both reliable and compatible with the existing computing base To demonstrate the approach we have constructed a prototype system targeting the Stanford FLASH shared memory multiprocessor 17 an experimental cache coherent non uniform memory architecture ccNUMA machine The prototype called Disco combines commodity operating systems not originally designed for such large scale multiprocessors to form a high performance system software base Disco contains many features that reduce or eliminate the problems associated with traditional virtual machine monitors Specifically it minimizes the overhead of virtual machines and enhances the resource sharing between virtual machines running on the same system Disco allows the operating systems running on different virtual machines to be coupled using standard distributed systems protocols such as NFS and TCP IP It also allows for efficient sharing of memory and disk resources between virtual machines The sharing support allows Disco to maintain a global buffer cache transparently shared by all the virtual machines even when the virtual machines communicate through standard distributed protocols Our experiments with realistic workloads on a detailed simulator of the FLASH machine show that Disco achieves its goals With a few simple modifications to an existing commercial operating system the basic overhead of virtualization is at most 16 for all our uniprocessor workloads We show that a system with eight virtual machines can run some workloads 40 faster than on a commercial symmetric multiprocessor operating system by increasing the scalability of the system software without substantially increasing the system s memory footprint Finally we show that page placement and dynamic page migration and replication allow Disco to hide the NUMA ness of the memory system improving the execution time by up to 37 In Section 2 we provide a more detailed presentation of the problem being addressed Section 3 describes an overview of the approach and the challenges of using virtual machines to construct the system software for large scale shared memory multiprocessors Section 4 presents the design and implementation of Disco and Section 5 shows experimental results We end the paper with a discussion of related work in Section 6 and conclude in Section 7 2 Problem Description This paper addresses the problems seen by computer vendors attempting to provide system software for their innovative hardware For the purposes of this paper the innovative hardware is scalable shared memory multiprocessors but the issues are similar for any hardware innovation that requires significant changes in the system software For shared memory multiprocessors research groups have demonstrated prototype operating systems such as Hive 5 and Hurricane 25 that address the challenges of scalability and fault containment Silicon Graphics has announced the Cellular IRIX kernel to support its shared memory machine the Origin2000 18 These designs require significant OS changes including partitioning the system into scalable units building a single Disco Running Commodity Operating Systems on Scalable Multiprocessors Page 1 Proceedings of the 16th Symposium on Operating Systems Principles SOSP Saint Malo France October 1997 OS Pmake DB NFS SMP OS OS OS