Page 1Disco: Running Commodity Operating Systems on Scalable MultiprocessorsProceedings of the 16th Symposium on Operating Systems Principles (SOSP). Saint-Malo, France. October 1997.In this paper we examine the problem of extending modern operat-ing systems to run efficiently on large-scale shared memory multi-processors without a large implementation effort. Our approachbrings back an idea popular in the 1970s, virtual machine monitors.We use virtual machines to run multiple commodity operating sys-tems on a scalable multiprocessor. This solution addresses many ofthe challenges facing the system software for these machines. Wedemonstrate our approach with a prototype called Disco that canrun multiple copies of Silicon Graphics’ IRIX operating system ona multiprocessor. Our experience shows that the overheads of themonitor are small and that the approach provides scalability as wellas the ability to deal with the non-uniform memory access time ofthese systems. To reduce the memory overheads associated withrunning multiple operating systems, we have developed techniqueswhere the virtual machines transparently share major data struc-tures such as the program code and the file system buffer cache. Weuse the distributed system support of modern operating systems toexport a partial single system image to the users. The overall solu-tion achieves most of the benefits of operating systems customizedfor scalable multiprocessors yet it can be achieved with a signifi-cantly smaller implementation effort.1 IntroductionScalable computers have moved from the research lab to the mar-ketplace. Multiple vendors are now shipping scalable systems withconfigurations in the tens or even hundreds of processors. Unfortu-nately, the system software for these machines has often trailedhardware in reaching the functionality and reliability expected bymodern computer users.Operating systems developers shoulder much of the blame forthe inability to deliver on the promises of these machines. Extensivemodifications to the operating system are required to efficientlysupport scalable machines. The size and complexity of modern op-erating systems have made these modifications a resource-intensiveundertaking.In this paper, we present an alternative approach for construct-ing the system software for these large computers. Rather than mak-ing extensive changes to existing operating systems, we insert anadditional layer of software between the hardware and operatingsystem. This layer acts like a virtual machine monitor in that multi-ple copies of “commodity” operating systems can be run on a singlescalable computer. The monitor also allows these commodity oper-ating systems to efficiently cooperate and share resources with eachother. The resulting system contains most of the features of customscalable operating systems developed specifically for these ma-chines at only a fraction of their complexity and implementationcost. The use of commodity operating systems leads to systems thatare both reliable and compatible with the existing computing base.To demonstrate the approach, we have constructed a prototypesystem targeting the Stanford FLASH shared memory multiproces-sor [17], an experimental cache coherent non-uniform memory ar-chitecture (ccNUMA) machine. The prototype, called Disco,combines commodity operating systems not originally designed forsuch large-scale multiprocessors to form a high performance sys-tem software base.Disco contains many features that reduce or eliminate theproblems associated with traditional virtual machine monitors. Spe-cifically, it minimizes the overhead of virtual machines and enhanc-es the resource sharing between virtual machines running on thesame system. Disco allows the operating systems running on differ-ent virtual machines to be coupled using standard distributed sys-tems protocols such as NFS and TCP/IP. It also allows for efficientsharing of memory and disk resources between virtual machines.The sharing support allows Disco to maintain a global buffer cachetransparently shared by all the virtual machines, even when the vir-tual machines communicate through standard distributed protocols.Our experiments with realistic workloads on a detailed simu-lator of the FLASH machine show that Disco achieves its goals.With a few simple modifications to an existing commercial operat-ing system, the basic overhead of virtualization is at most 16% forall our uniprocessor workloads. We show that a system with eightvirtual machines can run some workloads 40% faster than on acommercial symmetric multiprocessor operating system by in-creasing the scalability of the system software, without substantial-ly increasing the system’s memory footprint. Finally, we show thatpage placement and dynamic page migration and replication allowDisco to hide the NUMA-ness of the memory system, improvingthe execution time by up to 37%.In Section 2, we provide a more detailed presentation of theproblem being addressed. Section 3 describes an overview of theapproach and the challenges of using virtual machines to constructthe system software for large-scale shared-memory multiproces-sors. Section 4 presents the design and implementation of Discoand Section 5 shows experimental results. We end the paper with adiscussion of related work in Section 6 and conclude in Section 7.2 Problem DescriptionThis paper addresses the problems seen by computer vendors at-tempting to provide system software for their innovative hardware.For the purposes of this paper, the innovative hardware is scalableshared memory multiprocessors, but the issues are similar for anyhardware innovation that requires significant changes in the systemsoftware. For shared memory multiprocessors, research groupshave demonstrated prototype operating systems such as Hive [5]and Hurricane [25] that address the challenges of scalability andfault containment. Silicon Graphics has announced the CellularIRIX kernel to support its shared memory machine, theOrigin2000 [18]. These designs require significant OS changes, in-cluding partitioning the system into scalable units, building a singleDisco: Running Commodity Operating Systems on Scalable MultiprocessorsEdouard Bugnion, Scott Devine, and Mendel RosenblumComputer Systems LaboratoryStanford University,Stanford, CA 94305{bugnion, devine, mendel}@cs.stanford.eduhttp://www-flash.stanford.edu/DiscoSOSP 16.(c) ACM 1997.Page 2Disco: Running Commodity Operating Systems on Scalable