Book Chapter to be published inADVANCES IN COMPUTER SCIENCE AND ENGINEERINGVirtual Machine Security SystemsXin Zhao, Kevin Borders, Atul PrakashDepartment of EECS, University of MichiganAnn Arbor, MI, 48109-2121, USA{zhaoxin,kborders,aprakash}@eecs.umich.eduAbstractCurrent operating systems provide the process abstraction to achieve resource sharing andisolation. From a security perspective, however, an attacker who has compromised one processcan usually gain control of the entire machine. This makes security systems running on thesame computer, such as anti-virus programs or intrusion detection systems, also vulnerable toattack. In response to the imperfect isolation between processes in modern operating systems,security researchers have begun to use virtual machine technology when designing securitysystems. A virtual machine makes raw device requests to a set of devices that are emulated byunderlying software. So, software running in a VM has the appearance of its own dedicatedhardware, but is actually controlled and managed by a software layer on the physical com-puter. With reasonable assumptions, the level of isolation between virtual machines is nearlyequivalent to having two separate machines. At the same time, the underlying software has fullaccess to each virtual machine’s state. These properties make virtual machines very attractivefor designers of security systems. In this chapter we explore a wide variety of security applica-tions that utilize virtual machine technology, including intrusion detection and introspection,honeyfarms, logging and replaying, secure file systems, and even malicious software.3391 Overview of Virtual Machine TechnologyVirtual machines have been in existence since the mid 1970’s [Cre81, Gum83]. A virtual machine(VM) is a logical process (most often an operating system) that interfaces with emulated hardwareand is managed by an underlying control program. Originally, virtual machines were run on main-frames to provide resource multiplexing and isolation [Cre81, Gum83]. Recently, virtual machineshave been growing in popularity. Some virtual machine systems, such as VMWare [SVL01] andXen [BDF+03], have seen widespread deployment at many large corporations [Asl].Most modern virtual machine systems use the virtual machine monitor (VMM) model for man-aging and controlling individual virtual machines. The VMM is a thin software layer that runsdirectly on a physical machine’s hardware. On top of the virtual machine monitor, there can beone or more virtual machines. The VMM provides each virtual machine with a set of virtual inter-faces that resemble direct interfaces to the underlying hardware. Applications on a virtual machinecan run without modification as if they were on running on a dedicated physical machine. TheVMM allows multiple virtual machines to be running at the same time and transparently multi-plexes resources between them [Gol74]. The VMM also isolates the virtual machines from oneanother, preventing them from accessing each other’s memory or disk space. The operating systemthat runs inside of a virtual machine is traditionally referred to as the guest OS, and applicationsrunning on the guest OS are referred to as guest applications.2 Virtual Machine TypesAll computer programs consist of instructions. When a program runs inside of a virtual machine,these instructions either execute directly on the processor, or are interpreted by the underlyingsoftware. The proportion of instructions that run directly on the processor will significantly im-pact a virtual machine’s performance. In one extreme, a real machine only uses direct execution.Programs will run very fast on a real machine because their instructions do not require translation.Direct execution delivers the best performance, but has no support for checking machine state priorto executing privileged instructions. The other extreme is a complete software interpreter machine(CSIM). A CSIM translates and emulates every single processor instruction for a computer pro-gram. This requires a lot of extra work and is usually very slow. A virtual machine monitor usesa combination of these two methods. It executes a ”statistically dominant subset” of program in-structions (including all the basic arithmetic, memory, and branching operations) directly on theprocessor, while emulating privileged instructions such as system calls and device I/O requests[Gol72].Virtual machine monitors can be categorized into two groups: Type I VMMs or Type II VMMs340                      !" #$Figure 1: Architecture of Type I and II Virtual Machine Systems[Gol72]. Figure 1 shows the architecture of each VMM type. A Type I VMM runs directly on thephysical hardware. It does not have an operating system running below it; the Type I VMM is fullyresponsible for scheduling and allocating of the system’s resources between virtual machines. Ex-amples of Type-I VMM include VMWare ESX (enterprise) [Wal02], and Xen [BDF+03]. A Type IIVMM runs as an application in a normal operating system. This operating system controls the realhardware resources, and is typically referred to as the ”Host OS.” The host OS has no knowledgeof the Type II VMM, which is treated like any other process in the system. The operating systemthat runs inside of the Type II VMM is referred to as the ”Guest OS.” Examples of Type-II VMMinclude VMWare GSX (workstation) [SVL01], UML (User-Mode Linux) [Dik00], and FAUma-chine [HWS04]. Because the Type II VMM is running inside of a standard operating system, anysecurity vulnerabilities that lead to the compromise of the host OS will also give full control of theguest OS. Host operating systems for Type II VMMs are more heavyweight than Type I VMMs,and more