Guest-Transparent Prevention of Kernel Rootkits with VMM-Based Memory ShadowingIntroductionNICKLEDesignDesign Goals and Threat ModelEnabling Scheme and TechniquesNICKLE ImplementationMemory Shadowing and Guest Memory Access IndirectionFlexible ResponsePorting ExperienceNICKLE EvaluationEffectiveness Against Kernel RootkitsImpact on PerformanceDiscussionRelated WorkConclusionReferencesGuest-Transparent Prevention of KernelRootkits with VMM-Based Memory ShadowingRyan Riley1, Xuxian Jiang2, and Dongyan Xu11CERIAS and Department of Computer Science, Purdue University{rileyrd,dxu}@cs.purdue.edu2Department of Computer Science, North Carolina State [email protected]. Kernel rootkits pose a significant threat to computer systemsas they run at the highest privilege level and have unrestricted access tothe resources of their victims. Many current efforts in kernel rootkit de-fense focus on the detection of kernel rootkits – after a rootkit attack hastaken place, while the smaller number of efforts in kernel rootkit preven-tion exhibit limitations in their capability or deployability. In this paperwe present a kernel rootkit prevention system called NICKLE which ad-dresses a common, fundamental characteristic of most kernel rootkits: theneed for executing their own kernel code. NICKLE is a lightweight, vir-tual machine monitor (VMM) based system that transparently preventsunauthorized kernel code execution for unmodified commodity (guest)OSes. NICKLE is based on a new scheme called memory shadowing,wherein the trusted VMM maintains a shadow physical memory for arunning VM and performs real-time kernel code authentication so thatonly authenticated kernel code will be stored in the shadow memory.Further, NICKLE transparently routes guest kernel instruction fetchesto the shadow memory at runtime. By doing so, NICKLE guaranteesthat only the authenticated kernel code will be executed, foiling the ker-nel rootkit’s attempt to strike in the first place. We have implementedNICKLE in three VMM platforms: QEMU+KQEMU, VirtualBox, andVMware Workstation. Our experiments with 23 real-world kernel rootk-its targeting the Linux or Windows OSes demonstrate NICKLE’s effec-tiveness. Furthermore, our performance evaluation shows that NICKLEintroduces small overhead to the VMM platform.1 IntroductionKernel-level rootkits have proven to be a formidable threat to computer sys-tems: By subverting the operating system (OS) kernel, a kernel rootkit embedsitself into the compromised kernel and stealthily inflicts damages with full, un-restricted access to the system’s resources. Effectively omnipotent in the com-promised systems, kernel rootkits have increasingly been used by attackers tohide their presence and prolong their control over their victims.There have been a number of recent efforts in mitigating the threat of kernelrootkits and they can mainly be classified into two categories: (1) detecting theR. Lippmann, E. Kirda, and A. Trachtenberg (Eds.): RAID 2008, LNCS 5230, pp. 1–20, 2008.c Springer-Verlag Berlin Heidelberg 20082 R. Riley, X. Jiang, and D. Xupresence of kernel rootkits in a system [1, 2, 3, 4, 5] and (2) preventing thecompromise of OS kernel integrity [6, 7]. In the first category, Copilot [4] pro-poses the use of a separate PCI card to periodically grab the memory image ofa running OS kernel and analyze it to determine if the kernel has been compro-mised. The work which follows up Copilot [2] further extends that capability bydetecting the violation of kernel integrity using semantic specifications of staticand dynamic kernel data. SBCFI [3] reports violations of the kernel’s controlflow integrity using the kernel’s control-flow graph. One common attribute ofapproaches in this category is the detection of a kernel rootkit’s presence basedon certain symptoms exhibited by the kernel after the kernel rootkit has alreadystruck. As a result, these approaches are, by design, not capable of preventingkernel rootkit execution in the first place.In the second category, Livewire [6], based on a virtual machine monitor(VMM), aims at protecting the guest OS kernel code and critical kernel datastructures from being modified. However, without modifying the original ker-nel code, an attacker may choose to load malicious rootkit code into the kernelspace by either exploiting kernel vulnerabilities or leveraging certain kernel fea-tures (e.g., loadable kernel module support in modern OSes). More recently,SecVisor [7] is proposed as a hypervisor-based solution to enforce the W⊕Xproperty of memory pages of the guest machine, with the goal of preventingunauthorized code from running with kernel-level privileges. SecVisor requiresmodifying kernel source code and needs the latest hardware-based virtualiza-tion support and thus does not support closed-source OSes or legacy hardwareplatforms. Moreover, SecVisor is not able to function if the OS kernel has mixedpages that contain both code and data. Unfortunately, such mixed kernel pagesdo exist in modern OSes (e.g., Linux and Windows as shown in Section 2.2).To complement the existing approaches, we present NICKLE (“No InstructionCreeping into Kernel Level Executed”)1, a lightweight, VMM-based system thatprovides an important guarantee in kernel rootkit prevention: No unauthorizedcode can be executed at the kernel level. NICKLE achieves this guarantee on topof legacy hardware and without requiring guest OS kernel modification. As such,NICKLE is readily deployable to protect unmodified guest OSes (e.g., FedoraCore 3/4/5 and Windows 2K/XP) against kernel rootkits. NICKLE is based onobserving a common, fundamental characteristic of most modern kernel rootkits:their ability to execute unauthorized instructions at the kernel level. By removingthis ability, NICKLE significantly raises the bar for successfully launching kernelrootkit attacks.To achieve the “NICKLE” guarantee, we first observe that a kernel rootkitis able to access the entire physical address space of the victim machine. Thisobservation inspires us to impose restricted access to the instructions in thekernel space: only authenticated kernel instructions can be fetched for execution.Obviously, such a restriction cannot be enforced by the OS kernel itself. Instead,1With a slight abuse of terms, we use NICKLE to denote both the system itself andthe guarantee achieved by the system – when used in quotation marks.Guest-Transparent Prevention of