Scalability, Fidelity, and Containmentin the Potemkin Virtual HoneyfarmMichael Vrable, Justin Ma, Jay Chen, David Moore, Erik Vandekieft,Alex C. Snoeren, Geoffrey M. Voelker, and Stefan SavageCollaborative Center for Internet Epidemiology and DefensesDepartment of Computer Science and EngineeringUniversity of California, San DiegoABSTRACTThe rapid evolution of large-scale worms, viruses and bot-nets have made Internet malware a pressing concern. Suchinfections are at the root of modern scourges including DDoSextortion, on-line identity theft, SPAM, phishing, and piracy.However, the most widely used tools for gathering intelli-gence on new malware — network honeypots — have forcedinvestigators to choose between monitoring activity at alarge scale or capturing behavior with high fidelity. In thispaper, we describe an approach to minimize this tension andimprove honeypot scalability by up to six orders of magni-tude while still closely emulating the execution behavior ofindividual Internet hosts. We have built a prototype hon-eyfarm system, called Potemkin, that exploits virtual ma-chines, aggressive memory sharing, and late binding of re-sources to achieve this goal. While still an immature im-plementation, Potemkin has emulated over 64,000 Internethoneypots in live test runs, using only a handful of physicalservers.Categories and Subject DescriptorsD.4.6 [Operating Systems]: Security and Protection—In-vasive software; C.2.0 [Computer-Communication Net-works]: General—Security and protection; D.4.2 [Oper-ating Systems]: Storage Management—Virtual memory;C.2.3 [Computer-Communication Networks]: NetworkOperations—Network monitoringGeneral TermsMeasurement, SecurityKeywordscopy-on-write, honeyfarm, honeypot, malware, virtual ma-chine monitorPermission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.SOSP’05, October 23–26, 2005, Brighton, United Kingdom.Copyright 2005 ACM 1-59593-079-5/05/0010 ...$5.00.1. INTRODUCTIONThe ability to compromise large numbers of Internet hostshas emerged as the backbone of a new criminal economy en-compassing bulk-email (SPAM), denial-of-service extortion,phishing, piracy, and identify theft. Using tools such asworms, viruses and scanning botnets, the technical cadreof this community can leverage a handful of software vul-nerabilities into a large-scale virtual commodity — hun-dreds of thousands of remotely controlled “bot” hosts —that are then used, resold or leased for a variety of illegalpurposes [13].While a range of tactical countermeasures can be em-ployed against the uses of these hosts (e.g., SPAM filters,DoS defenses), combating the underlying infestation requiresfirst understanding the means and methods used to compro-mise and subsequently control the bot population. By far,the most important tool for this purpose is the honeypot.Put simply, a honeypot is a network-connected system thatis carefully monitored (and frequently left unprotected) sothat intrusions can be easily detected and precisely ana-lyzed. Such information is then used in turn to create anti-virus signatures (to limit further growth), to develop disin-fection algorithms (to eradicate existing infections), and tosupport criminal investigation and prosecution.In practice, however, deploying a large network of hon-eypot systems, a honeyfarm, exposes a sharp tradeoff be-tween scalability, fidelity, and containment. At one extreme,so-called “low-interaction honeypots” can monitor activityacross millions of IP addresses at a time. Such honeypotsachieve this scalability by only emulating the network in-terface exposed by common services and thus maintaininglittle or no per-honeypot state [27, 40]. However, since thesesystems do not execute any code from native applications oroperating systems they are unable to determine if an attackis effective, why a given exploit works, what the payloaddoes, or how the compromised system will be controlled,updated, or used. Indeed, such systems may be unable toeven elicit attacks that require multiple phases of communi-cation.In contrast, “high-interaction honeypots” execute nativesystem and application code and thus can capture maliciouscode behavior in its full complexity [12, 32]. Unfortunately,the price of this fidelity is invariably quite high. In theirsimplest form these systems require a single physical hostfor each monitored IP address, and while some systems usevirtual machines to reduce this requirement, it is rarely cost-effective to support more than a few thousand hosts.1Fi-nally, all of these systems struggle to balance the need forcontainment — preventing compromised honeypots from at-tacking third-party systems — and the desire to allow un-fettered network access to enhance system fidelity.In this paper, we describe a honeyfarm system architec-ture that can scale to design points previously reserved forstateless monitors (hundreds of thousands of IP addresses),while offering fidelity qualitatively similar to high-interactionhoneypots. The heart of our approach is to dynamically bindphysical resources to external requests only for the short pe-riods of time necessary to emulate the execution behaviorof dedicated hosts. By exploiting idleness at the networklayer and physical memory coherence between hosts, we ar-gue that the resource requirements of emulating an Internethost can be reduced by up to six orders of magnitude inpractice.To demonstrate our approach, we have implemented aprototype honeyfarm system, called Potemkin, based on aspecialized network gateway and a virtual machine monitorderived from Xen. At the network layer, individual flowsare dispatched to a collection of honeyfarm servers which,in turn, dynamically instantiate new virtual machines toassume the role of each destination IP address. To reduceoverhead and increase the number of VMs supported on eachhoneyfarm server we propose two techniques: flash cloningand delta virtualization. The former instantiates new VMsquickly by copying and modifying a host reference image,thus avoiding the startup overhead of system or applicationinitialization, while the latter optimizes this copy