Canopy: A Controlled Emulation Environmentfor Network System ExperimentationDan Ports, Austin Clements, Jeff ArnoldMIT 6.829 Computer Networks Project Report{drkp,amdragon,jbarnold}@mit.eduDecember 15, 2005AbstractNetwork systems are hard to debug because they areinherently parallel and non-deterministic. Canopyassists with network debugging by putting the entirenetwork system into a controlled emulation environ-ment constructed from virtual machines and a sim-ulated network. This puts all variables under theuser’s control and provides a coherent, omniscientviewpoint of the entire system. To aid the user in ob-serving and manipulating the system, Canopy pro-vides tools such as traffic visualization, packet ma-nipulation, rollback and replay.1 IntroductionTwo of the most important properties of network sys-tems — scale and nondeterminism — make con-structing a debugger for these systems particularlydifficult. These same properties also suggest how anetwork system debugger should be designed.Scale. Traditional debuggers operate on a singleprocess running on a single node. An ideal net-work system debugger should provide a coherentvantage point over an entire network and be capa-ble of efficiently observing and controlling an ar-bitrary number of nodes simultaneously. Addition-ally, single-node commands such as “step this ma-chine forward one instruction” are no longer mean-ingful when many nodes are involved, so a networksystem debugger should support system-wide com-mands such as “roll back the entire system to imme-diately before this event happened”.Tolerance to varying conditions. Network sys-tems are designed to work under many conditions,but this flexibility can make understanding how anetwork system will behave more difficult. Con-straints help designers understand how a system willbehave; conversely, flexibility tends to add complex-ity and introduce subtle problems. An ideal networkdebugger should provide the experimenter with con-trol over the conditions of the simulation that canlead to differing behavior. To facilitate this levelof control, when a simulation is rolled back and re-played without changes, the simulation should ex-hibit exactly the same behavior as it did during thefirst execution.Canopy, our network system debugger, includesthe following key features derived from the proper-ties discussed above:1. The ability to scale the system to utilize avail-able hardware resources2. The ability to roll the simulation back to anyprevious time3. The ability to replay the simulation from anypoint with only specified changesBuilding a scalable system that provides fea-ture (1) requires being able to take advantage ofmany physical machines. In a user-configured sce-nario involving n virtual computers, Canopy can dis-tribute the computational work across up to n phys-1ical machines. Inter-node communication and syn-chronization uses a master/slave system, as will bediscussed in detail in Section 4.Nondeterminism is a significant concern for fea-tures (2) and (3). If any of the individual nodesbehaves differently in any capacity during a replay,then the outcome of the replay could change for rea-sons unrelated to the modifications of the experi-menter. Such changes can easily arise because com-puters generally exhibit at least slightly different be-havior between different runs – for example, be-cause the operating system’s random number gener-ator seeds itself using hardware event timings thatvary across executions or because context switchesoccur at slightly different times. These unintendedvariations between executions can confuse the exper-imenter by adding confounding variables to a replayin which only a specified set of variables are sup-posed to change.Thus, to effectively experiment with and de-bug network systems, we need to reign-in non-determinism and form an omniscient, controlledviewpoint of the entire network system. This sit-uation calls for a new type of debugger that is notonly aware of network behavior, but that controls all“real-world” events in order to ensure perfectly re-peatable execution. Building such a debugger nat-urally requires some mechanism for putting the en-tire network system into a closed environment that’sboth controlled and observable. To create this closedenvironment, Canopy virtualizes the entire networksystem.The non-determinism of an individual node de-rives entirely from its coupling with the “real world”– specifically, this interaction includes the precisetiming and content of asynchronous hardware eventssuch as clock interrupts and input events. The non-determinism of an overall network system followsfrom the non-determinism of its individual nodes ,as well as from events that occur in the network fab-ric. Thus, to reign-in non-determinism, we strictlycontrol the passage of virtual time across the system,as well as the timing and content of “external inputs”at individual nodes, such as network packets.The rest of this paper is organized as follows. Sec-tion 2 discusses related work. Section 3 presents ba-sic information about using Canopy, including theCanopy interface (Section 3.1) and the Canopy net-work model (Section 3.2). Section 4.1 provides de-tails about how Canopy emulates individual nodes.Section 4.2 describes a simple version of Canopy thatachieves all desired network debugger functionalityexcept scalability. Section 4.3 deals with extendingCanopy so that it provides good scaling behavior.Sections 5 and 6 suggest future work and conclude.2 Related Work2.1 Network Simulators and EmulatorsThe ns2 network simulator [15] is commonly usedfor evaluating network protocols because it supportsa vast number of network elements and traffic mod-els. Typically it is used for packet-level simulation,in which packets are generated by synthetic sources.However, it also includes emulation extensions [6]which make it possible to “tap” live networks, in-troducing their packets into the simulated network,and to inject packets from the simulated networkinto a live network. This sort of emulation makes itpractical to test real systems under esoteric networktopologies, but does not provide the correspondinglevel of control over source behavior that Canopyprovides by virtualizing not just the network but alsothe individual nodes.Emulab [18] overcomes the lack of realism in net-work simulators by configuring physical computersand network switches to deploy the desired networktopology on actual hardware.