Sympathy for the Sensor Network DebuggerNithya Ramanathan, Kevin Chang, Rahul Kapur, Lewis Girod, Eddie Kohler, and Deborah EstrinUCLA Center for Embedded Network Sensing{nithya, kchang, rkapur, girod, kohler, destrin}@cs.ucla.eduABSTRACTBeing embedded in the physical world, sensor networks present awide range of bugs and misbehavior qualitatively different fromthose in most distributed systems. Unfortunately, due to resourceconstraints, programmers must investigate these bugs with onlylimited visibility into the application. This paper presents the de-sign and evaluation of Sympathy, a tool for detecting and debug-ging failures in sensor networks. Sympathy has selected metricsthat enable efficient failure detection, and includes an algorithmthat root-causes failures and localizes their sources in order to re-duce overall failure notifications and point the user to a small num-ber of probable causes. We describe Sympathy and evaluate its per-formance through fault injection and by debugging an active ap-plication, ESS, in simulation and deployment. We show that for abroad class of data gathering applications, it is possible to detectand diagnose failures by collecting and analyzing a minimal set ofmetrics at a centralized sink. We have found that there is a trade-off between notification latency and detection accuracy; that addi-tional metrics traffic does not always improve notification latency;and that Sympathy’s process of failure localization reduces primaryfailure notifications by at least 50% in most cases.Categories and Subject Descriptors: D.2.5 [Software Engi-neering]: Testing and Debugging—Distributed debugging; C.2.4[Computer Communication Networks]: Distributed SystemsGeneral Terms: Design, ReliabilityKeywords: sensor networks, debugging, failure detection, fail-ure localization, root causes1 INTRODUCTIONDeveloping and debugging sensor network applications in a dy-namic, distributed, and resource-constrained embedded environ-ment is an iterative and sometimes laborious process. Initial appli-cation development can use a protected and interactive simulation.Once an application is physically deployed, however, interactivityand visibility are greatly reduced, and it becomes difficult to de-tect and pinpoint problems when they occur. For example, a gapin returned sample data may be caused by a critical node failure,a transient change in link connectivity, or some other unexpectedcombination of inputs. Responding to a failure can require physicalaccess to a node; depending on the deployment scenario, even ob-taining access can be expensive and difficult—or, worse, a cause ofadditional failures [17]. A sensor network system should thereforePermission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.SenSys’05, November 2–4, 2005, San Diego, California, USA.Copyright 2005 ACM 1-59593-054-X/05/0011 . ..$5.00.help narrow down failures and diagnose their causes, as much aspossible, with minimal physical access and interactivity.This problem is not new, of course. Many network software andhardware tools help IP network administrators diagnose variousissues. Unfortunately, these tools often require network and noderesources not available to embedded network sensors, or assumenetworks that are more stable than sensor networks, whose nodesfrequently become inaccessible and have poor connectivity, limitedpower, and memory resources. Such tools may also ignore the costof transmitting debugging information, another significant concern.This paper presents Sympathy, a prototype tool for detecting anddebugging failures in pre- and post-deployment sensor networks.More specifically, Sympathy is designed for data collection appli-cations, which gather distributed data at a centralized sink locationfor analysis. (Most of today’s deployed sensor networks fit this de-scription, as will many networks deployed in future.) Nodes peri-odically send metrics back to a sink, which combines this infor-mation with passively-gathered metrics to detect failures and deter-mine their causes.Given sensor hardware and network limitations, these transmit-ted metrics must be minimized. Thus, Sympathy must find the de-bugging information that provides the maximum leverage: the in-formation that allows the most precise and meaningful failure de-tection and localization for the lowest overhead. We chose metricsusing a simple insight: in a data-collection network, there is a directrelationship between the amount of data collected at the sink andthe existence of failures in the system. Insufficient data at the sinkimplies failure; sufficient data at the sink implies acceptable net-work behavior. Thus, Sympathy can limit its metrics collection toinformation about connectivity and data flow. Furthermore, whena failure occurs, the user’s goal is to restore data collection. Sym-pathy’s algorithms for localizing failures thus have the more spe-cific and limited goal of telling the user which node or path is re-sponsible for missing data. Of course, this excludes some kindsof network misbehavior, such as overenthusiastic transmission andresulting reduced battery lifetime. However, our experience withSympathy—both when deployed alone and when deployed in con-cert with a real application [8]—indicates that many failures en-countered in today’s networks do fit our model, making Sympathy’sdebugging support largely sufficient as well as useful.Sympathy gathers and analyzes general system metrics such asnodes’ next hops and neighbors. Based on these metrics, it detectswhich nodes or components have not delivered sufficient data to thesink and infers the causes of these failures. Experimentation andactual deployments show that Sympathy can help detect internalfailures whose causes in most cases would not be readily apparent.Our deployment experience centers on the Extensible SensingSystem (ESS), which gathers environmental data for user analy-sis [8]. ESS-Sympathy deployments are intended for 10 to 100 dis-tributed sensor nodes routing data through a multihop tree backto a sink. During one deployment, Sympathy discovered data dis-ruption due to excessive flux in