11/11/09!1!UC Berkeley 1 Experiences with X-Trace: an end-to-end, datapath tracing framework!George Porter, Rodrigo Fonseca, Matei Zaharia, Andy Konwinski, Randy H. Katz, Scott Shenker, Ion Stoica!Sun Microsystems Laboratories!January 30, 2008!2 X-Trace !• Framework for capturing causality of events in a distributed system!– Coherent logging: events are placed in a causal graph!• Capture causality, concurrency!• Across layers, applications, administrative boundaries!• Audience!– Developers: debug complex distributed applications!– Operators: pinpoint causes of failures!– Users: report on anomalous executions!11/11/09!2!3 Many servers, 4 worldwide sites!A user gets a stale page. What went wrong?!Combinatorial explosion of possible paths through the app!Example: Wikipedia!DNS !Round-Robin!33 Web !Caches!4 Load !Balancers!105 HTTP +!App Servers!14 Database!Servers!4 Well Known Problem!- Disconnected logs in different components!- Multiple problems with the same symptoms!- Execution paths are ephemeral!• Troubleshooting distributed systems is hard!11/11/09!3!5 Coral CDN!• Open, distributed content distribution network!– Distributed cache, uses a self-organizing, locality-aware DHT!– Usage is simple: append .nyud.net to domain in url!– +25M requests/day, runs on > 250 Planetlab nodes!– Built with libasync!6 A Coral Request!DNS!Coral Nodes!HTTP!RPC!RPC!HTTP!• Interesting case for tracing!– Involves recursive DNS, HTTP, RPC (and ICMP)!– We trace DNS processing, HTTP (incl recursive), and RPC!www.cnn.com.nyud.net/somepage.html!cnn.com!11/11/09!4!7 Adding X-Trace!• Capture events within application!– Logging API!– Capture abstraction!– Capture parallelism!8 Adding X-Trace!• Capture events on different layers!– e.g. HTTP and RPC!11/11/09!5!9 Adding X-Trace!• Correlate events!– Across different machines!– Across different layers!10 X-Trace Mechanisms!• Each Task gets a unique TaskId!• Each Event within a task get a unique EventId!• When logging, each event must record “edge”:!previous EventId > new EventId!• <TaskId, last EventId> propagated with execution!– Within runtime environment (X-Trace libraries)!– Through API calls (augmented APIs)!– In protocol messages (as header metadata)!11/11/09!6!11 Trace collection and storage!• Trace data buffered and distributed across instrumented hosts!• Collection process!– Orthogonal!– Minimize collection overhead via buffering and compression!Back-end!Front-end (per host)!12 Explicit Path Tracing!• Advantages!– Deterministic causality and concurrency!– Handle on specific executions (name the needles)!– Does not depend on time synchronization!– Correlated logging!• Meaningful sampling (random, biased, triggered...)!• Disadvantages!– Modify applications and protocols (some)!11/11/09!7!13 Talk Roadmap • X-Trace motivation and mechanism!• Use cases:!1. Wide-area: Coral Content Distribution Network!2. Enterprise: 802.1X network authentication!3. Datacenter: Hadoop Map/Reduce!• Future work within the RAD Lab!– Debugging and performance!– Clustering and analysis of relationships between traces!– Applying tracing to energy conservation and datacenter management!14 Use Cases!1. Wide-area: Coral CDN!2. Enterprise: 802.1X Network Authentication!3. Datacenter: Hadoop Map/Reduce!11/11/09!8!15 Coral Deployment!• Running on production Coral network since Christmas !• 253 machines!• Sampling: tracing 0.1% of requests!16 Initial Findings!• Found at least 5 bugs :-)!• “Wrong timeout value for proxy”!• “Some paths when server fetch fail may not kill client connection”!• “Does Coral lookup even when likely to be over cache size”!• “Forwarding internal state to client”!• “Revalidation always goes to origin servers, not to peers”!• Some timing issues!– Very slow HTTP responses, investigating cause!11/11/09!9!17 Time of HTTP Responses!• 1. Client timeout, large object, slow node, 1 block!• 2. Very slow link to client!• 3. Very slow Coral node!• 4. Failure to connect to origin server, specific 189s timeout !• Can look at graph for each specific point, e.g.!4!3!18 Use Cases!1. Wide-area: Coral CDN!2. Enterprise: 802.1X Network Authentication!3. Datacenter: Hadoop Map/Reduce!11/11/09!10!19 Enterprise: IEEE 802.1X • Controls user access to network resources – Wireless access points – VPN endpoints – Wired ports • User-specific admission criteria • Audit for compliance purposes • Complex protocol – Distributed for scalability and reliability: no central point – Multi-protocol – Spans administrative domains – Multi-vendor 20 802.1X Overview!1! 2! 3! 4!5!6!7!8!1. Client sends credentials to authenticator 2. Authenticator forwards credentials to auhentication server with RADIUS 3. Authentication server queries identity store with LDAP 4. Identity store processes query 5. Identity store responds with success or failure using LDAP 6. Authentication server makes descision; sends RADIUS response to Authenticator 7. Authentication server receives response 8. Access is granted or denied11/11/09!11!21 802.1X and X-Trace: how to read a trace of a successful request!A! B!C!D!E!A!B!C!D!E!22 Approach!• Collect application traces with X-Trace!• Determine when a fault is occuring!• Localize the fault!• Determine the root cause, if possible!• Report problem and root cause (if known) to network operator!11/11/09!12!23 Root cause determination!• Why these tests?!– Occur in customer deployments!– Based on conversation with support technicians!24 Root test 1: insufficient timeout setting!• Timeouts spread throughout system; set by different administrators, vendors!• Detection: Authenticator times out at T1, Radius server issues report at T2 > T1!• Radius timeout at T1 = 2.034s!• LDAP responds at T2 = 3.449s > T1!11/11/09!13!25 Root test 2: udp packet loss (reverse path)!• Radius protocol based on udp: packet loss manifests as a dropped message!• Detection: evidence of forward progress coupled with timeout and retry!26 Inferring network failures with application
View Full Document
Unlocking...