DOC PREVIEW
U of I CS 525 - Distributed Monitoring and Management

This preview shows page 1-2-3-24-25-26-27-49-50-51 out of 51 pages.

Save
View full document
View full document
Premium Document
Do you want full access? Go Premium and unlock all 51 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 51 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 51 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 51 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 51 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 51 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 51 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 51 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 51 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 51 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 51 pages.
Access to all documents
Download any document
Ad free experience

Unformatted text preview:

Distributed Monitoring and ManagementLarge Distributed SystemsMonitoring and ManagementManagement and Monitoring OperationsChallengesAstrolabe: A Robust and Scalable Technology for Distributed System Monitoring, Management, and Data MiningOverviewAstrolabe Zone HierarchyAstrolabe Zone Hierarchy (2)Decentralized Hierarchical DatabaseDecentralized Hierarchical Database (2)State AggregationState Merge – Gossip ProtocolMore about Astrolabe GossipMore about Astrolabe Gossip (2)More about Astrolabe Gossip (3)Example: P2P Caching of Large ObjectsExample: P2P Caching of Large Objects (2)MembershipSimulation ResultsSimulation Results (2)DiscussionMoara: Flexible and Scalable Group-Based Querying SystemMotivationMotivation (cont.)Two ApproachesMoara FeaturesCommon QueriesGroup Size and DynamismMoara: Data and Query ModelScalable AggregationDHT TreeOptimizationsDynamic MaintenanceSlide 35Slide 36Adaptation PolicyAdaptation Policy (cont.)Separate Query PlaneSeparate Query Plane (cont.)Composite QueriesComposite Queries (cont.)Finding Low-Cost CoversPerformance Evaluation - Dynamic MaintenancePerformance Evaluation - Separate Query PlaneEmulab ExperimentsPlanetLab ExperimentsDiscussion PointsNetwork Imprecision: A New Consistency Metric for Scalable MonitoringSlide 50ThanksDistributed Monitoring and ManagementPresented by:Ahmed KhurshidAbdullah Al-NayeemCS 525 Spring 2009 Advanced Distributed SystemsLarge Distributed SystemsInfrastructurePlanetLab: 971 nodes in 485 sitesApplicationHadoop at Yahoo!: 4000 nodes2Google probably has more than 450,000 servers worldwide (Wikipedia)Not only nodes, data processed in commercial systems, e.g. Facebook isenormous (over 10 billion picture uploaded).3/12/2009 Department of Computer Science, UIUCMonitoring and Management•Monitoring and management of both infrastructures and applications–Corrective measures against failure, attacks, etc.–Ensuring better performance, e.g. load balancing•What resources are managed?–Distributed application processes, objects (log files, routing table, etc.)–System resources: CPU utilization, free disk space, bandwidth utilization33/12/2009 Department of Computer Science, UIUCManagement and Monitoring Operations•Query current system status–CPU utilization, disk space..–Process progress rate.. •Push software updates–Install the query program•Monitor dynamically changing state?n1n2n3n4n5n643/12/2009 Department of Computer Science, UIUCChallenges•Managing today’s large-scale systems is difficult.–A centralized solution doesn’t scale (no in-network aggregation)–Self-organization capability is becoming a necessity–Responses are expected in seconds, not in minutes/hours.–Node failure causes inconsistent results (network partition)•Brewer’s conjecture:–It is impossible for a web service to provide the following three guarantees: Consistency, Availability, Partition-tolerance (CAP Dilemma)53/12/2009 Department of Computer Science, UIUCAstrolabe: A Robust and Scalable Technology for Distributed System Monitoring, Management, and Data MiningPresented by:Abdullah Al-NayeemRobbert Van Renesse, Kenneth P. Birman, and Werner VogelsOverview•Astrolabe as an information management service.–Locates and collects the status of a set of servers.–Reports the summaries of this information (aggregation mechanism using SQL). –Automatically updates and reports any changed summaries. •Design principles:–Scalability through hierarchy of resources–Robustness through gossip protocol (p2p)–Flexibility through customization of queries (SQL)–Security through certificates73/12/2009 Department of Computer Science, UIUCn8Astrolabe Zone Hierarchyn1n2n3n4n5n7n6/berkeley/cornell/uiuc/uiuc/csZone:Zone:Zone:Zone:83/12/2009 Department of Computer Science, UIUCAstrolabe Zone Hierarchy (2)/uiuc/ece/n1 /uiuc/cs/n4 /uiuc/cs/n6 /cornell/n2 /cornell/cs/n3/berkeley/eecs/n5/berkeley/eecs/n7/berkeley/eecs/n8/uiuc/ece/uiuc/cs/cornell/cs/berkeley/eecs/uiuc /cornell /berkeley/It’s a virtual hierarchy. Only the host in leaf zone runs an Astrolabe agent- Zone hierarchy is determined by the administrators (less flexibility).- Assumption: zone names are consistent with the physical topology.- Zone hierarchy is determined by the administrators (less flexibility).- Assumption: zone names are consistent with the physical topology.93/12/2009 Department of Computer Science, UIUCDecentralized Hierarchical Database•An attribute list is associated with each zone.–This attribute list is defined as Management Information Base (MIB)•Attributes includes information on load, total free disk space, process information, etc.•Each internal zone has a relational table of MIBs of its child zones.–The leaf zone is an exception (.. next slide)103/12/2009 Department of Computer Science, UIUCDecentralized Hierarchical Database (2)/uiuc/ece/n1 /uiuc/cs/n4 /uiuc/cs/n6/uiuc/ece/uiuc/cs/uiuc/cornell/berkeley/cseceuiuccornellberkeley11… …n4n6Load = 0.1Load = 0.3systemprocessLoad = 0.1Disk = 1.2TBService: A(1.1) progress = 0.7filessystemprocessLoad = 0.3Disk = 0.6TBService: A(1.0) progress = 0.5files3/12/2009 Department of Computer Science, UIUCAgent (/uiuc/cs/n6) has its local copy ofthese management table of MIBs.Agent (/uiuc/cs/n6) has its local copy ofthese management table of MIBs.State Aggregationuiuccornellberkeley/uiuc/cs/n4/uiuc/cs/uiuc/csecen4n6Load = 0.3Load = 0.5(Own)(Own)(Own)SELECT MIN(Load) as LoadLoad = 0.3Time = 121Time = 101Time = 130Other aggregation includes:MAX (attribute)SUM (attribute)AVG (attribute)FIRST(n, attribute)12Aggregates the resultusing SQL query3/12/2009 Department of Computer Science, UIUCState Merge – Gossip Protocoluiuccornellberkeley/uiuc/cs/n4/uiuc/cs/uiuc/csecen4n6Load = 0.3Load = 0.5Load = 0.3(Own)(Own)(Own)/uiuc/cs/n6/uiuc/cs/uiuc/Time = 121Time = 101uiuccornellberkeleycsecen4n6Load = 0.5Load = 0.5Load = 0.5(Own)(Own)(Own)Time = 101Time = 101Each agent periodically contacts some other agent and exchangesthe state associated with MIB based on timestamp.Time = 130 Time = 110Time = 121Load = 0.3133/12/2009 Department of Computer Science, UIUCMore about Astrolabe Gossip/uiuc/ece/n1 /uiuc/cs/n4 /uiuc/cs/n6 /cornell/n2 /cornell/cs/n3/berkeley/eecs/n5/berkeley/eecs/n7/berkeley/eecs/n8/uiuc/ece/uiuc/cs/cornell/cs/berkeley/eecs/uiuc /cornell /berkeley/uiuccornellberkeleycseceHow does /uiuc/cs/n4 know the MIB of /cornell? Gossiped MIBs in /uiuc/cs14By gossiping with /cornell/n2 or


View Full Document

U of I CS 525 - Distributed Monitoring and Management

Documents in this Course
Epidemics

Epidemics

12 pages

LECTURE

LECTURE

7 pages

LECTURE

LECTURE

39 pages

LECTURE

LECTURE

41 pages

P2P Apps

P2P Apps

49 pages

Lecture

Lecture

48 pages

Epidemics

Epidemics

69 pages

GRIFFIN

GRIFFIN

25 pages

Load more
Download Distributed Monitoring and Management
Our administrator received your request to download this document. We will send you the file to your email shortly.
Loading Unlocking...
Login

Join to view Distributed Monitoring and Management and access 3M+ class-specific study document.

or
We will never post anything without your permission.
Don't have an account?
Sign Up

Join to view Distributed Monitoring and Management 2 2 and access 3M+ class-specific study document.

or

By creating an account you agree to our Privacy Policy and Terms Of Use

Already a member?