iKen BirmanCornell University. CS5410 Fall 2008. Welcome to CS5140!y A course on cloud computing, edge computing, and related systems technologiesW’ i bk i b Pf Bi ( yWe’re using a textbook written by Professor Birman, (a bit out of date). Copies on reserve.yGrading mostly based on three assignments aimed at yGrading mostly based on three assignments aimed at hands‐on experience with the things we’ re learning in classy Background: Ja va or C++ (or C#), familiar with threads, comfortable writing programs, had an hi d i architecture course and an operating systems course.Two side‐by‐side revolutionsy Cloud computing: trend is to move more and more computing functions into large shared data centersA EC “h ” d f yAmazon EC2 “hosts” data centers for customersy Google runs all sorts of office applications, email, etc on their systemstheir systemsy Yahoo! wants to be a one‐source computing solutiony IBM has a vision of computing “like electric power”y Edge computing: direct interactions among computers (peers) out in the Internety For example, multi‐user games, VR immersionCloud Computing ConceptEmail, file storage, IM, searchDatabases, spreadsheets, office appsClient systems use web technologiesWeb servicesweb technologiesGoogle/IBM/Amazon/Yahoo! host the servicesWeb servicesSupporting technologiesy Infrastructurey Core management and hdli ftiy Cloud “enablers”y Map‐Reduceblscheduling functionsy Event notification servicesyStorage systems (GFS)y BigTabley AstrolabeyAmazon’s shopping cartyStorage systems (GFS)y Monitoring, debugging, tuning assistanceyAmazons shopping cartyEven higher level?y Increasingly: virtualizationEven higher level?y Tools for building and analyzing massive graphs… Sadly, we can’t do everything!y In CS5140 we don’t have time to cover all of these topics, so we’ll focus on infrastructure toolsY ’ bild hi lik MRd ih h!yYou can’t build things like Map‐Reduce without them!y But you won’t learn to use Hadoop (a popular open‐source Map‐Reduce implementation) in this classsource MapReduce implementation) in this classy Even within the infrastructure space, we’ll pick and p, pchoose our topics to get at some of the key ideasy Secondary issue: we also want to look at the edgeWill the next big thing happen g g ppon the edge of the network? VR ii Di t ib t d …. VR immersion… Distributed programming by “drag and drop”http://liveobjects.cs.cornell.eduhttp://liveobjects.cs.cornell.eduLive objects are…y An integration tool –a “thin” layer that lets us glue components together into event‐driven applicationsA ki d f “d d d” i lyA kind of “drag and drop” programming tooly Common framework unifies replication technologiesEl Ali iExample Applications¾ Photo sharing that works ¾ Games and virtual worlds¾Collaborationtools¾Emergencyresponse¾Collaborationtools¾Emergencyresponse¾ Office automation ¾ Mobile services¾New Internet Services¾Coordinated planning¾New Internet Services¾Coordinated planning¾ Interactive television ¾ Social networkingBut they also depend on data y pcenter resourcesy Data centers host maps, databases, rendering softwarey Think of the “static” content as coming from a data center, and streams of events reflecting real‐time content coming directly from sensors and “synthetic content coming directly from sensors and synthetic content sources”, combined on your end‐user nodey All of this needs to scale to massive deploymentsIt’s a big, big g, gworld out onworld out on the edgethe edge….Our goals toda yy In CS5140, we’ll peel back the coversy Try and understand major technologies used to il t ld ti ltfimplement cloud computing platformsy How did IBM/Amazon/Google/etc build their cloud computing infrastructure?y What tools do all of these systems employ? How are they implemented, and what are the cost/performance tradeoffs?yHow robust are they?How robust are they?y And also, how to build your own cloud applicationsy Key issue: to scale well, they need to replicate functionalityy The underlying standards: Web Services and CORBAHow does this overlap with pedge technologies?y The edge is a world of peer‐to‐peer solutionsy BitTorrent, Napster/Gnutella, PPLiv e, Skype, and even Li Obj tLive Objectsy How are these built? What issues need to be addressed when systems live out in the wild (in the Internet)?y ( )y But those edge solutions are invariably supported by some kind of cloud service, and in the future the integration is going to become more significantWh h h f d li ld yWhat happens when we graft edge solutions to cloud platforms?Connecting the cloud to the edgey The cloud is a good place toy Store massive amounts of contenty Keep precompute d information, account informationy Run scalable servicesy The edge is a good place toyCapture data from the real world (sensors cameras )yCapture data from the real world (sensors, cameras…)y Share high‐rate video, voice, event streams, “updates”y Support direct collaboration, interactionpp ,Topics we’ll cover[//8] Wb Si d SOA [//8] Til b d Wb y[9/3/08] Web Ser vices and SOA standards. C ORBA and OO standards y [9/8/08] Key components of cloud computing platforms y[9/10/08] Cloud computing applications and y[10/15/08] Transactional subsystems and Web Ser v ices support for the transactional model y [10/20/08] How transactional servers are implemented y[10/22/08] Gossipbased replication and system y[9/10/08] Cloud computing applications and Map‐Reduc e y [9/15/08] Thinking about distributed systems: Models of time and event ordering y[9/17/08] Clock synchronization and the limits of y[10/22/08] Gossip‐based replication and system monitoring. Astrolabe y [10/27/08] DHTs. Chord, Pastry, Kelipsy [10/29/08] T‐Man y[11/03/08] Trusted computing issues seen in cloud y[9/17/08] Clock synchronization and the limits of real‐time y [9/22/08] Consensus on event ordering: The GMS Ser v ice(1) y[9/24/08] The GMS Service(2) y[11/03/08] Trusted computing issues seen in cloud settings. Practical Byzantine Agreement y [11/05/08] Interconnecting cloud platforms with Maelstrom. Mirrored file systems. y[11/10/08] Life on the Edge: BrowsersBitTorrent[9/ 4/ ] ()y [9/29/08] State machine concept. Possible functionality that our GMS can support y [10/1/08] Replication: basic goals. Ricochet y[10/6/08]