CS 525 Advanced Distributed Systems Spring 2011Slide 2So What’s new in Today’s Clouds?Slide 4Single site Cloud: to Outsource or Own? [OpenCirrus paper]Single site Cloud: to Outsource or Own?Slide 710 Challenges [Above the Clouds]A more Bottom-Up View of Open Research DirectionsSlide 10New Parallel Programming Paradigms: MapReduceWhat is MapReduce?MapSlide 14Slide 15ReduceSlide 17Some Other Applications of MapReduceProgramming MapReduceInside MapReduceInternal Workings of MapReduceFault ToleranceLocality and Backup tasksGrepSlide 25Slide 26Slide 27EmulabAnd then there were… Grids!Example: Rapid Atmospheric Modeling System, ColoState UDistributed Computing ResourcesAn Application Coded by a PhysicistSlide 33Slide 34Slide 35Slide 36Slide 37Tiered Architecture (OSI 7 layer-like)The Grid RecentlyGlobus AllianceSome Things Grid Researchers Consider ImportantSlide 42Entr. Tidbits: Where to get your ideas from (for your project)Administrative Announcements11Yeah! That’s what I’d like to know.Indranil Gupta (Indy)Lecture 3Cloud Computing ContinuedJanuary 25, 2011CS 525 Advanced Distributed SystemsSpring 2011All Slides © IG(Acknowledgments: Md Yusuf Sarwar and Md Ahsan Arefin)2Timesharing Industry (1975):•Market Share: Honeywell 34%, IBM 15%, •Xerox 10%, CDC 10%, DEC 10%, UNIVAC 10%•Honeywell 6000 & 635, IBM 370/168, Xerox 940 & Sigma 9, DEC PDP-10, UNIVAC 1108Grids (1980s-2000s):•GriPhyN (1970s-80s)•Open Science Grid and Lambda Rail (2000s)•Globus & other standards (1990s-2000s)First large datacenters: ENIAC, ORDVAC, ILLIACMany used vacuum tubes and mechanical relaysP2P Systems (90s-00s)•Many Millions of users•Many GB per dayData Processing Industry - 1968: $70 M. 1978: $3.15 Billion.Berkeley NOW ProjectSupercomputersServer Farms (e.g., Oceano)“A Cloudy History of Time” © IG 2010Clouds3So What’s new in Today’s Clouds?Besides massive scale, three major features:I. On-demand access: Pay-as-you-go, no upfront commitment.–Anyone can access it (e.g., Washington Post – Hillary Clinton example)II. Data-intensive Nature: What was MBs has now become TBs.–Daily logs, forensics, Web data, etc.–Do you know the size of Wikipedia dump?III. New Cloud Programming Paradigms: MapReduce/Hadoop, Pig Latin, DryadLinq, Swift, and many others.–High in accessibility and ease of programmability Combination of one or more of these gives rise to novel and unsolved distributed computing problems in cloud computing.4OK, so that’s what a cloud looks like today. Now, suppose I want to start my own company, Devils Inc. Should I buy a cloud and own it, or should I outsource to a public cloud?45Single site Cloud: to Outsource or Own? [OpenCirrus paper]•Medium-sized organization: wishes to run a service for M months–Service requires 128 servers (1024 cores) and 524 TB –Same as UIUC CCT cloud site•Outsource (e.g., via AWS): monthly cost–S3 costs: $0.12 per GB month. EC2 costs: $0.10 per Cpu hour–Storage ~ $62 K–Total ~ $136 K•Own: monthly cost–Storage ~ $349 K / M–Total ~ $ 1555 K / M + 7.5 K (includes 1 sysadmin / 100 nodes)•using 0.45:0.0.4:0.15 split for hardware:power:network56Single site Cloud: to Outsource or Own?•Breakeven analysis: more preferable to own if:–$349 K / M < $62 K (storage)–$ 1555 K / M + 7.5 K < $136 K (overall)Breakeven points–M > 5.55 months (storage)•Not surprising: Cloud providers benefit monetarily most from storage–M > 12 months (overall)•Assume hardware lasts for 3 years (typical lifetime)•Systems are typically underutilized•With system utilization of x%, still more preferable to own if:–x > 33.3%–Even with CPU util of 20% (typical value for industry clouds, e.g., at Yahoo), storage > 47% makes owning preferable 67I want to do research in this area. I am sure there are no grand challenges in cloud computing!7810 Challenges [Above the Clouds](Index: Performance Data-related Scalability Logisitical)•Availability of Service: Use Multiple Cloud Providers; Use Elasticity; Prevent DDOS•Data Lock-In: Enable Surge Computing; Standardize APIs•Data Confidentiality and Auditability: Deploy Encryption, VLANs, Firewalls, Geographical Data Storage•Data Transfer Bottlenecks: Data Backup/Archival; Higher BW Switches; New Cloud Topologies; FedExing Disks•Performance Unpredictability: QoS; Improved VM Support; Flash Memory; Schedule VMs•Scalable Storage: Invent Scalable Store•Bugs in Large Distributed Systems: Invent Debuggers; Real-time debugging; predictable pre-run-time debugging•Scaling Quickly: Invent Good Auto-Scalers; Snapshots for Conservation•Reputation Fate Sharing•Software Licensing: Pay-for-use licenses; Bulk use sales89A more Bottom-Up View of Open Research DirectionsMyriad interesting problems that acknowledge the characteristics that make today’s cloud computing unique: massive scale + on-demand + data-intensive + new programmability + and infrastructure- and application-specific details.Monitoring: of systems&applications; single site and multi-siteStorage: massive scale; global storage; for specific apps or classesFailures: what is their effect, what is their frequency, how do we achieve fault-tolerance?Scheduling: Moving tasks to data, dealing with federationCommunication bottleneck: within applications, within a siteLocality: within clouds, or across themCloud Topologies: non-hierarchical, other hierarchicalSecurity: of data, of users, of applications, confidentiality, integrityAvailability of DataSeamless Scalability: of applications, of clouds, of data, of everythingInter-cloud/multi-cloud computationsSecond Generation of Other Programming Models? Beyond MapReduce!Pricing ModelsExplore the limits today’s of cloud computing910Alright. But, I bet that if I a have a ton of data to process, it is very difficult to write a program for it!11New Parallel Programming Paradigms: MapReduce•Highly-Parallel Data-Processing•Originally designed by Google (OSDI 2004 paper)•Open-source version called Hadoop, by Yahoo!–Hadoop written in Java. Your implementation could be in Java, or any executable•Google (MapReduce)–Indexing: a chain of 24 MapReduce jobs–~200K jobs processing 50PB/month (in 2006)•Yahoo! (Hadoop + Pig)–WebMap: a chain of 100 MapReduce jobs–280 TB of data, 2500 nodes, 73 hours•Annual Hadoop Summit: 2008 had 300 attendees, 2009 had 700 attendeesWhat is MapReduce?•Terms are borrowed from Functional Language
View Full Document
Unlocking...