1CS514: Intermediate Course in Computer SystemsLecture 8: Sept. 24, 2003Cluster ComputingGuest lecture from Werner VogelsToday:Cluster ComputingWerner VogelsDept. of Computer ScienceCornell University2CS514Agenda| History of Cluster Computing| Major Cluster Systems| Technical Challenges| Software Architectures| Cluster Management Systemsz MSCSz Galaxy| Student Research Projects in Cluster ComputingCS514What do I want you to know?| Why and how clusters are used?| What is the difference between parallel and enterprise cluster computing?| What are the major issues in hardware and software?| What is a Cluster Management System?| What do I need to do to work on a cluster computing project myself?3CS514How to get more done …| Work Harder| Work Faster| Get Help| Processor Speed| Algorithms| Parallel processingCS514Some History| Von Neuman thought parallelism to be impossible| ILLIAC IV – first massive parallel machine (Illinois ’60)| Japan’s 5thGeneration Project| USA – Grand Challenges| Commercial: NCR, IBM Fijustu, Intel SSD, Gray, Convex4CS514Traditional Users| Scientists – investigate the unknown| Engineers – simulations| Retailers – data mining| Airlines – how to overbook| Financial – gaining 0.1% advantage| ……CS514The collapse of the Supercomputing Industry| ’97 the industry icon Cray Research went bankrupt.| Many reasons were given, among which the end of the cold war5CS514The Real Reasons| Microprocessors got fast, a lot faster| High Availability became a mass market.CS514Some early examples| Vax-Clusters / OpenVMS Cluster6CS514Tandem Himalaya| Traditional for the HA – marketCS514IBM SYSPLEX7CS514Cluster Definition| Consists of a collection of interconnected whole computers| Is used as a single, unified computing resourceCS514Distinction from Other Systems| Scaling: Adding a head or a whole dog| Availability: what if a dog breaks a leg?| System management: walking the dog| Software licensing: dog tax8CS514Technical Challenges - I| Cluster Hardware (NOW, rack&stack, NUMAs) | Cluster Communication (Interconnects, Communication Protocols)| Cluster System Middleware (management, availability, tools)| High-performance IO systems (storage, file systems, data placement and movement)CS514Technical Challenges - II| Job and Resource Management| Programming Environments (Distr. Objects, Message Passing)| Scalable Services.| Business frameworks (multi tiers, web based, decision support)| Applications (Scientific, High-Availability, Scalable performance)9CS514Single System Image| From the perspective ofz Userz Networkz Applicationz Administrator| Key Issues:z Each SSI has a boundaryz SSI support can exist at different levelsCS514Single System Image - II| Boundary: z Inside a single machinez Outside a collection of machines| SSI Levelsz Application z Middlewarez Operating Systemz Hardware10CS514Is Transparency a good thing?| Yes, but achieving it is close to impossible| Many transparencies were introduced in legacy code with disastrous side effects| User to cluster is possible| Server side should be avoidedCS514SwitchedEthernetSwitchedEthernetwww.m icrosoft.com(3)search.m icr osoft.com(1)premium .m icrosoft.com(1)European Data CenterFTPDownload Server(1)SQL SERVERS(2)Routermsid.m sn.c om(1)MOSWestAdmin LANSQLNetFeeder LANFDDI Ring(MIS4)Routerwww.microsoft.com(5)Building 11Live SQL ServerRouterhom e.microsoft.com(5)FDDI Ring(MIS2)www.microsoft.com(4)activex.microsoft.com(2)search.m icros oft.com(3)register.microsoft.com(2)msid.m sn.c om(1)FDDI Ring(MIS3)www.microsoft.com(3)premium .m icrosoft.com(1)msid.m sn.c om(1)FDDI Ring(MIS1)www.m icrosoft.com(4)premium .m icrosoft.com(2)register.m icrosoft.com(2)msid.m sn.com(1)PrimaryGigaswitchSecondaryGigaswitchStaging Servers(7)search.m icr os oft.com(3)support.microsoft.com(2)register.m sn.com(2)1997 Microsoft.Com Web site (4 farms).MOSW estDMZ Staging ServersLive SQL ServersSQL ConsolidatorsJapan Data Centerwww.m icrosoft.com(3)premium .m icrosoft.com(1)HTTPDownload Servers(2)Routersearch.m icr osoft.com(2)SQL SERVERS(2)msid.m sn.c om(1)FTPDownload Server(1)RouterRouterRouterRouterRouterRouterRouterRouterIntern al WWWSQL Reportinghom e.microsoft.com(4)hom e.microsoft.com(3)hom e.microsoft.com(2)register.microsoft.com(1)support.microsoft.com(1)Internet13DS3(45 Mb/Sec Each)2OC3(45Mb/Sec Each)2Ethernet(100 Mb/Sec Each)cdm.microsoft.com(1)FTP ServersDownloadReplic ationFTP.micros oft.com(3)0SwitchedEthernetSwitchedEthernetwww.m icrosoft.com(3)SwitchedEthernetSwitchedEthernetwww.m icrosoft.com(3)search.m icr osoft.com(1)premium .m icrosoft.com(1)European Data CenterFTPDownload Server(1)SQL SERVERS(2)Routermsid.m sn.c om(1)MOSW estAdmin LANSQLNetFeeder LANFDDI Ring(MIS4)Routerwww.microsoft.com(5)Building 11Live SQL ServerRouterhom e.microsoft.com(5)FDDI Ring(MIS2)www.microsoft.com(4)activex.microsoft.com(2)search.m icros oft.com(3)register.microsoft.com(2)msid.m sn.c om(1)search.m icros oft.com(3)register.microsoft.com(2)msid.m sn.c om(1)FDDI Ring(MIS3)www.microsoft.com(3)premium .m icrosoft.com(1)msid.m sn.c om(1)FDDI Ring(MIS1)www.m icrosoft.com(4)premium .m icrosoft.com(2)FDDI Ring(MIS3)www.microsoft.com(3)premium .m icrosoft.com(1)msid.m sn.c om(1)FDDI Ring(MIS1)www.m icrosoft.com(4)premium .m icrosoft.com(2)register.m icrosoft.com(2)msid.m sn.com(1)PrimaryGigaswitchSecondaryGigaswitchregister.m icrosoft.com(2)msid.m sn.com(1)PrimaryGigaswitchSecondaryGigaswitchStaging Servers(7)search.m icr os oft.com(3)support.microsoft.com(2)register.m sn.com(2)1997 Microsoft.Com Web site (4 farms).MOSW estDMZ Staging ServersLive SQL ServersSQL ConsolidatorsLive SQL ServersSQL ConsolidatorsJapan Data Centerwww.m icrosoft.com(3)premium .m icrosoft.com(1)HTTPDownload Servers(2)Routersearch.m icr osoft.com(2)SQL SERVERS(2)msid.m sn.c om(1)premium .m icrosoft.com(1)HTTPDownload Servers(2)Routersearch.m icr osoft.com(2)SQL SERVERS(2)msid.m sn.c om(1)FTPDownload Server(1)RouterRouterRouterRouterRouterRouterRouterRouterFTPDownload Server(1)RouterRouterRouterRouterRouterRouterRouterRouterIntern al WWWSQL ReportingIntern al WWWSQL Reportinghom e.microsoft.com(4)hom e.microsoft.com(3)hom e.microsoft.com(2)register.microsoft.com(1)support.microsoft.com(1)hom e.microsoft.com(4)hom e.microsoft.com(3)hom e.microsoft.com(2)register.microsoft.com(1)support.microsoft.com(1)Internet13DS3(45 Mb/Sec Each)2OC3(45Mb/Sec Each)2Ethernet(100 Mb/Sec Each)cdm.microsoft.com(1)FTP ServersDownloadReplic ationFTP.micros oft.com(3)011CS514| RACS – Reliable Array of
View Full Document
Unlocking...