Parallel Architectures HistoryCurrent Trends In Parallel ArchitecturesModern Parallel Architecture Layered FrameworkProgramming ModelsCommunication AbstractionCommunication ArchitectureToward Architectural ConvergenceConvergence of Scalable Parallel Machines: Generic Parallel ArchitectureUnderstanding Parallel ArchitectureFundamental Design IssuesSequential Programming ModelSAS Programming ModelSynchronizationMessage Passing Programming ModelDesign Issues Apply at All LayersLower Layers Support of Naming and OperationsSlide 17Lower Layers Support of OrderingLower Layers Support of ReplicationCommunication PerformanceSimple Communication Cost ExampleLinear Model of Data Transfer LatencyCommunication Cost ModelSlide 24Summary of Design IssuesEECC756 - Shaaban#1 lec # 2 Spring 2000 3-9-2000Parallel Architectures HistoryParallel Architectures HistoryApplication SoftwareSystem SoftwareSIMDMessage PassingShared MemoryDataflowSystolicArraysArchitectureHistorically, parallel architectures tied to programming models •Divergent architectures, with no predictable pattern of growth.EECC756 - Shaaban#2 lec # 2 Spring 2000 3-9-2000Current Trends In Parallel ArchitecturesCurrent Trends In Parallel Architectures •The extension of “computer architecture” to support communication and cooperation:–OLD: Instruction Set Architecture.–NEW: Communication Architecture.•Defines: –Critical abstractions, boundaries, and primitives (interfaces).–Organizational structures that implement interfaces (hardware or software).•Compilers, libraries and OS are important bridges todayEECC756 - Shaaban#3 lec # 2 Spring 2000 3-9-2000Modern Parallel ArchitectureModern Parallel ArchitectureLayered FrameworkLayered FrameworkCADMultiprogramming SharedaddressMessagepassingDataparallelDatabase Scientific modelingParallel applicationsProgramming modelsCommunication abstractionUser/system boundaryCompilationor libraryOperating systems supportCommunication hardwarePhysical communication mediumHardware/software boundaryEECC756 - Shaaban#4 lec # 2 Spring 2000 3-9-2000Programming ModelsProgramming Models•Programming methodology used in coding applications•Specifies communication and synchronization•Examples:–Multiprogramming: No communication or synchronization at program level–Shared memory address space: –Message passing: Explicit point to point communication–Data parallel: More regimented, global actions on data•Implemented with shared address space or message passingEECC756 - Shaaban#5 lec # 2 Spring 2000 3-9-2000Communication AbstractionCommunication Abstraction•User-level communication primitives provided–Realizes the programming model.–Mapping exists between language primitives of programming model and these primitives•Supported directly by hardware, or via OS, or via user software.•Lot of debate about what to support in software and gap between layers.•Today:–Hardware/software interface tends to be flat, i.e. complexity roughly uniform.–Compilers and software play important roles as bridges today.–Technology trends exert strong influence•Result is convergence in organizational structure–Relatively simple, general purpose communication primitives.EECC756 - Shaaban#6 lec # 2 Spring 2000 3-9-2000Communication ArchitectureCommunication Architecture = User/System Interface + Implementation•User/System Interface:–Communication primitives exposed to user-level by hardware and system-level software.•Implementation:–Organizational structures that implement the primitives: hardware or OS.–How optimized are they? How integrated into processing node?–Structure of network.•Goals:–Performance–Broad applicability–Programmability–Scalability–Low CostEECC756 - Shaaban#7 lec # 2 Spring 2000 3-9-2000Toward Architectural ConvergenceToward Architectural Convergence•Evolution and role of software have blurred boundary:–Send/receive supported on SAS machines via buffers.–Can construct global address space on massively parallel (MP) message-passing machines by carrying along pointers specifying the process and local virtual address space.–Shared virtual address space in message-passing machines can also be established at the page level generating a page fault for remote pages handled by sending a message.•Hardware organization converging too:–Tighter integration even for MP (low-latency, high-bandwidth):•Network interface tightly integrated with memory/cache controller.•Transfer data directly to/from user address space.•DMA transfers across the network.–At lower level, even hardware SAS passes hardware messages.•Even clusters of workstations/SMPs are becoming parallel systems:–Emergence of fast system area networks (SAN): ATM, fiber channel ...•Programming models distinct, but organizations converging:–Nodes connected by general network and communication assists.–Implementations also converging, at least in high-end machines.EECC756 - Shaaban#8 lec # 2 Spring 2000 3-9-2000Convergence of Scalable Parallel Machines: Convergence of Scalable Parallel Machines: Generic Parallel ArchitectureGeneric Parallel Architecture•A generic modern multiprocessor:Node: processor(s), memory system, plus communication assist:•Network interface and communication controller.•Scalable network:•Convergence allows lots of innovation, now within framework•Integration of assist with node, what operations, how efficiently...MemNetworkP$Communicationassist (CA)EECC756 - Shaaban#9 lec # 2 Spring 2000 3-9-2000Understanding Parallel ArchitectureUnderstanding Parallel Architecture•Traditional taxonomies not very useful.•Programming models are not enough, nor hardware structures.–Can be supported by radically different architectures.•Architectural distinctions that affect software–Compilers, libraries, programs.•Design of user/system and hardware/software interface–Constrained from above by programming models and below by technology.•Guiding principles provided by layers.–What primitives are provided at communication abstraction.–How programming models map to these.–How they are mapped to hardware.EECC756 - Shaaban#10 lec # 2 Spring 2000 3-9-2000Fundamental Design IssuesFundamental Design Issues•At any layer, interface (contract) aspect and performance aspects:–Naming: How are logically shared data and/or