1CSCI 8150Advanced Computer ArchitectureHwang, Chapter 1Parallel Computer Models1.2 Multiprocessors and MulticomputersCategories of Parallel ComputersConsidering their architecture only, there are two main categories of parallel computers:systems with shared common memories, andsystems with unshared distributed memories. Shared-Memory MultiprocessorsShared-memory multiprocessor models:Uniform-memory -access (UMA)Nonuniform-memory -access (NUMA)Cache-only memory architecture (COMA)These systems differ in how the memory and peripheral resources are shared or distributed.The UMA Model - 1Physical memory uniformly shared by all processors, with equal access time to all words.Processors may have local cache memories.Peripherals also shared in some fashion.Tightly coupled systems use a common bus, crossbar, or multistage network to connect processors, peripherals, and memories.Many manufacturers have multiprocessor (MP) extensions of uniprocessor (UP) product lines.The UMA Model - 2Synchronization and communication among processors achieved through shared variables in common memory.Symmetric MP systems – all processors have access to all peripherals, and any processor can run the OS and I/O device drivers.Asymmetric MP systems – not all peripherals accessible by all processors; kernel runs only on selected processors (master); others are called attached processors (AP).The UMA Multiprocessor ModelP1P2Pn…System Interconnect(Bus, Crossbar, Multistage network)I/O SM1…SMm2Example: Performance CalculationConsider two loops. The first loop adds corresponding elements of two N-element vectors to yield a third vector. The second loop sums elements of the third vector. Assume each add/assign operation takes 1 cycle, and ignore time spent on other actions (e.g. loop counter incrementing/testing, instruction fetch, etc.). Assume interprocessor communication requires kcycles.On a sequential system, each loop will require N cycles, for a total of 2N cycles of processor time.Example: Performance CalculationOn an M-processor system, we can partition each loop into M parts, each having L = N / M add/assigns requiring Lcycles. The total time required is thus 2L. This leaves us with M partial sums that must be totaled.Computing the final sum from the M partial sums requires l= log2(M) additions, each requiring k cycles (to access a non-local term) and 1 cycle (for the add/assign), for a total of l × (k+1) cycles.The parallel computation thus requires2N / M + (k + 1) log2(M) cycles.Example: Performance CalculationAssume N = 220.Sequential execution requires 2N = 221cycles.If processor synchronization requires k = 200 cycles, and we have M = 256 processors, parallel execution requires2N / M + (k + 1) log2(M)= 221/ 28+ 201 × 8= 213+ 1608 = 9800 cyclesComparing results, the parallel solution is 214 times faster than the sequential, with the best theoretical speedup being 256 (since there are 256 processors). Thus the efficiency of the parallel solution is 214 / 256 = 83.6 %.The NUMA Model - 1Shared memories, but access time depends on the location of the data item.The shared memory is distributed among the processors as local memories, but each of these is still accessible by all processors (with varying access times).Memory access is fastest from the locally -connected processor, with the interconnection network adding delays for other processor accesses.Additionally, there may be global memory in a multiprocessor system, with two separate interconnection networks, one for clusters of processors and their cluster memories, and another for the global shared memories.Shared Local MemoriesP1P2Pn...LM1Inter-connectionNetworkLM2LMn...Hierarchical Cluster ModelGSM …Global Interconnect NetworkGSM GSMPPP...CINCSM...CSMCSM…PPP...CINCSM...CSMCSM3The COMA ModelIn the COMA model, processors only have cache memories; the caches, taken together, form a global address space.Each cache has an associated directory that aids remote machines in their lookups; hierarchical directories may exist in machines based on this model.Initial data placement is not critical, as cache blocks will eventually migrate to where they are needed.Cache-Only Memory ArchitectureInterconnection Network…CDPCDPCDPOther ModelsThere can be other models used for multiprocessor systems, based on a combination of the models just presented. For example:cache-coherent non-uniform memory access (each processor has a cache directory, and the system has a distributed shared memory)cache-coherent cache-only model (processors have caches, no shared memory, caches must be kept coherent).Multicomputer ModelsMulticomputers consist of multiple computers, or nodes, interconnected by a message-passing network.Each node is autonomous, with its own processor and local memory, and sometimes local peripherals.The message-passing network provides point-to-point static connections among the nodes.Local memories are not shared, so traditional multicomputers are sometimes called no-remote-memory-access (or NORMA) machines.Inter-node communication is achieved by passing messages through the static connection network.Generic Message-Passing MulticomputerMessage-passinginterconnection networkM PM P P MP MPMPMPMPM……Multicomputer GenerationsEach multicomputer uses routers and channels in its interconnection network, and heterogeneous systems may involved mixed node types and uniform data representation and communication protocols.First generation: hypercube architecture, software-controlled message switching, processor boards.Second generation: mesh-connected architecture, hardware message switching, software for medium-grain distributed computing.Third generation: fine-grained distributed computing, with each VLSI chip containing the processor and communication resources.4Multivector and SIMD ComputersVector computers often built as a scalar processor with an attached optional vector processor.All data and instructions are stored in the central memory, all instructions decoded by scalar control unit, and all scalar instructions handled by scalar processor.When a vector instruction is decoded, it is sent to the vector processor’s control unit which supervises the flow of data and execution of the instruction.Vector Processor ModelsIn register-to-register models, a fixed number of possibly reconfigurable registers are used to hold all vector operands, intermediate, and final vector results. All registers are accessible in user instructions.In a