Parallel Architecture Fundamentals CS 740 September 22 2003 Topics What is Parallel Architecture Why Parallel Architecture Evolution and Convergence of Parallel Architectures Fundamental Design Issues What is Parallel Architecture A parallel computer is a collection of processing elements that cooperate to solve large problems fast Some broad issues Resource Allocation how large a collection how powerful are the elements how much memory Data access Communication and Synchronization how do the elements cooperate and communicate how are data transmitted between processors what are the abstractions and primitives for cooperation Performance and Scalability how does it all translate into performance how does it scale 2 Why Study Parallel Architecture Role of a computer architect To design and engineer the various levels of a computer system to maximize performance and programmability within limits of technology and cost CS 740 F 03 Why Study it Today History diverse and innovative organizational structures often tied to novel programming models Rapidly maturing under strong technological constraints The killer micro is ubiquitous Laptops and supercomputers are fundamentally similar Technological trends cause diverse approaches to converge Parallelism Provides alternative to faster clock for performance Applies at all levels of system design Is a fascinating perspective from which to view architecture Is increasingly central in information processing 3 Technological trends make parallel computing inevitable In the mainstream Need to understand fundamental principles and design tradeoffs not just taxonomies Naming Ordering Replication Communication performance CS 740 F 03 4 CS 740 F 03 Conventional Processors No Longer Scale Performance by 50 each year 1e 7 1e 7 52 ye ps ar g a 1e 6 1e 5 1e 4 1e 3 1e 2 Future potential of novel architecture is large 1000 vs 30 52 ye ar 1e 6 Perf ps Inst 1e 5 1e 4 te Ga 19 te s c Clo l oc ck k9 s ins t1 8 Perf ps Inst Delay CPUs 1e 3 30 1 74 yea r 1e 2 1e 1 19 yea r 1e 0 19 ye ar 1 000 1 30 000 1 1e 1 1e 2 1e 1 1e 3 1e 0 1980 1990 2000 2010 2020 1e 4 1980 Bill Dally 5 1990 2000 2010 2020 Bill Dally CS 740 F 03 6 Inevitability of Parallel Computing CS 740 F 03 Application Trends Application demands Our insatiable need for cycles Demand for cycles fuels advances in hardware and vice versa Technology Trends Range of performance demands Architecture Trends Goal of applications in using parallel machines Speedup Scientific computing CFD Biology Chemistry Physics General purpose computing Video Graphics CAD Databases TP Need range of system performance with progressively increasing cost Platform pyramid Number of transistors on chip growing rapidly Clock rates expected to go up only slowly Instruction level parallelism valuable but limited Coarser level parallelism as in MPs the most viable approach Speedup p processors Performance p processors Performance 1 processor For a fixed problem size input data set performance 1 time Economics Current trends Today s microprocessors have multiprocessor support Servers even PCs becoming MP Sun SGI COMPAQ Dell Tomorrow s microprocessors are multiprocessors 7 Cycle drives exponential increase in microprocessor performance Drives parallel architecture harder most demanding applications CS 740 F 03 Speedup fixed problem p processors 8 Time 1 processor Time p processors CS 740 F 03 Scientific Computing Demand Engineering Computing Demand Large parallel machines a mainstay in many industries Petroleum reservoir analysis Automotive crash simulation drag analysis combustion efficiency Aeronautics airflow analysis engine efficiency structural mechanics electromagnetism Computer aided design Pharmaceuticals molecular modeling Visualization in all of the above entertainment films like Toy Story architecture walk throughs and rendering Financial modeling yield and derivative analysis etc 9 CS 740 F 03 10 Learning Curve for Parallel Programs CS 740 F 03 Commercial Computing Also relies on parallelism for high end Scale not so large but use much more wide spread Computational power determines scale of business that can be handled Databases online transaction processing decision support data mining data warehousing TPC benchmarks TPC C order entry TPC D decision support AMBER molecular dynamics simulation program Starting point was vector code for Cray 1 145 MFLOP on Cray90 406 for final version on 128processor Paragon 891 on 128 processor Cray T3D 11 CS 740 F 03 Explicit scaling criteria provided Size of enterprise scales with size of system Problem size no longer fixed as p increases so throughput is used as a performance measure transactions per minute or tpm 12 CS 740 F 03 TPC C Results for March 1996 Summary of Application Trends 25 000 20 000 Throughput tpmC Transition to parallel computing has occurred for scientific and engineering computing In rapid progress in commercial computing Tandem Himalaya DEC Alpha SGI PowerChallenge HP PA IBM PowerPC Other Database and transactions as well as financial Usually smaller scale but large scale systems also used 15 000 Desktop also uses multithreaded programs which are a lot like parallel programs Demand for improving throughput on sequential workloads 10 000 5 000 0 0 20 40 60 80 100 120 Greatest use of small scale multiprocessors Number of processors Parallelism is pervasive Small to moderate scale parallelism very important Difficult to obtain snapshot to compare across vendor platforms 13 CS 740 F 03 Solid application demand exists and will increase 14 Technology Trends Performance 100 CS 740 F 03 Architectural Trends Architecture translates technology s gifts to performance and capability Resolves the tradeoff between parallelism and locality Supercomputers Current microprocessor 1 3 compute 1 3 cache 1 3 off chip connect Tradeoffs may change with scale and technology advances 10 Understanding microprocessor architectural trends Mainframes Helps build intuition about design issues or parallel machines Shows fundamental role of parallelism even in sequential computers Microprocessors Minicomputers 1 0 1 1965 Four generations of architectural history tube transistor IC VLSI Here focus only on VLSI generation 1970 1975 1980 1985 1990 Commodity microprocessors have caught up with supercomputers 15 CS 740 F 03 1995 Greatest delineation in VLSI has been in type of parallelism exploited 16 CS 740 F 03 Arch Trends Exploiting Parallelism Phases in VLSI Generation Greatest trend in VLSI
View Full Document
Unlocking...