CISC 879 : Software Support for Multicore Architectures John Cavazos!Dept of Computer & Information Sciences!University of Delaware!!www.cis.udel.edu/~cavazos/cisc879!Lecture 4!Patterns for !Parallel Programming!CISC 879 : Software Support for Multicore Architectures Lecture Overview • Writing a Parallel Program • Design Patterns for Parallel Programs • Finding Concurrency • Algorithmic Structure • Supporting Structures • Implementation Mechanisms Slide Source: Dr. Rabbah, IBM, MIT Course 6.189 IAP 2007CISC 879 : Software Support for Multicore Architectures Parallelization Common Steps Slide Source: Dr. Rabbah, IBM, MIT Course 6.189 IAP 2007 1. Study problem or code 2. Look for parallelism opportunities 3. Try to keep all cores busy doing useful workCISC 879 : Software Support for Multicore Architectures Decomposition Slide Source: Dr. Rabbah, IBM, MIT Course 6.189 IAP 2007 • Identify concurrency • Decide level to exploit it • Requires understanding the algorithm! • May require restructing program/algorithm • May require entirely new algorithm • Break computation into tasks • Divided among processes • Tasks may become available dynamically • Number of tasks can vary with time Want enough tasks to keep processors busy.CISC 879 : Software Support for Multicore Architectures Assignment Slide Source: Dr. Rabbah, IBM, MIT Course 6.189 IAP 2007 • Specify mechanism to divide work • Balance of computation • Reduce communication • Structured approaches work well • Code inspection and understanding algorithm • Using design patterns (second half part of lecture)CISC 879 : Software Support for Multicore Architectures Granularity • Ratio of computation and communication • Fine-grain parallelism • Coarse-grain parallelism Most efficient granularity depends on algorithm/hardware.CISC 879 : Software Support for Multicore Architectures Fine-grain Parallelism • Tasks execute little comp. between comm. • Easy to load balance • If too fine, comm. may take longer than comp.CISC 879 : Software Support for Multicore Architectures Coarse-grain Parallelism • Long computations between communication • More opportunity for performance increase • Harder to load balanceCISC 879 : Software Support for Multicore Architectures Orchestration and Mapping Slide Source: Dr. Rabbah, IBM, MIT Course 6.189 IAP 2007 • Computation and communication concurrency • Preserve locality of data • Schedule task to satisfy dependencesCISC 879 : Software Support for Multicore Architectures Lecture Overview • Parallelizing a Program • Design Patterns for Parallelization • Finding Concurrency • Algorithmic Structure • Supporting Structures • Implementation MechanismsCISC 879 : Software Support for Multicore Architectures Patterns for Parallelization • Parallelization is a difficult problem • Hard to fully exploit parallel hardware • Solution: Design PatternsCISC 879 : Software Support for Multicore Architectures What are Design Patterns? • Cookbook for parallel programmers • Can lead to high quality solutions • Provides a common vocabulary • Each pattern has a name and associated vocabulary for discussing solutions • Helps with software reusability and modularityCISC 879 : Software Support for Multicore Architectures Architecture Patterns • Christopher Alexander • Berkeley architecture professor • Developed patterns for architecture • City planning • Layout of windows in a room • Attempt to capture principles for “living” designsCISC 879 : Software Support for Multicore Architectures Patterns for OOP • First to bring pattens to CS • Design Patterns: Elements of Reusable Object-Oriented Software (1994) • Gamma et al. (Gang of Four) • Catalogue of “patterns” • Solutions to common problems in software design • Not a finished solution! • Rather a template for how to solve a problemCISC 879 : Software Support for Multicore Architectures Patterns Parallelization Book • Patterns for Parallel Programming. • Mattson et al. (2005) • Four Design Spaces • Finding Concurrency • Expose concurrent task or data • Algorithm Structure • Map tasks to processes • Supporting Structures • Code and data structuring patterns • Implementation Mechanisms • Low-level mechanisms for writing programsCISC 879 : Software Support for Multicore Architectures Finding Concurrency • Decomposition • Data, Task, Pipeline • Dependency Analysis • Control dependences • Data dependences • Design Evaluation • Suitability for target platform • Design qualityCISC 879 : Software Support for Multicore Architectures Decomposition • Data (domain) decomposition • Break data up independent units • Task (functional) decomposition • Break problem into parallel tasks • Case for Pipeline decomposition • Special case of task decompositionCISC 879 : Software Support for Multicore Architectures Data (Domain) Decomposition • Also known as Domain Decomposition • Implied by Task Decomposition • Which decomposition more natural to start with: • 1) Decide how data elements divided among cores • 2) Decide which tasks each core should be performing • Data decomposition is good starting point when • Main computation manipulating a large data structure • Similar operations applied to different parts of a data structure (SPMD) • Example : Vector operationsCISC 879 : Software Support for Multicore Architectures Data Decomposition Slide Source: Intel Software College, Intel Corp. Find the largest element of an arrayCISC 879 : Software Support for Multicore Architectures Data Decomposition Slide Source: Intel Software College, Intel Corp. Find the largest element of an array CPU 0 CPU 1 CPU 2 CPU 3CISC 879 : Software Support for Multicore Architectures Data Decomposition Slide Source: Intel Software College, Intel Corp. Find the largest element of an array CPU 0 CPU 1 CPU 2 CPU 3CISC 879 : Software Support for Multicore Architectures Data Decomposition Slide Source: Intel Software College, Intel Corp. Find the largest element of an array CPU 0 CPU 1 CPU 2 CPU 3CISC 879 : Software Support for Multicore Architectures