CS 267 Applications of Parallel Computers Lecture 23: Load Balancing and SchedulingOutlineReview of Graph PartitioningLoad Balancing in GeneralUnderstanding Different Load Balancing ProblemsTask cost spectrumTask Dependency SpectrumTask Locality Spectrum (Data Dependencies)Spectrum of SolutionsApproachesStatic Load BalancingSemi-Static Load BalanceSelf-SchedulingWhen is Self-Scheduling a Good Idea?Variations on Self-SchedulingVariation 1: Fixed Chunk SizeVariation 2: Guided Self-SchedulingVariation 3: TaperingVariation 4: Weighted FactoringDistributed Task QueuesTheoretical ResultsEngineering Distributed Task QueuesDiffusion-Based Load BalancingDiffusion-based load balancingDAG SchedulingMixed ParallelismMixed Parallelism StrategiesWhich Strategy to UseSwitch Parallelism: A Special CaseA Simple Performance Model for Data ParallelismSlide 31Values of Sigma (Problem Size for Half Peak)Modeling performanceSimulated efficiency of Sign Function EigensolverSimulated efficiency of Sparse CholeskyActual Speed of Sign Function EigensolverCS267 L23 Load Balancing and Scheduling.1Demmel Sp 1999CS 267 Applications of Parallel ComputersLecture 23: Load Balancing and SchedulingJames Demmelhttp://www.cs.berkeley.edu/~demmel/cs267_Spr99CS267 L23 Load Balancing and Scheduling.2Demmel Sp 1999Outline°Recall graph partitioning as load balancing technique°Overview of load balancing problems, as determined by•Task costs•Task dependencies•Locality needs°Spectrum of solutions•Static - all information available before starting•Semi-Static - some info before starting•Dynamic - little or no info before starting°Survey of solutions•How each one works•Theoretical bounds, if any•When to use itCS267 L23 Load Balancing and Scheduling.3Demmel Sp 1999Review of Graph Partitioning°Partition G(N,E) so that•N = N1 U … U Np, with each |Ni| ~ |N|/p•As few edges connecting different Ni and Nk as possible°If N = {tasks}, each unit cost, edge e=(i,j) means task i has to communicate with task j, then partitioning means•balancing the load, i.e. each |Ni| ~ |N|/p•minimizing communication°Optimal graph partitioning is NP complete, so we use heuristics (see Lectures 14 and 15)•Spectral•Kernighan-Lin•Multilevel°Speed of partitioner trades off with quality of partition•Better load balance costs more; may or may not be worth itCS267 L23 Load Balancing and Scheduling.4Demmel Sp 1999Load Balancing in GeneralEnormous and diverse literature on load balancing°Computer Science systems •operating systems•parallel computing•distributed computing°Computer Science theory°Operations research (IEOR)°Application domainsA closely related problem is scheduling, which is to determine the order in which tasks runCS267 L23 Load Balancing and Scheduling.5Demmel Sp 1999Understanding Different Load Balancing ProblemsLoad balancing problems differ in:°Tasks costs•Do all tasks have equal costs?•If not, when are the costs known?-Before starting, when task created, or only when task ends°Task dependencies•Can all tasks be run in any order (including parallel)?•If not, when are the dependencies known?-Before starting, when task created, or only when task ends°Locality•Is it important for some tasks to be scheduled on the same processor (or nearby) to reduce communication cost?•When is the information about communication between tasks known?CS267 L23 Load Balancing and Scheduling.6Demmel Sp 1999Task cost spectrumCS267 L23 Load Balancing and Scheduling.7Demmel Sp 1999Task Dependency SpectrumCS267 L23 Load Balancing and Scheduling.8Demmel Sp 1999Task Locality Spectrum (Data Dependencies)CS267 L23 Load Balancing and Scheduling.9Demmel Sp 1999Spectrum of SolutionsOne of the key questions is when certain information about the load balancing problem is knownLeads to a spectrum of solutions:°Static scheduling. All information is available to scheduling algorithm, which runs before any real computation starts. (offline algorithms)°Semi-static scheduling. Information may be known at program startup, or the beginning of each timestep, or at other well-defined points. Offline algorithms may be used even though the problem is dynamic.°Dynamic scheduling. Information is not known until mid-execution. (online algorithms)CS267 L23 Load Balancing and Scheduling.10Demmel Sp 1999Approaches°Static load balancing°Semi-static load balancing°Self-scheduling°Distributed task queues°Diffusion-based load balancing°DAG scheduling°Mixed ParallelismNote: these are not all-inclusive, but represent some of the problems for which good solutions exist.CS267 L23 Load Balancing and Scheduling.11Demmel Sp 1999Static Load Balancing°Static load balancing is use when all information is available in advance°Common cases:•dense matrix algorithms, such as LU factorization -done using blocked/cyclic layout-blocked for locality, cyclic for load balance•most computations on a regular mesh, e.g., FFT-done using cyclic+transpose+blocked layout for 1D-similar for higher dimensions, i.e., with transpose•sparse-matrix-vector multiplication-use graph partitioning-assumes graph does not change over time (or at least within a timestep during iterative solve)CS267 L23 Load Balancing and Scheduling.12Demmel Sp 1999Semi-Static Load Balance°If domain changes slowly over time and locality is important•use static algorithm•do some computation (usually one or more timesteps) allowing some load imbalance on later steps•recompute a new load balance using static algorithm°Often used in:•particle simulations, particle-in-cell (PIC) methods-poor locality may be more of a problem than load imbalance as particles move from one grid partition to another•tree-structured computations (Barnes Hut, etc.)•grid computations with dynamically changing grid, which changes slowlyCS267 L23 Load Balancing and Scheduling.13Demmel Sp 1999Self-Scheduling°Self scheduling:•Keep a centralized pool of tasks that are available to run•When a processor completes its current task, look at the pool•If the computation of one task generates more, add them to the pool°Originally used for:•Scheduling loops by compiler (really the runtime-system)•Original paper by Tang and Yew, ICPP 1986CS267 L23 Load Balancing and Scheduling.14Demmel Sp 1999When is Self-Scheduling a Good Idea?Useful when:°A batch (or set) of tasks without dependencies•can also be used with dependencies, but most analysis has only been done for task
View Full Document
Unlocking...