MIT OpenCourseWare http://ocw.mit.edu 6.189 Multicore Programming Primer, January (IAP) 2007 Please use the following citation format: Saman Amarasinghe, 6.189 Multicore Programming Primer, January (IAP) 2007. (Massachusetts Institute of Technology: MIT OpenCourseWare). http://ocw.mit.edu (accessed MM DD, YYYY). License: Creative Commons Attribution-Noncommercial-Share Alike. Note: Please use the actual date you accessed this material in your citation. For more information about citing these materials or our Terms of Use, visit: http://ocw.mit.edu/terms6.189 IAP 2007 Lecture 11 Parallelizing Compilers Prof. Saman Amarasinghe, MIT. 1 6.189 IAP 2007 MITOutline ● Parallel Execution ● Parallelizing Compilers ● Dependence Analysis ● Increasing Parallelization Opportunities ● Generation of Parallel Loops ● Communication Code Generation Prof. Saman Amarasinghe, MIT. 2 6.189 IAP 2007 MITTypes of Parallelism ● Instruction Level Parallelism (ILP)  Scheduling and Hardware ● Task Level Parallelism (TLP)  Mainly by hand ● Loop Level Parallelism (LLP)  Hand or Compiler Generatedor Data Parallelism ● Pipeline Parallelism  Hardware or Streaming ● Divide and Conquer  Recursive functions Parallelism Prof. Saman Amarasinghe, MIT. 3 6.189 IAP 2007 MITWhy Loops? ● 90% of the execution time in 10% of the code  Mostly in loops ● If parallel, can get good performance  Load balancing ● Relatively easy to analyze Prof. Saman Amarasinghe, MIT. 4 6.189 IAP 2007 MITProgrammer Defined Parallel Loop ● FORALL ● FORACROSS  No “loop carried  Some “loop carried dependences” dependences”  Fully parallel 5 6.189 IAP 2007 MIT Prof. Saman Amarasinghe, MIT.Parallel Execution ● ExampleFORPAR I = 0 to N A[I] = A[I] + 1 ● Block Distribution: Program gets mapped intoIters = ceiling(N/NUMPROC); FOR P = 0 to NUMPROC-1 FOR I = P*Iters to MIN((P+1)*Iters, N) A[I] = A[I] + 1 ● SPMD (Single Program, Multiple Data) CodeIf(myPid == 0) { … Iters = ceiling(N/NUMPROC); }Barrier();FOR I = myPid*Iters to MIN((myPid+1)*Iters, N)A[I] = A[I] + 1 Barrier(); Prof. Saman Amarasinghe, MIT. 6 6.189 IAP 2007 MITParallel Execution ● ExampleFORPAR I = 0 to N A[I] = A[I] + 1 ● Block Distribution: Program gets mapped intoIters = ceiling(N/NUMPROC);FOR P = 0 to NUMPROC-1 FOR I = P*Iters to MIN((P+1)*Iters, N)A[I] = A[I] + 1 ● Code that fork a function Iters = ceiling(N/NUMPROC);ParallelExecute(func1); … void func1(integer myPid){ FOR I = myPid*Iters to MIN((myPid+1)*Iters, N)A[I] = A[I] + 1} Prof. Saman Amarasinghe, MIT. 7 6.189 IAP 2007 MITOutline ● Parallel Execution ● Parallelizing Compilers ● Dependence Analysis ● Increasing Parallelization Opportunities ● Generation of Parallel Loops ● Communication Code Generation Prof. Saman Amarasinghe, MIT. 8 6.189 IAP 2007 MITParallelizing Compilers ● Finding FORALL Loops out of FOR loops ● Examples FOR I = 0 to 5 A[I+1] = A[I] + 1 FOR I = 0 to 5 A[I] = A[I+6] + 1 For I = 0 to 5 A[2*I] = A[2*I + 1] + 1 Prof. Saman Amarasinghe, MIT. 9 6.189 IAP 2007 MITIteration Space ● N deep loops  n-dimensional discrete cartesian space  Normalized loops: assume step size = 1 0 1 2 5 4 3 6 7  J FOR I = 0 to 6 FOR J = I to 7 I ● Iterations are represented ascoordinates in iteration space  i = [i1, i2, i3,…, in] 0123456 Prof. Saman Amarasinghe, MIT. 10 6.189 IAP 2007 MITIteration Space ● N deep loops  n-dimensional discrete cartesian space  Normalized loops: assume step size = 1 0 1 2 5 4 3 6 7  J FOR I = 0 to 6 FOR J = I to 7 I ● Iterations are represented ascoordinates in iteration space ● Sequential execution order of iterations  Lexicographic order [0,0], [0,1], [0,2], …, [0,6], [0,7], [1,1], [1,2], …, [1,6], [1,7], [2,2], …, [2,6], [2,7], ……… [6,6], [6,7], 0123456 Prof. Saman Amarasinghe, MIT. 11 6.189 IAP 2007 MITIteration Space ● N deep loops  n-dimensional discrete cartesian space  Normalized loops: assume step size = 1 0 1 2 5 4 3 6 7  J FOR I = 0 to 6 FOR J = I to 7 I ● Iterations are represented ascoordinates in iteration space ● Sequential execution order of iterations  Lexicographic order 0123456 ● Iteration i is lexicograpically less than j , i < j iff there exists c s.t. i1 = j1, i2 = j2,… ic-1 = jc-1 and ic < jc Prof. Saman Amarasinghe, MIT. 12 6.189 IAP 2007 MITIteration Space ● N deep loops  n-dimensional discrete cartesian space  Normalized loops: assume step size = 1 0 1 2 5 4 3 6 7  J FOR I = 0 to 6 FOR J = I to 7 I ● An affine loop nest 0123456  Loop bounds are integer linear functions of constants, loop constant variables and outer loop indexes  Array accesses are integer linear functions of constants, loop constant variables and loop indexes Prof. Saman Amarasinghe, MIT. 13 6.189 IAP 2007 MITIteration Space ● N deep loops  n-dimensional discrete cartesian space  Normalized loops: assume step size = 1 FOR I = 0 to 6 FOR J = I to 7 ● Affine loop nest  Iteration space as a set of liner inequalities 0 ≤ I I ≤ 6 I ≤ J J ≤ 7 0 1 2 3 4 5 6 7  J 0 1 2 3 4 5 6 I Prof. Saman Amarasinghe, MIT. 14 6.189 IAP 2007 MITData Space ● M dimensional arrays  m-dimensional discrete cartesian space  a hypercube Integer A(10) 012 345 6789 Float B(5, 6) 012 345 0 1 2 3 4 Prof. Saman Amarasinghe, MIT. 15 6.189 IAP 2007 MITDependences ● True dependence a = = a ● Anti dependence = a a = ● Output dependence a = a = ● Definition: Data dependence exists for a dynamic instance i and j iff  either i or j is a write operation  i and j refer to the same variable  i executes before j ● How about array accesses within loops? Prof. Saman Amarasinghe, MIT. 16 6.189 IAP 2007 MITOutline ● Parallel Execution ● Parallelizing Compilers ● Dependence Analysis ● Increasing Parallelization Opportunities ● Generation of Parallel Loops ● Communication Code Generation Prof. Saman Amarasinghe, MIT. 17 6.189 IAP 2007 MITArray Accesses in a loop FOR I = 0 to 5 A[I] = A[I] + 1 01 2 3 4 5 Iteration Space 0 1 2 3 4 5 6 7 8 Data Space 9 10 11 12 Prof. Saman Amarasinghe, MIT. 18 6.189 IAP 2007 MIT19Array Accesses in a loop FOR I = 0 to 5 A[I] = A[I] + 1 Iteration Space Data Space 6.189 IAP 2007 MIT Prof. Saman Amarasinghe, MIT. 0 1 2 3 4 5 6