Simulate with feature turned off and turned on to examine impact Perhaps examine sensitivity to some parameters that were fixed Contention difficult to model correctly Processors becoming increasingly complex themselves Building accurate simulators is complex Can assume technological and architectural parameters Various sizes granularities and organizations performance characteristics Now machine parameters also variable Case II Want to examine benefit of idea in a more general context Case I Want to examine in the context of a given Building prototypes for evaluation is too expensive Build a simulator Typically many things change from one generation to next Evaluating an Idea or Tradeoff Should cover realistic regimes of operation Working sets and cache replication size Application parameters and number of processors 50 48 In addition to assignment orchestration many important properties of a parallel program depend on Some Important Observations realistic operating points Size of Cache or Replication Store unrealistic operating point Simulated processes are interleaved on the processor And schedules simulated processes based on simulated time 49 Execution driven simulation in both directions more accurate Simulator keeps track of simulated time and detailed statistics Trace driven simulation from generator to simulator 51 Simulates operations references commands issued by reference generator Simulator of extended memory hierarchy Reference generator plays role of simulated processors Coupling or information flow between the two parts varies Two parts to a simulator Simulation runs on a uniprocessor can be parallelized too Multiprocessor Simulation Some operating points are realistic some aren t operating point f cache replication size application parameters p Some working sets scale with application parameters and p some don t not fitting it may dramatically increase local miss rate or even communication A working set may consist of local and or nonlocal data Miss rate or Comm vol Many applications have a hierarchy of working sets Operating Points Based on Working Sets Memory and inter connect simulator p N e t w o r k application parameters as before machine parameters depending on generality of evaluation context number of processors cache replication size associativity granularities of allocation transfer coherence communication parameters latency bandwidth occupancies cost of simulation makes it all the more critical to prune the space Others can be scaled according to earlier scaling arguments First look for such parameters Otherwise can obtain highly unrepresentative behavior Scaling them down requires scaling down no of processors too But many application parameters affect key characteristics Should understand limitations and guidelines to avoid pitfalls need a few to allow settling down but don t need more may need to omit cold start when recording time and statistics Common example is no of time steps in many scientific applications Some parameters don t affect parallel performance much but do affect runtime and can be scaled down But very important since reality of most evaluation Then lower level machine parameters Focus on cache coherent SAS for concreteness Scaling Down Problem Parameters First examine scaling down problem size and no of processors cannot simulate the problem machine sizes we care about have to use scaled down problem and machine sizes how to scale down and stay representative Huge design space Cost of simulation in time and memory No good formulas exist 54 52 Two major problems beyond accuracy and reliability Difficulties in Simulation based Evaluation Want scaled down machine running scaled down problem to be representative of full sized scenario Scaling Down Parameters for Simulation Reference generator Pp Memp Mem3 3 P3 Mem 2 2 P2 Mem 1 1 P1 Memory hierarchy simulator returns simulated time information to reference generator which is used to schedule simulated processes Execution driven Simulation 55 53 Key behavioral characteristics Scaling relationships among application parameters Contention and communication parameters Avoid unrealistic scenarios Gain insights and estimates of performance Analyze effects where possible Look for knees and flat regions to prune where possible Understand growth rate of characteristic with parameter Perform sensitivity analysis where necessary context of evaluation may be restricted Identify values of interest for them Determine which parameters are relevant to evaluation Steps in an evaluation study Dealing with the Parameter Space Cover range of realistic operating points Can t really hope for full representativeness but can Distribution of time in different phases Want to preserve many aspects of full scale scenario 58 56 Many goals difficult individually and often impossible so to reconcile Difficulties in Scaling N p Representatively E g may not represent working set not fitting if cache not scaled Require detailed understanding of application system interactions An Example Evaluation Use guidelines to cover key operating points and extrapolate with caution Let s use equation solver as example goals of study restrictions imposed by technology or assumptions understanding of parameter interactions Prune the parameter space based on Obtain good coverage of realistic characteristics Avoid unrealistic execution characteristics Choosing parameters is more difficult 3 goals Goal of study To determine the value of adding a block transfer facility to a cache coherent SAS machine with distributed memory Workloads Choose at least some that have communication that is amenable to block transfer e g grid solver Should try to use as realistic sizes as possible Solutions and confidence levels are application specific should try to keep unchanged since hard to predict effects but greater impact with scaled down application and system parameters difficult to find good solutions for both communication and local access Associativity and Granularities more difficult Cache replication size guide by scaling of working sets not data set More difficult to do with confidence Often necessary when scaling down problem size Scaling Down Other Machine Parameters 59 57 For example low c to c ratio will not allow block transfer to help much Suppose one size chosen is 514 by 514 grid with 16 processors Choosing Parameters contd Knees can be determined by analysis or by very simple simulation Sharp knees in working set curve can help