Experimental LifecycleWorkloadsWhat is a Workload?Workload IssuesTypes: Real (“live”) WorkloadsTypes: Synthetic WorkloadsInstruction WorkloadsInstruction Workloads (cont’d)Synthetic ProgramsReal-World BenchmarksApplication Benchmarks“Standard” Benchmarks“Standard” Benchmarks (cont’d)Popular “Standard” BenchmarksSieve and Ackermann’s FunctionWhetstoneLINPACKDhrystoneLivermore LoopsDebit/Credit BenchmarkSPEC SuiteModified Andrew BenchmarkExercisers and Drivers (Microbenchmarks)Advantages /DisadvantagesWorkload SelectionServices ExercisedCompletenessComponent TestingService TestingCharacterizing a ServiceExample: Web ServerWeb Client AnalysisNetwork AnalysisWeb Server AnalysisFile System AnalysisDisk Drive AnalysisLevel of DetailRepresentativenessTimelinessOther ConsiderationsFor Discussion Next Wednesday© 1998, Geoff Kuenning Evidence of real problem,justification, opportunity, feasibility, understandingBoundary of system under test, workload & system parameters that affect behavior.Questions thattest the model.metrics to answer questions, factors to vary, levels of factors.Vague idea“groping around” experiencesModelInitialobservationsExperimentData, analysis, interpretationResults & finalPresentationExperimental LifecycleHypothesisWorkloadsExperimentalenvironmentprototypereal sysexec-drivensimtrace-drivensimstochasticsimLiveworkloadBenchmarkapplicationsMicro-benchmarkprogramsSyntheticbenchmarkprogramsTracesDistributions& otherstatisticsmonitoranalysisgeneratorSynthetictraces“Real”workloadsMade-up© 2003, Carla EllisDatasets© 1998, Geoff Kuenning What is a Workload?•A workload is anything a computer is asked to do•Test workload: any workload used to analyze performance•Real workload: any observed during normal operations•Synthetic: created for controlled testingWorkload Issues•Selection of benchmarks–Criteria:•Repeatability•Availability and community acceptance•Representative of typical usage (e.g. timeliness)•Predictive of real performance – realistic (e.g. scaling issue)–Types•Tracing workloads & using traces–Monitor design–Compression, anonymizingrealism (no feedback)•Workload characterization•Workload generators© 2003, Carla EllisChoosing anunbiased workloadis key to designingan experiment thatcan disprove an hypothesis.© 1998, Geoff Kuenning Types: Real (“live”) Workloads•Advantage is they represent reality•Disadvantage is they’re uncontrolled–Can’t be repeated–Can’t be described simply–Difficult to analyze•Nevertheless, often useful for “final analysis” papers•“Deployment experience”© 1998, Geoff Kuenning Types: Synthetic Workloads•Advantages:–Controllable–Repeatable–Portable to other systems–Easily modified•Disadvantage: can never be sure real world will be the same (i.e., are they representative?)© 1998, Geoff Kuenning Instruction Workloads•Useful only for CPU performance–But teach useful lessons for other situations•Development over decades–“Typical” instruction (ADD)–Instruction mix (by frequency of use)•Sensitive to compiler, application, architecture•Still used today (MFLOPS)© 1998, Geoff Kuenning Instruction Workloads (cont’d)•Modern complexity makes mixes invalid–Pipelining–Data/instruction caching–Prefetching•Kernel is inner loop that does useful work:–Sieve, matrix inversion, sort, etc.–Ignores setup, I/O, so can be timed by analysis if desired (at least in theory)© 1998, Geoff Kuenning Synthetic Programs•Complete programs–Designed specifically for measurement–May do real or “fake” work–May be adjustable (parameterized)•Two major classes:–Benchmarks–Exercisers© 1998, Geoff Kuenning Real-World Benchmarks•Pick a representative application•Pick sample data•Run it on system to be tested•Modified Andrew Benchmark, MAB, is a real-world benchmark•Easy to do, accurate for that sample data•Fails to consider other applications, data© 1998, Geoff Kuenning Application Benchmarks•Variation on real-world benchmarks•Choose most important subset of functions•Write benchmark to test those functions•Tests what computer will be used for•Need to be sure important characteristics aren’t missed© 1998, Geoff Kuenning “Standard” Benchmarks•Often need to compare general-purpose computer systems for general-purpose use–E.g., should I buy a Compaq or a Dell PC?–Tougher: Mac or PC?•Desire for an easy, comprehensive answer•People writing articles often need to compare tens of machines© 1998, Geoff Kuenning “Standard” Benchmarks (cont’d)•Often need to make comparisons over time–Is this year’s PowerPC faster than last year’s Pentium?•Obviously yes, but by how much?•Don’t want to spend time writing own code–Could be buggy or not representative–Need to compare against other people’s results•“Standard” benchmarks offer a solution© 1998, Geoff Kuenning Popular “Standard” Benchmarks•Sieve•Ackermann’s function•Whetstone•Linpack•Dhrystone•Livermore loops•Debit/credit•SPEC•MAB© 1998, Geoff Kuenning Sieve andAckermann’s Function•Prime number sieve (Erastothenes)–Nested for loops–Usually such small array that it’s silly•Ackermann’s function–Tests procedure calling, recursion–Not very popular in Unix/PC community© 1998, Geoff Kuenning Whetstone•Dates way back (can compare against 70’s)•Based on real observed frequencies•Entirely synthetic (no useful result)•Mixed data types, but best for floating•Be careful of incomparable variants!© 1998, Geoff Kuenning LINPACK•Based on real programs and data•Developed by supercomputer users•Great if you’re doing serious numerical computation© 1998, Geoff Kuenning Dhrystone•Bad pun on “Whetstone”•Motivated by Whetstone’s perceived excessive emphasis on floating point•Dates back to when p’s were integer-only•Very popular in PC world•Again, watch out for version mismatches© 1998, Geoff Kuenning Livermore Loops•Outgrowth of vector-computer development•Vectorizable loops•Based on real programs•Good for supercomputers•Difficult to characterize results simply© 1998, Geoff Kuenning Debit/Credit Benchmark•Developed for transaction processing environments–CPU processing is usually trivial–Remarkably demanding I/O, scheduling requirements•Models real TPS workloads synthetically•Modern version is TPC benchmark© 1998, Geoff Kuenning
View Full Document
Unlocking...