Test LoadsApplying Test Loads to SystemsTest Load DesignTypes of Test LoadsLoads Caused by Real UsersTracesIssues in Using TracesRunning TracesLoad-Generation ProgramsBuilding the ModelUsing the ModelApplying Test LoadsExample of Applying Test LoadsFactors in Designing This ExperimentExperiment SetupNetwork Traffic IssuesControlling Other LoadCaching EffectsAutomating ExperimentsCommon Mistakes in BenchmarkingOnly Testing Average BehaviorIgnoring Skewness of Device DemandsLoading Levels Controlled InappropriatelyCaching Effects IgnoredInappropriate Buffer SizesSampling Inaccuracies IgnoredIgnoring Monitoring OverheadNot Validating MeasurementsNot Ensuring Constant Initial ConditionsNot Measuring Transient PerformancePerformance Comparison Using Device UtilizationsLots of Data, Little AnalysisWhite SlideTest LoadsAndy WangCIS 5930-03Computer SystemsPerformance Analysis2Applying Test Loadsto Systems•Designing test loads•Tools for applying test loads3Test Load Design•Most experiments require applying test loads to system•General characteristics of test loads already discussed•How do we design test loads?4Types of Test Loads•Real users•Traces•Load-generation programs5Loads Caused byReal Users•Put real people in front of your system•Two choices:–Have them run pre-arranged set of tasks–Have them do what they’d normally do•Always a difficult approach–Labor-intensive–Impossible to reproduce given load–Load is subject to many external influences•But highly realistic6Traces•Collect set of commands/accesses issued to system under test (or similar system)•Replay against your system•Some traces of common activities available from others (e.g., file accesses)–But often don’t contain everything you need7Issues in Using Traces•May be hard to alter or extend•Accuracy of trace may depend on behavior of system–If it takes twice as long in your system as in traced system, maybe traced execution would have been different–Only truly representative of traced system and execution8Running Traces•Need process that reads trace, keeps track of progress, and issues commands from trace when appropriate•Process must be reasonably accurate in timing–But must also have little performance impact•If trace is large, can’t keep it all in main memory–But be careful of disk overheads9Load-Generation Programs•Create model for load you want to apply•Write program implementing that model•Program issues commands & requests synthesized from model–E.g., if model says open file, program builds appropriate open() command10Building the Model•Tradeoff between ease of creation and use of model vs. its accuracy•Base model on everything you can find out about the real system behavior–Which may include examining traces•Consider whether model can be memoryless, or requires keeping track of what’s already happened (Markov)11Using the Model•May require creation of test files or processes or network connections–Model should include how they should be created•Shoot for minimum performance impact on system under test of program that implements model12Applying Test Loads•Most experiments will need multiple repetitions–Details covered later in course•Results most accurate if each repetition runs in identical conditionsTest software should work hard to duplicate conditions on each run13Example of ApplyingTest Loads•Using Ficus experiments discussed earlier, want performance impact of update propagation for multiple replicas•Test load is set of benchmarks involving file access & other activities•Must apply test load for varying numbers of replicas14Factors in DesigningThis Experiment•Setting up volumes and replicas•Network traffic•Other load on test machines•Caching effects•Automation of experiment–Very painful to start each run by hand15Experiment Setup•Need volumes to read and write, and replicas of each volume on various machines•Must be certain that setup completes before we start running experiment16Network Traffic Issues•If experiment is distributed (like ours), how is it affected by other traffic on network?•Is traffic seen on network used in test similar to traffic expected on network you would actually use?•If not, do you need to run on isolated network? And/or generate appropriate background network load?17Controlling Other Load•Generally, want to have as much control as possible over other processes running on test machines•Ideally, use dedicated machines•But also be careful about background and periodic jobs–In Unix context, check carefully on cron and network-related daemons18Caching Effects•Many types of jobs run much faster if things are in cache–Other things also change•Is caching effect part of what you’re measuring?–If not, do something to clean out caches between runs–Or arrange experiment so caching doesn’t help•But sometimes you should measure caching19Automating Experiments•For all but very small experiments, it pays to automate•Don’t want to start each run by hand•Automation must be done with care–Make sure previous run is really complete–Make sure you completely reset your state–Make sure the data is really collected!20Common Mistakesin Benchmarking•Many people have made these•You will make some of them, too•But watch for them, so you don’t make too many21Only TestingAverage Behavior•Test workload should usually include divergence from average workload–Few workloads always remain at their average–Behavior at extreme points is often very different•Particularly bad if only average behavior is used22Ignoring Skewnessof Device Demands•More generally, not including skewness of any component–E.g., distribution of file accesses among set of users•Leads to unrealistic conclusions about how system behaves23Loading LevelsControlled Inappropriately•Not all methods of controlling load are equivalent–More users (more cache misses)–Less think time (fewer disk flushes)–Increased resource demands per user•Choose methods that capture effect you are testing for•Prefer methods allowing more flexibility in control over those allowing less24Caching Effects Ignored•Caching occurs many places in modern systems•Performance on given request usually very different depending on cache hit or miss•Must understand how cache works•Must design experiment to use it realistically25Inappropriate Buffer Sizes•Slight changes in buffer sizes can