IntroductionOverview of Analysis of Software SystemsSlide 3Why Do We Care?Importance of Performance in ResearchState of Performance Evaluation in the FieldWhat’s the Result?Where Does This Problem Come From?But the Field Is ImprovingWhat Do You Need To Know To Analyze Performance?Selecting Appropriate Experiment CharacteristicsEvaluation TechniquesThe Role of Modeling and SimulationPerformance MetricsWorkloadsProper Performance Measurement TechniquesProper Statistical TechniquesMinimizing Your WorkProper Data Presentation TechniquesWhy Is Performance Analysis Difficult?Bad Example: Ratio GameSlide 22ExampleStarting on an AnswerSome Common Mistakes in Performance EvaluationNo GoalsBiased GoalsUnsystematic ApproachAnalysis Without Understanding the ProblemIncorrect Performance MetricsUnrepresentative WorkloadWrong Evaluation TechniqueOverlooking Important ParametersIgnoring Significant FactorsInappropriate Experiment DesignInappropriate Level of DetailNo AnalysisErroneous AnalysisNo Sensitivity AnalysisIgnoring Errors in InputImproper Treatment of OutliersAssuming Static SystemsIgnoring VariabilityOverly Complex AnalysisImproper Presentation of ResultsIgnoring Social AspectsOmitting Assumptions and LimitationsA Systematic Approach To Performance EvaluationSystematic Approach, Cont.State Goals and Define the SystemList Services and OutcomesSelect MetricsList ParametersSelect Factors To StudySelect Evaluation TechniqueSelect WorkloadDesign ExperimentsAnalyze and Interpret DataPresent ResultsIntroductionAndy WangCIS 5930-03Computer Systems Performance Analysis2Overview of Analysis of Software Systems•Introduction•Common mistakes made in system software analysis–And how to avoid them•Selection of techniques and metrics3Introduction•Why do we care?•Why is it hard?•What tools can we use to understand system performance?4Why Do We Care?•Performance is almost always a key issue in software–Especially in system software•Everyone wants best possible performance•Cost of achieving performance also key•Reporting performance is necessary in many publication venues–Both academic and industry5Importance of Performancein Research•Performance is key in almost all CS research•A solution that doesn’t perform well isn’t a solution at all•Successful research must prove performance characteristics to a skeptical community6State of Performance Evaluation in the Field•Generally regarded as poor•Many systems have little performance data presented•Many systems are measured by improper criteria•Many experiments are poorly designed•Many results are badly or incorrectly presented•Replication not generally respected7What’s the Result?•You can’t always trust what you read in a research paper•Authors may have accidentally or intentionally misled you–Overstating performance–Hiding problems–Not answering the important questions8Where Does This Problem Come From?•Mostly from ignorance of proper methods of measuring and presenting performance•Abetted by reader’s ignorance of what questions they should be asking9But the Field Is Improving•People are taking performance measurement more seriously•Quality of published experiments is increasing•Yours had better be of high quality, too–Publishing is tough–Be at the top of the heap of papers10What Do You Need To Know To Analyze Performance?•Selection of appropriate experiment characteristics•Proper performance measurement techniques•Proper statistical techniques•Proper data presentation techniques11Selecting Appropriate Experiment Characteristics•Evaluation techniques•Performance metrics•Workloads12Evaluation Techniques•What’s the best technique for getting an answer to a performance question?•Modeling?•Simulation?•Experimentation?13The Role of Modelingand Simulation•Actual measurement not always best tool•Modeling might be better•Simulation might be better•But that’s not what this class is about•We will discuss when to use those techniques, though14Performance Metrics•The criteria used to evaluate the performance of a system•E.g., response time, transactions per second, bandwidth delivered, etc.•Choosing the proper metrics is key to a real understanding of system performance15Workloads•The requests users make on a system•If you don’t evaluate with a proper workload, you aren’t measuring what real users will experience•Typical workloads:–Types of queries–Jobs submitted to an OS–Messages sent through a protocol16Proper Performance Measurement Techniques•You need at least two components to measure performance:1. A load generator•To apply a workload to the system2. A monitor•To find out what happened17Proper Statistical Techniques•Most computer performance measurements not purely deterministic•Most performance evaluations weigh effects of different alternatives•How to separate meaningless variations from vital data in measurements?•Requires proper statistical techniques18Minimizing Your Work•Unless you design carefully, you’ll measure a lot more than you need to•A careful design can save you from doing lots of measurements•Should identify critical factors•And determine smallest number of experiments that gives sufficiently accurate answer19Proper DataPresentation Techniques•You’ve got pertinent, statistically accurate data that describes your system•Now what?•How to present it–Honestly–Clearly–Convincingly20Why Is Performance Analysis Difficult?•Because it’s an art - it’s not mechanical•You can’t just apply a handful of principles and expect good results•You’ve got to understand your system•You’ve got to select your measurement techniques and tools properly•You’ve got to be careful and honestBad Example: Ratio Game•Throughput in transactions per secondSystem Workload 1 Workload 2 Average A 20 10 15B 10 20 15Bad Example: Ratio Game•Throughput normalized to system B•Throughput normalized to system ASystem Workload 1 Workload 2 Average A 2 0.5 1.25B 1 1 1System Workload 1 Workload 2 Average A 1 1 1B 0.5 2 1.2523Example •Suppose you’ve built an operating system for a special-purpose Internet browsing box•How well does it perform?•Indeed, how do you even begin to answer that question?24Starting on an Answer•What’s the OS supposed to do?•What demands will be put on it?•What hardware will it work with, and what are that hardware’s characteristics?•What