Understanding Performance in Operating Systems Andy Wang COP 5611 Advanced Operating Systems Outline Importance of operating systems performance Major issues in understanding operating systems performance Issues in experiment design Importance of OS Performance Performance is almost always a key issue in operating systems File system research OS tools for multimedia Practically any OS area Since everyone uses the OS sometimes heavily everyone is impacted by its performance A solution that doesn t perform well isn t a solution at all Importance of Understanding OS Performance Great so we work on improving OS performance How do we tell if we succeeded Successful research must prove its performance characteristics to a skeptical community So What Proper performance evaluation is difficult Knowing what to study is tricky Performance evaluations take a lot of careful work Understanding the results is hard Presenting them effectively is challenging For Example An idea save power from a portable computer s battery by using its wireless card to execute tasks remotely Maybe that s a good idea maybe it isn t How do we tell Performance experiments to validate concept But What Experiments What tasks should we check What should be the conditions of the portable computer What should be the conditions of the network What should be the conditions of the server How do I tell if my result is statistically valid Issues in Understanding OS Performance Techniques for understanding OS performance Elements of performance evaluation Common mistakes in performance evaluation Choosing proper performance metrics Workload design selection Monitors Software measurement tools Techniques for Understanding OS Performance Analytic modeling Simulation Measurement Which technique is right for a given situation Analytic Modeling Sometimes relatively quick Within limitations of model testing alternatives usually easy Mathematical tractability may require simplifications Not everything models well Question of validity of model Simulation Great flexibility Can capture an arbitrary level of detail Often a tremendous amount of work to write and run Testing a new alternative often requires repeating a lot of work Question of validity of simulation Experimentation Lesser problems of validity Sometimes easy to get started Can be very labor intensive Often hard to perform measurement Sometimes hard to separate out effects you want to study Sometimes impossible to generate cases you need to study Elements of Performance Evaluation Performance metrics Workloads Proper measurement technique Proper statistical techniques Minimization of effort Proper data presentation techniques Performance Metrics The criteria used to evaluate the performance of a system E g response time cache hit ratio bandwidth delivered etc Choosing the proper metrics is key to a real understanding of system performance Workloads The requests users make on a system If you don t evaluate with a proper workload you aren t measuring what real users will experience Typical workloads Stream of file system requests Set of jobs performed by users List of URLs submitted to a Web server Proper Performance Measurement Techniques You need at least two components to measure performance 1 A load generator To apply a workload to the system 2 A monitor To find out what happened Proper Statistical Techniques Computer performance measurements generally not purely deterministic Most performance evaluations weigh the effects of different alternatives How to separate meaningless variations from vital data in measurements Requires proper statistical techniques Minimizing Your Work Unless you design carefully you ll measure a lot more than you need to A careful design can save you from doing lots of measurements Should identify critical factors And determine the smallest number of experiments that gives a sufficiently accurate answer Proper Data Presentation Techniques You ve got pertinent statistically accurate data that describes your system Now what How to present it Honestly Clearly Convincingly Why Is Performance Analysis Difficult Because it s an art it s not mechanical You can t just apply a handful of principles and expect good results You ve got to understand your system You ve got to select your measurement techniques and tools properly You ve got to be careful and honest Some Common Mistakes in Performance Evaluation No goals Biased goals Unsystematic approach Analysis without understanding Incorrect performance metrics Unrepresentative workload Wrong evaluation technique More Common Performance Evaluation Mistakes Overlooking important parameters Ignoring significant factors Inappropriate experiment design No analysis Erroneous analysis No sensitivity analysis Yet More Common Mistakes Ignoring input errors Improper treatment of outliers Assuming static systems Ignoring variability Too complex analysis Improper presentation of results Ignoring social aspects Omitting assumptions limitations Choosing Proper Performance Metrics Three types of common metrics Time responsiveness Processing rate productivity Resource consumption utilization Can also measure various error parameters Response Time How quickly does system produce results Critical for applications such as Time sharing interactive systems Real time systems Parallel computing Processing Rate How much work is done per unit time Important for Determining feasibility of hardware Comparing different configurations Multimedia Resource Consumption How much does the work cost Used in Capacity planning Identifying bottlenecks Also helps to identify the next bottleneck Typical Error Metrics Successful service speed Incorrect service reliability No service availability Characterizing Metrics Usually necessary to summarize Sometimes means are enough Variability is usually critical Essentials of Statistical Evaluation Choose an appropriate summary Mean median and or mode Report measures of variation Standard deviation range etc Provide confidence intervals 95 Use confidence intervals to compare means Choosing What to Measure Pick metrics based on Completeness Non redundancy Variability Designing Workloads What is a workload Synthetic workloads Real World benchmarks Application benchmarks Standard benchmarks Exercisers and drivers What is a Workload A workload is anything a computer is asked to do Test workload any workload used to analyze performance Real workload any workload observed during normal operations Synthetic
View Full Document
Unlocking...