Measurement ToolsMonitorsClassifications of MonitorsHardware vs. Software MonitorsEvent-Driven vs. Sampling MonitorsOn-Line vs. Batch MonitorsIssues in Monitor DesignActivation MechanismBuffer IssuesData Compression or AnalysisEnabling/Disabling MonitorsPriority of MonitorMonitoring Abnormal EventsTools and Methods For Software MeasurementTools For Software MeasurementCode InstrumentationAdvantages of Code InstrumentationDisadvantages of Instrumenting CodeTypical Types of InstrumentationCountersCounter ExamplesLogsDesigning a Log EntrySlide 29TimersTracing PackagesAdvantages of Tracing PackagesDisadvantages of Tracing PackagesHow Do Tracing Packages Work?System-Provided Metrics and UtilitiesWhat a Typical System ProvidesTiming ToolsProcess-State ToolsUsing Process-State ToolsSystem-State ToolsOS Accounting LogsSystem Software Accounting LogsWhite SlideMeasurement ToolsAndy WangCIS 5930-03Computer SystemsPerformance Analysis2Monitors•A monitor is a tool used to observe system activity•Proper use of monitors is key to performance analysis•Also useful for other system observation purposes3Classifications of Monitors•Hardware vs. software monitors•Event-driven vs. sampling monitors•On-line vs. batch monitors4Hardware vs. Software Monitors•Hardware monitors used primarily by hardware designers–Requires substantial knowledge of hardware details–VLSI limits monitoring possibilities•Software monitors used (mostly) by everyone else5Event-Driven vs. Sampling Monitors•Event-driven monitors notice every time a particular type of event occurs–Ideal for rare events–Require low per-invocation overheads•Sampling monitors check the state of the system periodically–Good for frequent events–Can afford higher overheads6On-Line vs. Batch Monitors•On-line monitors can display their information continuously–Or, at least, frequently•Batch monitors save it for later–Usually have separate analysis procedures7Issues in Monitor Design•Activation mechanism•Buffer issues•Data compression/analysis•Enabling/disabling monitors•Priority issues•Abnormal events monitoring8Activation Mechanism•When do you collect the data?–When an interesting event occurs, trap to data collection routine–Analyze every step taken by system–Go to data collection routine when timer expires9Buffer Issues•Buffer size –Big enough to avoid frequent disk writes–Small enough to make disk writes cheap•Number of buffers–At least two, typically–One to fill up, one to record•Buffer overflow–Overwrite old data you haven’t recorded–Or lose new data you don’t have room for–In either case, count what’s lost•Sometimes can wait for buffer to empty10Data Compressionor Analysis•Data can be literally compressed•Or can be reduced to a summary form•Both methods save space for holding data•But at cost of extra overhead in gathering it•Sometimes can use idle time to compress–But maybe better spent dumping data to disk11Enabling/Disabling Monitors•Most system monitors have some overhead•Need to turn them off if high performance required–Unless overhead is trivial–Or if primary system purpose is gathering data•As with many research systems12Priority of Monitor•How high a priority for monitor’s operations?•Trade off performance impact against timely & complete data gathering•Not always simple question13Monitoring Abnormal Events•Often, knowing about failures and errors more important than knowing about normal operation•Sometimes requires special attention–System may not be operating very well at time of failure!19Tools and Methods For Software Measurement•OK, so how do I actually measure a piece of software?•What practical tools and methods are available to me?•How do I get my project done?20Tools ForSoftware Measurement•Code instrumentation•Tracing packages•System-provided metrics and utilities•Profiling21Code Instrumentation•Adding monitoring code to system under study•Basically, just add code that does what you want22Advantages ofCode Instrumentation+Usually most direct way to gather data+Complete flexibility in where to insert monitoring code+Strong control over costs of monitoring+Resulting measurements always available23Disadvantages of Instrumenting Code–Requires access to source–Requires strong knowledge of design and details of code–Watch out for asynchronous calls–Requires recompilation to change monitoring facility–If overdone, strong potential to affect performance24Typical Types of Instrumentation•Counters–Cheap and fast–Low level of detail•Logs–More detail–More costly–Require occasional dumping or digesting•Timers–To determine elapsed time for operations–Typically using OS-provided system calls25Counters•Useful only if number of times an event occurs is of interest•Can be used to accumulate totals•In modern systems, make them wide enough to not overflow (long long is good)26Counter Examples•Number of times a network protocol transmits packets•Number of times programs are swapped out due to exceeding time slices•Number of incoming requests to Web server27Logs•Can log arbitrarily complex data about an event•But more complex data takes more space•Typically, log data into reserved buffer•When full, ask that buffer be written to disk–Often want second buffer to gather data while awaiting disk write28Designing a Log Entry•What form should a log entry take?•Designing for compactness vs. human readability–Former better for most purposes•Easy to post-format for printing–Latter useful for system debugging–Make sure no important information is lost in compacting log entry29Designing a Log Entry•Should include –A version stamp–Delimiters and checksum30Timers•Many OSs provide system calls that start and stop timers–Allows measuring how long things took•Usually, only elapsed time measurable–Not necessarily time spent running particular process•Care required to capture real meaning of timings31Tracing Packages•Allow dynamic monitoring of code that doesn’t have built-in monitors•Basically, augment code to call monitoring routines when desired•Akin to debuggers•Typically allow counters and some forms of logging32Advantagesof Tracing Packages+Allows pretty arbitrary insertion of monitoring code+Don’t need recompilation to instrument code+Tremendous flexibility at measurement time+No instrumentation overhead when you’re not using