3-1©2008 Raj JainCSE567MWashington University in St. LouisSelection of Selection of Techniques and MetricsTechniques and MetricsRaj Jain Washington University in Saint LouisSaint Louis, MO [email protected] slides are available on-line at:http://www.cse.wustl.edu/~jain/cse567-08/3-2©2008 Raj JainCSE567MWashington University in St. LouisOverviewOverview! Criteria for Selecting an Evaluation Technique! Three Rules of Validation! Selecting Performance Metrics! Commonly Used Performance Metrics! Utility Classification of Metrics! Setting Performance Requirements3-3©2008 Raj JainCSE567MWashington University in St. LouisCriteria for Selecting an Evaluation TechniqueCriteria for Selecting an Evaluation Technique3-4©2008 Raj JainCSE567MWashington University in St. LouisThree Rules of ValidationThree Rules of Validation! Do not trust the results of an analytical model until they have been validated by a simulation model or measurements.! Do not trust the results of a simulation model until they have been validated by analytical modeling or measurements.! Do not trust the results of a measurement until they have been validated by simulation or analytical modeling.3-5©2008 Raj JainCSE567MWashington University in St. LouisSelecting Performance MetricsSelecting Performance Metrics3-6©2008 Raj JainCSE567MWashington University in St. LouisSelecting MetricsSelecting Metrics! Include:" Performance Time, Rate, Resource" Error rate, probability" Time to failure and duration! Consider including:" Mean and variance" Individual and Global! Selection Criteria:" Low-variability" Non-redundancy" Completeness3-7©2008 Raj JainCSE567MWashington University in St. LouisCase Study: Two Congestion Control AlgorithmsCase Study: Two Congestion Control Algorithms! Service: Send packets from specified source to specified destination in order.! Possible outcomes:" Some packets are delivered in order to the correct destination." Some packets are delivered out-of-order to the destination." Some packets are delivered more than once (duplicates)." Some packets are dropped on the way (lost packets).3-8©2008 Raj JainCSE567MWashington University in St. LouisCase Study (Cont)Case Study (Cont)! Performance: For packets delivered in order," Time-rate-resource ⇒! Response time to deliver the packets! Throughput: the number of packets per unit of time.! Processor time per packet on the source end system.! Processor time per packet on the destination end systems.! Processor time per packet on the intermediate systems." Variability of the response time ⇒ Retransmissions! Response time: the delay inside the network3-9©2008 Raj JainCSE567MWashington University in St. LouisCase Study (Cont)Case Study (Cont)" Out-of-order packets consume buffers⇒ Probability of out-of-order arrivals." Duplicate packets consume the network resources⇒ Probability of duplicate packets" Lost packets require retransmission⇒ Probability of lost packets" Too much loss cause disconnection⇒ Probability of disconnect3-10©2008 Raj JainCSE567MWashington University in St. LouisCase Study (Cont)Case Study (Cont)! Shared Resource ⇒ Fairness! Fairness Index Properties:" Always lies between 0 and 1." Equal throughput ⇒ Fairness =1." If k of n receive x and n-k users receive zero throughput: the fairness index is k/n.3-11©2008 Raj JainCSE567MWashington University in St. LouisCase Study (Cont)Case Study (Cont)" Throughput and delay were found redundant ⇒Use Power." Variance in response time redundant with the probability of duplication and the probability of disconnection" Total nine metrics.3-12©2008 Raj JainCSE567MWashington University in St. LouisCommonly Used Performance MetricsCommonly Used Performance Metrics! Response time and Reaction time3-13©2008 Raj JainCSE567MWashington University in St. LouisResponse Time (Cont)Response Time (Cont)3-14©2008 Raj JainCSE567MWashington University in St. LouisCapacityCapacity3-15©2008 Raj JainCSE567MWashington University in St. LouisCommon Performance Metrics (Cont)Common Performance Metrics (Cont)! Nominal Capacity: Maximum achievable throughput under ideal workload conditions. E.g., bandwidth in bits per second. The response time at maximum throughput is too high.! Usable capacity: Maximum throughput achievable without exceeding a pre-specified response-time limit! Knee Capacity: Knee = Low response time and High throughput3-16©2008 Raj JainCSE567MWashington University in St. LouisCommon Performance Metrics (cont)Common Performance Metrics (cont)! Turnaround time = the time between the submission of a batch job and the completion of its output.! Stretch Factor: The ratio of the response time with multiprogramming to that without multiprogramming.! Throughput: Rate (requests per unit of time) Examples:" Jobs per second" Requests per second" Millions of Instructions Per Second (MIPS)" Millions of Floating Point Operations Per Second (MFLOPS)" Packets Per Second (PPS)" Bits per second (bps)" Transactions Per Second (TPS)3-17©2008 Raj JainCSE567MWashington University in St. LouisCommon Performance Metrics (Cont)Common Performance Metrics (Cont)! Efficiency: Ratio usable capacity to nominal capacity. Or, the ratio of the performance of an n-processor system to that of a one-processor system is its efficiency.! Utilization: The fraction of time the resource is busy servicing requests. Average fraction used for memory.3-18©2008 Raj JainCSE567MWashington University in St. LouisCommon Performance Metrics (Cont)Common Performance Metrics (Cont)! Reliability:" Probability of errors" Mean time between errors (error-free seconds).! Availability:" Mean Time to Failure (MTTF)" Mean Time to Repair (MTTR)" MTTF/(MTTF+MTTR)3-19©2008 Raj JainCSE567MWashington University in St. LouisUtility Classification of MetricsUtility Classification of Metrics3-20©2008 Raj JainCSE567MWashington University in St. LouisSetting Performance RequirementsSetting Performance Requirements! Examples:“ The system should be both processing and memory efficient. It should not create excessive overhead”“ There should be an extremely low probability that the network will duplicate a packet, deliver a packet to the wrong destination, or change the data in a packet.”! Problems:Non-SpecificNon-MeasurableNon-AcceptableNon-RealizableNon-Thorough⇒ SMART3-21©2008 Raj JainCSE567MWashington University in St. LouisCase Study 3.2: Local Area NetworksCase Study 3.2: Local Area Networks! Service: Send frame to D!
View Full Document
Unlocking...