Slide 1MotivationBackgroundBackground ContinuedOptimistic Replication ExampleExample ContinuedGoalServicesOutcomesMetricsMetrics not MeasuredMonitor ImplementationParametersParameters (Continued)WorkloadsExperimental SettingsExperimental SettingsExperimental Design255 Full Factorial Design255 Full Factorial AnalysisVariation of EffectsResiduals vs. Predicted TimeResiduals vs. Experiment NumbersQuantile-Quantile PlotMultivariate RegressionMultivariate RegressionResiduals vs. Predicted TimeResiduals vs. Experiment NumbersQuantile-Quantile PlotLog Transform (User Time)Residual Analyses (User Time)Possible ExplanationsLinear RegressionLinear RegressionResiduals vs. Predicted TimeResiduals vs. Experiment NumbersQuantile-Quantile PlotPossible ExplanationsConclusionWhite SlideExample: Rumor Performance EvaluationAndy WangCIS 5930-03Computer SystemsPerformance AnalysisMotivation•Optimistic peer replication is popular–Intermittent connectivity–Availability of replicas for concurrent updates–Convergence and correctness for updates•Example: Rumor, Coda, Ficus, Lotus Notes, Outlook Calendar, CVS2Background•Replication provides high availability•Optimistic replication allows immediate access to any replicated item, at the risk of permitting concurrent updates•Reconciliation process makes replicas consistent (i.e., two replicas for peer-to-peer)3Background Continued•Conflicts occur when different replicas of the same file are updated subsequent to the previous reconciliation4Optimistic Replication Example5Log on Desktop10:00 Update10:25 UpdateLog on Portable10:00 Update10:25 UpdateconnectedLog on Desktop10:00 Update10:25 Update10:40 UpdateLog on Portable10:00 Update10:25 Update10:51 UpdatedisconnectedExample Continued6Log on Desktop10:00 Update10:25 Update10:40 UpdateLog on Portable10:00 Update10:25 Update10:51 UpdatedisconnectedLog on Desktop10:00 Update10:25 Update10:40 Update10:51 UpdateLog on Portable10:00 Update10:25 Update10:40 Update10:51 Update connected•Run reconciliation•Detect a conflict•Propagate updatesGoal•Understand the cost characteristics of the reconciliation process for Rumor7Services•Reconciliation–Exchange file system states–Detect new and conflicting versions•If possible, automatically resolve conflicts•Else, prompt user to resolve conflicts–Propagate updates8Outcomes•Two reconciled replicas become consistent for all files and directories•Some files remain inconsistent and require user to resolve conflicts9Metrics•Time–Elapsed time •From the beginning to the completion of a reconciliation request–User time (time spent using CPU)–System time (time spent in the kernel)•Failure rate–Number of incomplete reconciliations and infinite loops (none observed)10Metrics not Measured•Disk access time–Require complex instrumentations •E.g., buffering, logging, etc.•Network and memory resources–Not heavily used•Correctness–Difficult to evaluate11Monitor Implementation12Spool-to-dump Spool-to-dumpReconScanner Rfindstored Rrecon ServerPerl libraryC++Reconciliation Process•Top-level Perl time commandParameters•System parameters–CPU (speed of local and remote servers)–Disk (bandwidth, fragmentation level)–Network (type, bandwidth, reliability)–Memory (size, caching effects, speed)–Operating system (type, version, VM management, etc.)13Parameters (Continued)•Workload parameters–Number of replicas–Number of files and directories–Number of conflicts and updates–Size of volumes (file size)14Workloads•Update characteristics extracted from Geoff Kuenning’s traces15File accessRead-only accessRead-write accessNonshared access Shared accessRead accessWrite access2-way sharing 3+way sharingRead accessWrite accessRead accessWrite accessExperimental Settings•Machine model: Dell Latitude XP•CPU: x486 100 MHz•RAM: 36MB•Ethernet: 10Mb•Operating system: Linux 2.0.x•File system: ext316Experimental Settings•Should have documented the following as well–CPU: L1 and L2 cache sizes–RAM: Brand and type–Disk: brand, model, capacity, RPM, and the size of on-disk cache–File system version17Experimental Design•255 full factorial design •Linear regression or multivariate linear regression to model major factors•Target: 95% confidence interval18255 Full Factorial Design•Number of replicas: 2 and 6•Number of files: 10 and 1,000•File size: 100 and 22,000 bytes•Number of directories: 10 and 100•Number of updates: 10 and 450–Capped at 10 updates for 10 files•Number of conflicts: 0 /* typical */19255 Full Factorial Analysis•Experiment errors < 3%200 5 101520253035020406080100120140160elapsed timemeasured timerun 2run 3run 4run 5predicted timeexperiment numbertime (seconds)0 5 10 15 20 25 30 350510152025303540user timemeasured timerun 2run 3run 4run 5predicted timeexperiment numbertime (seconds)0 5 10 15 20 25 30 350123456system timemeasured timerun 2run 3 run 4run 5predicted timeexperiment numbertime (seconds)Variation of Effects•All major effects significant at 95% confidence interval21#fles#dirsfle size * #flesfle size#updates0102030405060708090100top 5 effects for elapsed time% variation#fles#updates#fles * #updatesfle sizefle size * #fles0102030405060708090100top 5 effects for system time% variation#fles#replicas#dirs#replicas * #fles#fles * #updates0102030405060708090100top 5 effects for user time% variantionResiduals vs. Predicted Time•Clusters caused by dominating effects of files220 20 40 60 80 100 120 140-20-15-10-505101520elapsed timepredicted timeresiduals0 5 10 15 20 25 30 35 40-0.6-0.4-0.200.20.40.6user timepredicted timeresiduals0.5 1 1.5 2 2.5 3 3.5 4 4.5 5-0.5-0.4-0.3-0.2-0.100.10.20.30.40.5system timepredicted timeresidualsResiduals vs. Experiment Numbers•Residuals show homoscedasticity, almost230 20 40 60 80 100 120 140 160 180-0.6-0.4-0.200.20.40.6user timeexperiment numberresiduals0 20 40 60 80 100 120 140 160 180-0.5-0.4-0.3-0.2-0.100.10.20.30.40.5system timeexperiment numberresiduals0 20 40 60 80 100 120 140 160 180-20-15-10-505101520elapsed timeexperiment numberresidualsQuantile-Quantile Plot•Residuals are normally distributed, almost24-3 -2 -1 0 1 2 3 4-20-15-10-505101520f(x) = 5.61x − 0R² = 0.98elapsed timenormal quantilesresidual quantiles-3 -2 -1 0 1 2 3 4-0.6-0.4-0.200.20.40.6f(x) = 0.12x − 0R² = 0.95user timenormal quantilesresidual quantiles-3 -2 -1 0 1 2 3 4-0.5-0.4-0.3-0.2-0.100.10.20.30.40.5f(x) = 0.11x −