Experimentation in Computer Systems ResearchWhy?Experimental LifecycleCommon PracticeLots of Ways to Attack ExperimentationA Systematic ApproachSlide 7Slide 8Summary ofTopicsRange of Experimental Systems Research at DukeHow Course Will WorkDiscussion© 2003, Carla EllisExperimentation in Computer Systems ResearchWhy:“It doesn’t matter how beautiful your theory is, it doesn’t matter how smart you are – if it doesn’t agree with the experiment, it’s wrong.”R. Feynman© 2003, Carla EllisWhy?W. Tichy in “Should Computer Scientists Experiment More?” argues for experimentation.•Model / theory testingAs Feynman’s quotation suggests, an experiment can identify flaws in a theory (e.g. underlying assumptions that are violated by reality)•ExplorationGo where no system has gone before (i.e., open whole new areas for investigation)Demonstrate / identify the importance of a potential new research problem© 2003, Carla EllisEvidence of real problem,justification, opportunity, feasibility, understandingBoundary of system under test, workload & system parameters that affect behavior.Questions thattest the model.metrics to answer questions, factors to vary, levels of factors.Vague idea“groping around” experiencesHypothesisModelInitialobservationsExperimentData, analysis, interpretationResults & finalPresentationExperimental Lifecycle© 2003, Carla EllisVague ideaModelExperimentData, analysis, interpretationResults & finalPresentationCommon PracticeNo iterationNo preliminary investigation No hypothesisprecisely articulated© 2003, Carla EllisLots of Ways to Attack Experimentation•Not general – only applies to the “system under test”.•Not forward-looking – motivations and observations based on the past not the future.•Lack of representative workloads – inadequate benchmarks.•No culture of independent replication of other’s experiments.•Real data can be messy.Learn to do it “right”© 2003, Carla EllisA Systematic Approach1. Understand the problem, frame the questions, articulate the goals.A problem well-stated is half-solved.•Must remain objective•Be able to answer “why” as well as “what”2. Select metrics that will help answer the questions.3. Identify the parameters that affect behavior•System parameters (e.g., HW config)•Workload parameters (e.g., user request patterns)4. Decide which parameters to study (vary).© 2003, Carla Ellis5. Select technique:•Measurement of prototype implementationHow invasive? Can we quantify interference of monitoring? Can we directly measure what we want?•Simulation – how detailed? Validated against what?•Repeatability6. Select workload•Representative?•Community acceptance•AvailabilityA Systematic Approach© 2003, Carla Ellis7. Run experiments•How many trials? How many combinations of parameter settings?•Sensitivity analysis on other parameter values.8. Analyze and interpret data•Statistics, dealing with variability, outliers9. Data presentation10. Where does it lead us next?•New hypotheses, new questions, a new round of experimentsA Systematic Approach© 2003, Carla EllisSummary ofTopics•Choosing measurement or simulation techniques•Metrics•Workload selection–Standard benchmark suites–Micro benchmarks–Synthetic benchmarks–Representativeness•Monitoring–Instrumentation techniques–Timing issues–Data collection–Intrusiveness•Data analysis / statistics–Misleading with data •Workload characterization–Workload generators•Experimental design•Data presentation •Different kinds of simulators–Event driven–Trace driven–Execution driven–Validation© 2003, Carla EllisRange of Experimental Systems Research at Duke•In support of–Benchmarking (e.g., Fstress)–Workload characterization / tools (e.g., Cprof, Trickle-down)–Simulators (e.g., Modelnet)–Testbeds (e.g., PlanetLab)•Used in evaluation of systems–Almost everything else we do here.© 2003, Carla EllisHow Course Will Work•Webpage //www.cs.duke.edu/courses/fall04/cps296.2•Approximately 40 pages per week in Jain•Reading list – papers from the literature (coming soon)•Course project (leverage whatever experimental projects you have to do for thesis, 2nd year project, coursework).–Systematic approach, experimental design decisions made explicitly and justified–Mini-conference during exam week.•2 exams over readings•Class sessions: 40 minutes lecture, discussion–Assigned expertise© 2003, Carla EllisDiscussion•Introduce yourself.•What are your current research interests?–What conference represents your research community?•Do you have an on-going project?•What are your reasons for taking the course? –e.g. my advisor made me do it•Do you have prior expertise we can benefit