CPE 619 Experimental DesignPART IV: Experimental Design and AnalysisIntroductionIntroduction (cont’d)Slide 5OutlineTerminologyTerminology (cont’d)Slide 9Slide 10Slide 11Common Mistakes in Experiments (cont’d)Slide 13Slide 14Simple DesignsExample of Interaction of FactorsSlide 17Full Factorial Designs2k Factorial Designs22 Factorial Design22 Factorial Design (cont’d)Slide 22Slide 23Allocation of VariationAllocation of Variation (cont’d)Slide 26General 2k Factorial DesignsGeneral 2k Factorial Designs (cont’d)Slide 29Slide 30Slide 312kr Factorial Designs22r Factorial Design Errors22r Factorial Design Errors (cont’d)22r Factorial Allocation of Variation22r Factorial Allocation of Variation ExampleConfidence Intervals for EffectsConfidence Intervals for Effects (Example)Confidence Intervals for Predicted ResponsesConfidence Intervals for Predicted Responses (cont’d)Confidence Intervals for Predicted Responses ExampleConfidence Intervals for Predicted Responses Example (cont’d)Homework #6CPE 619Experimental DesignAleksandar MilenkovićThe LaCASA LaboratoryElectrical and Computer Engineering DepartmentThe University of Alabama in Huntsvillehttp://www.ece.uah.edu/~milenkahttp://www.ece.uah.edu/~lacasa2PART IV: Experimental Design and AnalysisHow to:Design a proper set of experiments for measurement or simulationDevelop a model that best describes the data obtainedEstimate the contribution of each alternative to the performanceIsolate the measurement errorsEstimate confidence intervals for model parametersCheck if the alternatives are significantly differentCheck if the model is adequate3IntroductionGoal is to obtain maximum information with minimum number of experimentsProper analysis will help separate out the factorsStatistical techniques will help determine if differences are caused by variations from errors or notNo experiment is ever a complete failure. It can always serve as a negativeexample. – Arthur BlochThe fundamental principle of science, the definition almost, is this:the sole test of the validity of any idea is experiment.– Richard P. Feynman4Introduction (cont’d)Key assumption is non-zero costTakes time and effort to gather dataTakes time and effort to analyze and draw conclusions Minimize number of experiments runGood experimental design allows you to:Isolate effects of each input variableDetermine effects due to interactions of input variablesDetermine magnitude of experimental errorObtain maximum info with minimum effort5Introduction (cont’d)ConsiderVary one input while holding others constantSimple, but ignores possible interaction between two input variablesTest all possible combinations of input variablesCan determine interaction effects, but can be very largeEx: 5 factors with 4 levels  45 = 1024 experiments Repeating to get variation in measurement error 1024x3 = 3072There are, of course, in-between choices…Chapter 196OutlineIntroductionTerminologyGeneral MistakesSimple DesignsFull Factorial Designs2k Factorial Designs2kr Factorial Designs7TerminologyConsider an example: Personal workstation designCPU choice: 6800, z80, 8086Memory size: 512 KB, 2 MB, 8 MBDisk drives: 1-4Workload: secretarial, managerial, scientificUser’s education: high school, college, graduateResponse variable – the outcome or the measured performanceE.g.: throughput in tasks/min or response time for a task in seconds8Terminology (cont’d)Factors – each variable that affects responseE.g., CPU, memory, disks, workload, user’s ed.Also called predictor variables or predictorsLevels – the different values factors can takeE.g., CPU 3, memory 3, disks 4, workload 3, user education 3Also called treatmentPrimary factors – those of most important interestE.g., maybe CPU, memory size, # of disks9Terminology (cont’d)Secondary factors – of less importanceE.g., maybe user type not as importantReplication – repetition of all or some experimentsE.g., if run three times, then three replicationsDesign – specification of the replication, factors, levelsE.g., specify all factors, at above levels with 5 replications so 3x3x4x3x3 = 324 time 5 replications yields 1215 total10Terminology (cont’d)Interaction – two factors A and B interact if one shows dependence upon anotherE.g.: non-interacting, since A always increases by 2 A1 A2 B13 6 B25 10E.g.: interacting factors since A change depends upon BA1 A2 B13 6 B25 15A1A2B1B2A1A2B1B211OutlineIntroductionTerminologyGeneral MistakesSimple DesignsFull Factorial Designs2k Factorial Designs2kr Factorial Designs12Common Mistakes in Experiments (cont’d)Variation due to experimental error is ignoredMeasured values have randomness due to measurement error. Do not assign (or assume) all variation is due to factorsImportant parameters not controlledAll parameters (factors) should be listed and accounted for, even if not all are variedEffects of different factors not isolatedMay vary several factors simultaneously and then not be able to attribute change to any one Use of simple designs (next topic) may help but have their own problems13Common Mistakes in Experiments (cont’d)Interactions are ignoredOften effect of one factor depend upon another. E.g.: effects of cache may depend upon size of program. Need to move beyond one-factor-at-a-time designsToo many experiments are conductedRather than running all factors, all levels, at all combinations, break into stepsFirst step, few factors and few levelsDetermine which factors are significantTwo levels per factor (details later)More levels added at later design, as appropriate14OutlineIntroductionTerminologyGeneral MistakesSimple DesignsFull Factorial Designs2k Factorial Designs2kr Factorial Designs15Simple DesignsStart with typical configurationVary one factor at a timeEx: typical may be PC with z80, 2 MB RAM, 2 disks, managerial workload by college studentVary CPU, keeping everything else constant, and compareVary disk drives, keeping everything else constant, and compareGiven k factors, with ith having ni levelsTotal = 1 + (ni-1) for i = 1 to kExample: in workstation study1 + (3-1) + (3-1) + (4-1) + (3-1) + (3-1) + (3-1) = 14But may ignore interaction(Example next)16Example of