Design of ExperimentsIntroduction to Simulation - Spring 2010Dr. Louis LuangkesornUniversity of PittsburghMarch 2, 2010Dr. Louis Luangkesorn ( University of Pittsburgh ) Design of Experiments March 2, 2010 1 / 21Outline1Introduction2Basic concepts3Potential Experimental Designs4SummaryDr. Louis Luangkesorn ( University of Pittsburgh ) Design of Experiments March 2, 2010 2 / 21IntroductionGoalsTo understand the basic concepts of experimental design.To recognize the types of goals that experimental design represents.To understand the what makes simulation experiments different.Dr. Louis Luangkesorn ( University of Pittsburgh ) Design of Experiments March 2, 2010 3 / 21IntroductionThe curse of dimensionalityThe “Roadrunner” computer - 1 pentaflop (thousand trillion operations persecond)If you have 100 factors, two levels each, one operations per simulation,one run -> 400 years.Moore’s law - computing performance doubles every two years.But our problem sizes are factorial.We cannot rely on brute force to explore a system with simulation.Dr. Louis Luangkesorn ( University of Pittsburgh ) Design of Experiments March 2, 2010 4 / 21IntroductionGoals of a simulation analystsDevelop a basic understanding of the simulation model or system.Find robust decisions or policies, orCompare the merits of various decisions or policiesDr. Louis Luangkesorn ( University of Pittsburgh ) Design of Experiments March 2, 2010 5 / 21Basic conceptsBasic conceptsFactor - Input variables. Can be treatment factors (controlable) or noisefactors (not controlable).Levels - The values a factor can take on.Response - Output of the system.Interaction - The value of one factor affects the response of the systemdue to another factor.Significance - A finding that a factor or interaction has a non-negligibleeffect on a response.Experimental units - The subjects that the experiment is run on (i.e. thesystem under consideration)Experimental Design - The rule that identifies factor combinations andassigns them to experimental units.Replication - Each repetition of a factor combination. The total number ofexperimental units is the sum of the replications for each experimentalunit.Dr. Louis Luangkesorn ( University of Pittsburgh ) Design of Experiments March 2, 2010 6 / 21Basic conceptsTypes of factors (variables)Quantitative vs. qualitative - Quantitative factors take on numericalvalues, while qualitative factors do not.Discrete vs. continuous - Discrete factors can have levels only at certainseparated values. Continuous factors can assume any real value (within aspecified range).Binary - Factor is naturally constrainted to just to levels. e.g. “hi-low”,“on-off”, “yes-no”.Controllable or uncontrollable - If a factor can be effectively controlled ornot. While in computer simulation all factors are controlled, in a realsystem it may not be cost-effective to control certain factors.Dr. Louis Luangkesorn ( University of Pittsburgh ) Design of Experiments March 2, 2010 7 / 21Basic conceptsTypes of modelsMain-effects model - Responses are due only to a linear combination offactor levels, with no interaction effects.Y = β0+k∑i=1βiXi+ εComplete models - Responses due to both main effects and interactions.Y = β0+k∑i=1βiXi+ (k−1∑i=1k∑j=i+1βij(XiXj)) + (∑i∑j∑kβijkXiXjXk) + . . .Second order - Responses due to main effects, two-way interactionsDr. Louis Luangkesorn ( University of Pittsburgh ) Design of Experiments March 2, 2010 8 / 21Basic conceptsAlternatives to designed experimentsTrial-and-error“This combination looks interesting”Baseline and vary one factor at a time.Dr. Louis Luangkesorn ( University of Pittsburgh ) Design of Experiments March 2, 2010 9 / 21Basic conceptsChoosing factorsInput or distribution parameters of a simulation model.DecisionsTry not to have too many, only the ones relavent to the question at hand.Dr. Louis Luangkesorn ( University of Pittsburgh ) Design of Experiments March 2, 2010 10 / 21Potential Experimental DesignsClasses of Experimental Designs2kfactorial designsmkfactorial designs2k−pResolution 5 fractional factorial designsCentral Composite Design (CCD)Latin HypercubeRobust design methodsSequential creening methodsResponse Surface MethodologyDr. Louis Luangkesorn ( University of Pittsburgh ) Design of Experiments March 2, 2010 11 / 21Potential Experimental Designs2kfactorial designsFor k factors, two levelsAll combinations of each level of each factorOrthogonal (easy to analyze)Often used to screenFigure: 22and 23factorial designs Sanchez and Wan (2008)Dr. Louis Luangkesorn ( University of Pittsburgh ) Design of Experiments March 2, 2010 12 / 21Potential Experimental Designsmkfactorial designsm levels for each of k factorsReveals complexity in the response (i.e. non-linear responses)Number of design points increases very fast.Unless there are complex interaction effects, very inefficient.Figure: mkfactorial designs Sanchez and Wan (2008)Dr. Louis Luangkesorn ( University of Pittsburgh ) Design of Experiments March 2, 2010 13 / 21Potential Experimental Designs2k−pResolution 5 fractional factorial designsWhat if we did not have to consider all potential interactions?We would require fewer treatment combinationsResolution 3 - Estimate main-effect only.Resolution 4 - Estimate main effects only when two-way interations arepresent.Resulution 5 - A half-fraction that will allow two-way interactions to beexplored.Fractional factorial designs are usually generated using computerprograms to determine the correct design points.Dr. Louis Luangkesorn ( University of Pittsburgh ) Design of Experiments March 2, 2010 14 / 21Potential Experimental DesignsCentral Composite DesignStart with 2kdesign, then fill in star pointsUsed to estimate second-order models with fewer points then 3kdesignFigure: Partial factorial and Central Composite Design Sanchez and Wan (2008)Dr. Louis Luangkesorn ( University of Pittsburgh ) Design of Experiments March 2, 2010 15 / 21Potential Experimental DesignsLatin Hypercube SamplingChoose m design points with k factors.Each factor divided into m equally spaced values.To determine design points, random sampling without replacement fromthe candidate values for each factor.Figure: Latin Hypercube Sampling Sanchez and Wan (2008)Dr. Louis Luangkesorn ( University of Pittsburgh ) Design of Experiments March 2, 2010 16 / 21Potential Experimental DesignsRobust design methodsRecognize the difference between decision factors and noise factors.Use