DOC PREVIEW
AUBURN COMP 8700 - Experimentation

This preview shows page 1-2-3-26-27-28 out of 28 pages.

Save
View full document
View full document
Premium Document
Do you want full access? Go Premium and unlock all 28 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 28 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 28 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 28 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 28 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 28 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 28 pages.
Access to all documents
Download any document
Ad free experience

Unformatted text preview:

ExperimentationChoice of Run LengthsIdentification of Steady StateNumber of Simulation RunsUsing Multiple RunsChapter 4 SummarySargent’s DefinitionsValidation ProcessModeling TermsValidation TechniquesValidation Techniques (2)Validation Techniques (3)Validation Techniques (4)Validation Techniques (5)Data ValidityConceptual Model ValidationModel VerificationOperational ValidityHypothesis TestingType I and Type II ErrorsEvery hypothesis test has five essential componentsHypothesis Testing ProtocolHypothesis Testing Protocol (cont.)Formulate the conclusionDocumentationRecommended ProcedureRecommended ProcedureDistributed Simulation – Verification and Validation1Experimentation• Having constructed a model and obtained an implementation, the stage is set for actual use of the simulation. • Even at this point, however, it does not suffice simply to applythe inputs and collect whatever output follows. • Lest we come to erroneous conclusions about the meaning of a particular simulated result, it is necessary to remember that every simulation run is an experiment. • The inputs that we provide are the controlled parameters for the experiment, and the simulation itself governs the outcomes that are observed. • As with all experiments, it is necessary to take precautions that ensure that the combined influence of factors which are exogenous to the system of interest is minimized.Distributed Simulation – Verification and Validation2Choice of Run Lengths• Assuming an adequate model, the validity of conclusions may be dependent upon dynamic characteristics that are inherent in the system. • Under such circumstances, output from identical inputs to the model may vary considerably as a function ofthe time that the simulation is allowed to run. • An example of such behavior from a simulation of disk queueing in a client-server system is shown reaching steady-state. 0.0680.0690.070.0710.0720.0730.0740.0750.0760.0770 50000 100000 150000ObservationsQueuing delay (sec)Simulation dataQuadratic fitDistributed Simulation – Verification and Validation3Identification of Steady State• The phenomenon observed in the previous slide is a familiar one in dynamic systems such as networks of queues. – In such systems, a relatively short period of transient behavior is observed at system start-up, during which the state variables which describe the system tend to change quickly with small increments of simulated time. – This stage is followed by a resolution of the system at equilibrium, where system outputs are essentially stable. – As this steady-state condition dominates the behavior of the system over time, it is often the case that the primary interest of the investigation concerns the response of the system once equilibrium is achieved. – It is therefore necessary to consider the dynamic characteristics of the model before measurements are taken.• When the presence of a significant transient period is suspected, it becomes necessary to estimate its length. • Examination of the system output in a graphical form over time is a useful technique, but because the interpretation of the magnitude of a state variable’s change is largely scale-dependent, it may not be obvious when the system’s steady-state condition has been achieved.Distributed Simulation – Verification and Validation4Number of Simulation Runs• Typically one must make several runs of a simulation in order to obtain confidence intervals for the statistic of interest. • Choosing the exact number of runs may be straightforward, if it is known that a single run produces a single independent observation of the desired behavior. – For example, if we are estimating the mean of a normally distributed random variable, we simply specify a desired precision ∆ and confidence α and solve the equations given in Chapter 3 for the width of the confidence interval.– If each run produces more than one independent observation, then the choice of the number of runs becomes more philosophical.Distributed Simulation – Verification and Validation5Using Multiple Runs• If one agrees that the random number generator is well validated, then we would have no reason to believe that any particular run is not representative, assuming a large number of observations. – Therefore a single run of some length essentially equates to as many individual runs of length one. – Most people that work with simulations would feel uncomfortable with this approach, though, because the random number generator tests that are employed are generally empirical. – As such, they do not offer proof that the RNG is satisfactory over all intervals. – Thus, most investigators would want to examine the performance of a simulation over the course of several independent runs, using different seeds, even if it would be feasible to collect enough observations to form an acceptable confidence interval in a single runDistributed Simulation – Verification and Validation6Chapter 4 Summary• The creation of a simulation that accurately portrays the relevant characteristics of the system of interest can be very challenging. • While many aspects of the process are mechanical, there are many issues which must be resolved through observation and experience. • As such, the production of a successful simulation incorporates a certain amount of artistry in addition to hard science. • In the process of designing and using a simulation, one must always guard against the tendency to assume that every result that we observe has a basis in fact. • Thus, a healthy measure of good sense, coupled with knowledge of the application domain, must be employed when analyzing the output.Distributed Simulation – Verification and Validation7Sargent’s Definitions• Model validation is usually defined to mean “substantiation that a computerized model within its domain of applicability possesses a satisfactory range of accuracy consistent with the intended application of the model.”(Schlesinger, et al. 1979 “Terminology for Model Credibility,” Simulation, 34, 3 pp 101-105.)• Model verification is often defined as “ensuring that the computer program of the computerized model and its implementation is correct.• Credibility is concerned with sufficiently developing the confidence that (potential) users have in the model and the information derived from that model and the derived information.Distributed Simulation –


View Full Document

AUBURN COMP 8700 - Experimentation

Download Experimentation
Our administrator received your request to download this document. We will send you the file to your email shortly.
Loading Unlocking...
Login

Join to view Experimentation and access 3M+ class-specific study document.

or
We will never post anything without your permission.
Don't have an account?
Sign Up

Join to view Experimentation 2 2 and access 3M+ class-specific study document.

or

By creating an account you agree to our Privacy Policy and Terms Of Use

Already a member?