Validation & VerificationVALIDATION & VERIFICATIONVALIDATIONDEFINITIONSValidation & Verification ProcessMODEL BUILDINGSUGGESTIONS FOR VERIFICATIONSlide 8Suggestion 3Suggestion 3 (cont’d.)CALIBRATION & VALIDATIONSlide 123-Step Approach to ValidationStep 1. FACE VALIDITYStep 2. Validation of AssumptionsStep 2. Validation of Assumptions (contd)Step3. Validating I/O TransformationsStep 3. Validating (cont’d)Using Historical Input DataI/O Validation – Turing TestValidation TechniquesValidation Techniques (cont’d)Conclusion1Validation & VerificationChapter 102VALIDATION & VERIFICATION•Very Difficult•Very Important•Conceptually distinct, but performed simultaneously•You must be sure your model is correct •Your client must be confident that your model is correct•Should be an integral part of model building3VALIDATION•Goals –Produce a model that represents system behavior closely enough to be a substitute for the system for experimentation•Analyzing & predicting performance–Increase credibility of model to managers & decision makers4DEFINITIONS•Verification–Ensures that the model developed is correctly implemented in the software•Validation–Ensures that the model accurately represents the real-world system5Validation & Verification Process•An integral part of model design & implementation process – not separate•Most methods are informal or heuristic in nature •Experience in model development, simulation programming, & the system are essential6MODEL BUILDING1. Observation of real system1. Collect data2. Talk to members of system2. Construct conceptual model1. Assumptions about components & structure of system2. Hypothesis – values of input parameters3. Implement operational model1. Usually using simulation software**Not linear process, iterative!!7SUGGESTIONS FOR VERIFICATION1. Operational model checked by simulation software expert – not developer2. Flow diagram for each possible action3. Examine output for reasonableness under various inputs – use wide variety of output statistics4. Print input at end of run to ensure stability5. DOCUMENTATION!!!6. Ensure animation of model is correct8SUGGESTIONS FOR VERIFICATION7. Use debugger of interactive run controller (IRC) – advantages1. Can monitor simulation progress & display results2. Can focus on single line or process3. Can observe model components & variables4. Can pause; reassign values8. GUI – always recommended** Basic Software Engineering Principles9Suggestion 3•Examine output for reasonableness–Calculate expected results–Vary input–Ask users to review•Examples Suppose multiple servers & only look at throughput. Maybe too many went to one server & too few to the other. If priority queue, are they actually processed in correct order.10Suggestion 3 (cont’d.)•Utilize standard output from simulation environments•Current Count & Total Count are important variables•Consider predictions–Mathematical (e.g. Utilization)–Experts•Perform a Trace–Snapshots–By hand11CALIBRATION & VALIDATION•Validation – comparing model to system•Calibration – iterative process of comparing model to real system & adjusting the model – repeat!•Comparisons–Subjective – experts review–Objective – use of data & results12VALIDATION•Never truly completely validated–Model never totally represents the real system•Be sure model is not “fit” to one set of data133-Step Approach to ValidationNaylor & Finger [1967]1. Build a model with high face validity2. Validate model assumptions3. Compare model I/O transformations to corresponding I/O transformations for the real system14Step 1. FACE VALIDITY•Construct a model that seems valid to the users/experts knowledgeable with system•Include users in calibration – builds perceived credibility•Sensitivity Analysis – change one or more input value & examine change in results – Are results consistent with real system?–Choose most critical variables to reduce cost of experimentation15Step 2. Validation of Assumptions•2 categories of assumptions–Structural assumptions–Data Assumptions•Structural Assumptions–Involves how system operates –Includes simplifications & abstractions of reality•Data Assumptions–Based on data collection & statistical analysis16Step 2. Validation of Assumptions (contd)•Review – Analysis of Data–Identify probability distribution–Estimate parameters of distribution–Perform goodness-of-fit test•Chi Square, Kolmogorov-Smirnov tests•Test homogeneity of data–Are two independently collected sets of data come from the same parent population?17Step3. Validating I/O Transformations •Ultimate Test of a Model–Ability to predict the future behavior of the real system•Model viewed as an I/O Transformation•Can also us historical data to test prediction ability•Note: If main purpose of simulation changes, model should be revalidated18Step 3. Validating (cont’d)•Models are often used for comparing alternate system designs–Minor changes in parameters •IA rate, # servers–Minor change in statistical distribution–Major change in logical structure of subsystem•Queue discipline–Major design change•Manual to automated system19Using Historical Input Data•An alternative to randomly generated data – don’t mix different data sets•File, Spreadsheet, or Database–{A1, A2,…,An} & {S1, S2,…Sn}–Feed data into the FEL•Compare output to what happened in the real system•May be able to use technology to collect historical data for use20I/O Validation – Turing Test•What is the Turing Test?•Generate 5 “fake” reports from simulation & mix with 5 real reports; ask experts if they can distinguish fake from real•If cannot, then pass Turing Test!21Validation TechniquesIn order of cost-to-value ratio – Van Horn (1969, 1971)1. Develop model with high face validity by including experts, previous research, studies, observation, experience2. Test input data for homogeneity, randomness, goodness-of-fit3. Turing test – use knowledgeable people22Validation Techniques (cont’d)4. Compare model & system output using statistical tests5. After model development, collect new data & repeat steps 2 to 46. Build new system or redesign old one, collect data on new system & use to validate model (not recommended)7. Do little or no validation. Implement. (not recommended)23Conclusion•Do not become obsessed with validation & verification to the