Slide 1Slide 2Slide 3Slide 4Slide 5Slide 6Slide 7Slide 8Slide 9Importance of Flux TargetsSlide 11Slide 12Slide 13Slide 14Slide 15Slide 16Slide 17Slide 18Slide 19Slide 20Slide 21Slide 22Slide 23Slide 24Slide 25Slide 26Slide 27Slide 28Slide 29Slide 30Slide 31Slide 32Slide 33Slide 34Slide 35Slide 36Uncertainty Analysis and Model “Validation” or Confidence BuildingConclusions• Calibrations are non-unique.• A good calibration (even if ARM = 0) does not ensure that the model will make good predictions.• Field data are essential in constraining the model so that the model can capture the essential features of the system.• Modelers need to maintain a healthy skepticism about their results.• Head predictions are more robust (consistent among different calibrated models) than transport (particle tracking) predictions.Conclusions• Need for an uncertainty analysis to accompany calibration results and predictions.Ideally models should be maintained for the long termand updated to establish confidence in the model.Rather than a single calibration exercise, a continualprocess of confidence building is needed.Uncertainty in the CalibrationInvolves uncertainty in: Parameter values Conceptual model including boundary conditions,zonation, geometry, etc.  TargetsZonationKrigingTo use conventional inverse models/parameter estimationmodels in calibration, you need to have a pretty good idea of zonation (of K, for example).Also need to identify reasonable ranges for thecalibration parameters and weights.(New version of PEST with pilot points does not need zonation as it works with continuous distribution of parameter values.)Zonation vs Pilot Points• Field data are essential in constraining the model so that the model can capture the essential features of the system.Parameter ValuesCalibration Targetscalibration valueassociated error20.24 m0.80 mTarget with relativelylarge associated error.Target with smaller associated error.Need to establish model specific calibration criteria and define targets including associated error.Examples of Sources of Errorin a Calibration Target• Surveying errors • Errors in measuring water levels• Interpolation error• Transient effects• Scaling effects• Unmodeled heterogeneitiesImportance of Flux Targets When recharge rate (R) is a calibration parameter, calibrating to fluxes can help in estimating K and/or R.R was not a calibration parameter in our final project.H1H2q = KIIn this example, flux information helps calibrate K.Here discharge information helps calibrate R.QH1H2q = KIIn this example, flux information helps calibrate K.All water discharges to the playa.Calibration to ET merely fine tunesthe discharge rates within the playaarea. Calibration to ET does nothelp calibrate the heads and K valuesexcept in the immediate vicinityof the playa.In our example, total recharge is known/assumed to be 7.14E08 ft3/year and discharge = recharge.Smith Creek Valley (Thomas et al., 1989)Calibration Objectives (matching targets)1. Heads within 10 ft of measured heads. Allows forMeasurement error and interpolation error.2. Absolute residual mean between measured and simulated heads close to zero (0.22 ft) and standard deviation minimal (4.5 ft). 3. Head difference between layers 1&2 within 2 ft of field values.4. Distribution of ET and ET rates match field estimates.Includes results from2006 and 4 other years724 Project ResultsA “good” calibrationdoes not guaranteean accurate prediction.?Sensitivity analysis to analyze uncertaintyin the calibrationUse an inverse model (automated calibration) to quantify uncertainties and optimize the calibration.Perform sensitivity analysis during calibration.Sensitivity coefficients(Zheng and Bennett)Sensitivityanalysis performedduring the calibrationSteps in ModelingcalibrationloopUncertainty in the Prediction Involves uncertainty in how parameter values(e.g., recharge) or pumping rates will varyin the future.  Reflects uncertainty in the calibration.Stochastic simulationWays to quantify uncertaintyin the predictionScenario analysis - stressesSensitivity analysis - parameters(Zheng and Bennett)Sensitivityanalysis performedafter the predictionSteps in ModelingTraditional ParadigmMulti-modelAnalysis (MMA)Predictions and sensitivityanalysis are insidethe calibration loopFrom J. Doherty 2007New Paradigmfor Sensitivity& ScenarioAnalysisStochastic simulationWays to quantify uncertaintyin the predictionScenario analysis - stressesSensitivity analysis - parametersMADE site – Feehley and Zheng, 2000, WRR 36(9).Stochastic simulationStochastic modeling option available in GW VistasA Monte Carlo analysis considers 100 or more realizations.0204060801001201401 2 3 4 5 6 7Drawdown at pumping wellnumber of realizationsZheng & BennettFig. 13.2Hydraulic conductivityInitial concentrations(plume configuration)BothZheng & BennettFig. 13.5Reducing UncertaintyHard data onlySoft and hard dataWith inverse flow modelingHypotheticalexampletruthZ&BFig. 13.6How do we “validate” a model so thatwe have confidence that it will makeaccurate predictions?Confidence BuildingModeling Chronology1960’s Flow models are great!1970’s Contaminant transport models are great!1975 What about uncertainty of flow models?1980s Contaminant transport models don’t work. (because of failure to account for heterogeneity) 1990s Are models reliable? Concerns overreliability in predictions arose over efforts to modelgeologic repositories for high level radioactive waste.“The objective of model validation is to determine how well the mathematical representation of the processes describes the actual system behavior in terms of the degree of correlation between model calculations and actual measured data”(NRC, 1990)Hmmmmm…. Sounds like calibration…What they really mean is that a valid model willyield an accurate prediction.Oreskes et al. (1994): paper in Science Calibration = forced empirical adequacy Verification = assertion of truth (possible in a closed system, e.g., testing of codes) Validation = establishment of legitimacy (does not contain obvious errors), confirmation, confidence building What constitutes “validation”? (code vs. model)NRC study (1990): Model validation is not possible.How to build confidence in a modelCalibration (history matching) steady-state calibration(s) transient calibration“Verification” requires an