David Tenenbaum – GEOG 110 – UNC-CH Fall 2005Strategies for Analyzing and Using Environmental Systems Models1. Problem Definition – From the outset we need to know what questions we want to answer2. Model Validation and Calibration – We need to make sure that the model is structurally and predictively valid3. Exploratory Analysis – Through disturbance experiments and sensitivity analysis, we can determine how the system will perform under various conditions4. Case Studies – We apply the model to specific scenarios of interest so we can use it to help understand a system and/or make decisionsDavid Tenenbaum – GEOG 110 – UNC-CH Fall 2005Analysis and Use Strategy –Model Calibration• Ideally, the models we use fulfill the requirement of having structural validity, representing the real world in a way that is very logical and transparent, with the cause and effect relationships well defined• However, there are situations where we can create models that have superb predictive validity that sacrifice structural validity:WhiteBoxBlackBoxHigher structural validityLower predictive validityProcess-oriented modelsLower structural validityHigher predictive validityEmpirical ModelsDavid Tenenbaum – GEOG 110 – UNC-CH Fall 2005Analysis and Use Strategy –Model Calibration• When we use models that have some ‘black box’ qualities, in order to get useful results, we usual have to calibrate the model• In such a model, it is difficult to know beforehandwhat some of the parameter values should be to produce predictively valid results, usually because the parameter itself represents a quantity that cannot simply be observed in the environment• In this situation, we often run the model iteratively, and with each iteration we adjust the parameter(s)slightly, and compare results with observations• We aim to find the parameter value(s) that produces the best results and use themDavid Tenenbaum – GEOG 110 – UNC-CH Fall 2005Analysis and Use Strategy –Model Calibration•We compare observations from the real world to simulated results using any of the techniquesdescribed in the context of validation, e.g. R2, RMSE, r, the mean to variance ratio, etc.• However, while they can make use of the some of the same mathematical approaches, its important to distinguish between validation and calibration:– The goal of calibration is to find the ideal parameters for the model for a particular situation/application– The goal of validation is to verify the model’s validity in a particular situation/applicationDavid Tenenbaum – GEOG 110 – UNC-CH Fall 2005Analysis and Use Strategy –Model Calibration• Validation and calibration are closely related …1. To validate a model, you already must have selected some parameters to use (e.g. calibrated it)2. If a model is not valid, there is no point in calibration3. Both activities rely on comparing observed data to model output4. A calibrated model can only be validated in the same situation/application where it has been calibrated5. The same observations cannot be used for calibration and validation, as this introduces circular logic (we must set aside some observed data for validation … a typical approach is an 80/20 split)David Tenenbaum – GEOG 110 – UNC-CH Fall 2005Analysis and Use Strategy –Exploratory Analysis• Once we have our model built to address a particular problem, and have validated (and if necessary, also calibrated) it, we turn towards how it will performunder various conditions• We are interested not just in how it will perform under normal conditions, but also in what it will do in times of disturbance– disturbance – a dramatic change in the system within a short period of time• We can consider these to be experiments with the model to evaluate performance under different conditionsDavid Tenenbaum – GEOG 110 – UNC-CH Fall 2005Analysis and Use Strategy –Exploratory Analysis• Goal: To understand how the system reacts to changing stimuli• Applications/implications:1. Scientists can project how the application of a certain policy or introduction of a new technology might impact a system2. This can lead to new or unanticipated options for affecting the behavior of a system by changing something• While in principle, we believe that system behavior is a function of the structure, in practice it sometimes is difficult or impossible to predict behavior• The same structure can lead to dramatically different behaviors depending upon the exact details of the system (rates, initialization values etc.)David Tenenbaum – GEOG 110 – UNC-CH Fall 2005Analysis and Use Strategy –Exploratory Analysis• We can address the following kinds of questionthrough exploratory analysis:1. What is the role of a particular system element in the overall model behavior?2. Which model elements have big impacts on the overall system, and which have limited impacts?3. What are the synergistic effects of groups of elements in the system that together exert an unusually large influence on the system behavior?4. How does the system react to disturbance?5. What conditions will cause the system to collapse or run out of control?David Tenenbaum – GEOG 110 – UNC-CH Fall 2005Analysis and Use Strategy –Disturbance Experiments• When the condition of a system is stable, it is often difficult to understand its dynamics– By definition, we observe dynamics when something is happening and changing, so if a system reaches a steady state, even though its processes are all still functioning, theyare doing so in an equilibrium that does not allow you to really see what is going on•A deliberate disturbance, or shock to the system often forces the system to readjust, possibly in ways that are not predictable– For example, if an exotic species is introduced into an ecosystem, it may disturb the system to such an extent that many of the native species perishDavid Tenenbaum – GEOG 110 – UNC-CH Fall 2005Analysis and Use Strategy –Disturbance Experiments• We do not necessarily have to introduce a new element (e.g. an exotic species) to a system to shock it; we can simply make a change in some quantity in the system– e.g. we can increase some reservoir’s content by a significant amount, or change a rate (which in turn will affect stocks …)• It usually takes some time for the system to adjust to a radically new condition … during the adjustment period we may observe rather different behavior•