Model criticism and selectionModel criticism and selectionModel criticism and selectionModel criticism and selectionModel criticism and selectionSensitivity analysisSensitivity analysisSensitivity analysisSensitivity analysisSensitivity analysisPrior partitioningPrior partitioningPrior partitioningPrior partitioningModel assessmentModel assessmentModel assessmentModel assessmentBayes factorsBayes factorsBayes factorsBayes factorsBayes factorsOther Classes of AlternativesOther Classes of AlternativesOther Classes of AlternativesProduct Space SearchProduct Space SearchProduct Space SearchProduct Space SearchProduct Space SearchProduct Space SearchProduct Space Search``Metropolized'' Product Space Search``Metropolized'' Product Space Search``Metropolized'' Product Space Search``Metropolized'' Product Space Search``Metropolized'' Product Space SearchPredictive Model SelectionPredictive Model SelectionPredictive Model SelectionPredictive Model SelectionExtension to Hierarchical ModelsExtension to Hierarchical ModelsExtension to Hierarchical ModelsExtension to Hierarchical ModelsExtension to Hierarchical ModelsExtension to Hierarchical ModelsHierarchical model complexityHierarchical model complexityHierarchical model complexityHierarchical model complexityModel selection via DICModel selection via DICModel selection via DICModel selection via DICIssues in using DICIssues in using DICIssues in using DICIssues in using DICIssues in using DICIssues in using DICModel criticism and selectionThree related issues to consider:Chapter 6: Model Criticism and Selection – p. 1/17Model criticism and selectionThree related issues to consider:Robustness: Are any model assumptions having anundue impact on the results? (text, Sec. 6.1)Chapter 6: Model Criticism and Selection – p. 1/17Model criticism and selectionThree related issues to consider:Robustness: Are any model assumptions having anundue impact on the results? (text, Sec. 6.1)Assessment: Does the model provide adequate fit tothe data? (text, Sec. 6.2)Chapter 6: Model Criticism and Selection – p. 1/17Model criticism and selectionThree related issues to consider:Robustness: Are any model assumptions having anundue impact on the results? (text, Sec. 6.1)Assessment: Does the model provide adequate fit tothe data? (text, Sec. 6.2)Selection: Which model (or models) should wechoose for final presentation? (text, Secs. 6.3–6.5)Chapter 6: Model Criticism and Selection – p. 1/17Model criticism and selectionThree related issues to consider:Robustness: Are any model assumptions having anundue impact on the results? (text, Sec. 6.1)Assessment: Does the model provide adequate fit tothe data? (text, Sec. 6.2)Selection: Which model (or models) should wechoose for final presentation? (text, Secs. 6.3–6.5)Consider each in turn...Chapter 6: Model Criticism and Selection – p. 1/17Sensitivity analysisMake modifications to an assumption and recompute theposterior; any impact on interpretations or decisions?No: The data are strongly informative with respect tothis assumption (robustness)Chapter 6: Model Criticism and Selection – p. 2/17Sensitivity analysisMake modifications to an assumption and recompute theposterior; any impact on interpretations or decisions?No: The data are strongly informative with respect tothis assumption (robustness)Yes: Document the sensitivity, think more carefullyabout it, and perhaps collect more data.Chapter 6: Model Criticism and Selection – p. 2/17Sensitivity analysisMake modifications to an assumption and recompute theposterior; any impact on interpretations or decisions?No: The data are strongly informative with respect tothis assumption (robustness)Yes: Document the sensitivity, think more carefullyabout it, and perhaps collect more data.Examples of assumptions to modify: increasing/decreasing a prior mean by one prior s.d.; doubling/halving a prior s.d.; case deletion.Chapter 6: Model Criticism and Selection – p. 2/17Sensitivity analysisMake modifications to an assumption and recompute theposterior; any impact on interpretations or decisions?No: The data are strongly informative with respect tothis assumption (robustness)Yes: Document the sensitivity, think more carefullyabout it, and perhaps collect more data.Examples of assumptions to modify: increasing/decreasing a prior mean by one prior s.d.; doubling/halving a prior s.d.; case deletion.Importance sampling and asymptotic methods cangreatly reduce computational overhead, even if thesemethods were not used in analysis of original model.Chapter 6: Model Criticism and Selection – p. 2/17Sensitivity analysisMake modifications to an assumption and recompute theposterior; any impact on interpretations or decisions?No: The data are strongly informative with respect tothis assumption (robustness)Yes: Document the sensitivity, think more carefullyabout it, and perhaps collect more data.Examples of assumptions to modify: increasing/decreasing a prior mean by one prior s.d.; doubling/halving a prior s.d.; case deletion.Importance sampling and asymptotic methods cangreatly reduce computational overhead, even if thesemethods were not used in analysis of original model.⇒ Run and diagnose convergence for “base” model;use approximate method for robustness studyChapter 6: Model Criticism and Selection – p. 2/17Prior partitioning– a “backwards” approach to robustness!What if the range of plausible assumptions isunimaginably broad, as in the summary of agovernment-sponsored clinical trial?Chapter 6: Model Criticism and Selection – p. 3/17Prior partitioning– a “backwards” approach to robustness!What if the range of plausible assumptions isunimaginably broad, as in the summary of agovernment-sponsored clinical trial?Potential solution: Determine the set of prior inputs thatare consistent with a given conclusion, given the dataobserved so far. The consumer may then compare thisprior class to his/her own personal prior beliefs.Chapter 6: Model Criticism and Selection – p. 3/17Prior partitioning– a “backwards” approach to robustness!What if the range of plausible assumptions isunimaginably broad, as in the summary of agovernment-sponsored clinical trial?Potential solution: Determine the set of prior inputs thatare consistent with a given conclusion, given the dataobserved so far. The consumer may then compare thisprior class to his/her own personal prior beliefs.Thus we are partitioning the prior class based onpossible outcomes.Chapter 6: Model