Qualitative Model ComparisonsQ550: Models in Cognitive ScienceLecture 2Comparing ModelsBuilding a model to produce and explain a behavioralphenomenon is both challenging and rewardingBut, we further science by constraining between differentmodels…formalize and reject models based on dataIdeally, we would love to do a qualitative comparisonbetween models, that is, a comparison of formalizedprinciples: free from ad hoc assumptions independent of specific model parametersE.g., in Minerva, if we compare exemplar and prototypeversions, the ordinal predictions should not differ atvarious levels of FOur ideal situation:Theory 1 is consistent w/ empirical data, whereas Theory 2 isnot. Further, Theory 2 cannot produce the result for any set ofparameters, whereas, Theory 1 produces the result for allparameter values and assumptions…of course, this rarely occurs in practiceInputTheory 1A > BInputTheory 2A < BInputHumanA > BComparing Models…of course, this rarely occurs in practiceOften, we look at the proportion of total parameters thatwould produce the observed effectOR: we find the best fitting set of parameters (estimatedfrom the data) and compare the models qualitativelyComparing Models• Levels of model comparison: Pure qualitative comparison independent of specificparameter values Proportion of total parameter that produce the effect(signed-difference test) Quantitative comparison to determine which model ismost likely to have produced the data (assuming allmodels predict the ordinal differences) Generalizing to other paradigmsMinimize task-specific parameters -- when these arechanged, model should generalize to other tasks w/ thesame general domainA Qualitative Comparison in MinervaWe have an existence proof: It is unnecessary to positboth an abstract and episodic memoryBut: do we have any reason to prefer one over the other?Could we produce the data as well using a single prototypeper category as well as multiple episodes? Let’s keep everything the same in Minerva, but insteadof storing multiple episodes, we’ll create a composite foreach category as they are learned, and will just store thesethree prototypesWhat happens?A prototype version of Minerva cannot explain theimmediate benefit for old exemplarsIt also cannot explain the interaction (crossover) betweenold exemplars and the prototype as a function of forgetting A prototype version of Minerva would not explain theordinal differences in conditions b/c we would losedifferential activation of stored episodes(actually, I’m guessing here --> test this intuition byactually manipulating the code)Serial Learning and Serial/Free RecallLet’s consider a second example: we’re interested inlearning items in a sequence and recalling them (in theorder presented, or in any order)Random series of words: 1 word per second. Rememberthe words and recall them after the presentationYou may be asked to recall them in the order presented, orin any order you chose+gravelkeyboardgiraffefountainpaperangelmagazinedoorcamera********Free Recall: What are we doing?Rehearsing? We’ll call this a strength accountEach goes in once? Decay accountAssociative chaining? (S-R theory)Combinations?Serial position curves: How would these accounts producethem?If a model doesn’t predict R > P > M then reject itIf our models all make the same ordinal predictions, thenwe move onto qualitative model comparisonsRemember Gallileo’s ballsTODAM, SAM, Pertubation theory (Neath Website)Example from BusemeyerA well-designed 2-D category learning experimentDescribes sophisticated connectionist versions of exemplarand prototype models--bivariate receptive fields, deltalearning error backprop, etc.The models have the same parameters:α: learning rateb: sensitivity for category choiceσ: discriminability for width of generalization gradientsc: recognition sensitivityβ: recognition response biasExample from BusemeyerXOR learning:.98.05.03.94S2=1 S2=10S1=10S1=1He shows w/ a closed form solution that a prototype modelcannot account for the crossover due to the independenceaxiom, and w/ simulation that the exemplar model can for alarge proportion of its parameters…or, we could just simulate bothNosofsky & Zaki (2002)Using the exemplar-based model that Busemeyerdescribes, they show that the dissociation betweenamnesiacs and normals can be accounted for byparameter shiftsAnother existence proof: no need to postulate multiplememory systemsWe’ll talk about things like backprop, d-prime, etc. in detaillater.Quantitative Model ComparisonIt is often impossible to differentiate between modelsbased on a qualitative comparison, so we decide whichmodel is most likely to have generated the dataA model should make quantitative predictions that aremore accurate than its competitorsThe qualitative test must be based on an optimal selectionof parameters…otherwise we could reject a perfectlygood model simply by choosing a poor set or parmsFor each model1. Find the best-fitting model parameters2. Compare the quantitative accuracy of predictionsbased on these optimal parametersQuantitative Model ComparisonThis is complicated b/c model complexity needs to beconsidered in the comparison # of free parameters, model assumptions, etc. Nested and non-nested models Selection of the best model must satisfy bothaccuracy and parsimonyNonlinear parameter estimationObjective functions to minimize/maximize (e.g., chi-square,least squares, log-likelihood)Null and saturated modelsLinear vs. nonlinear models (special condition: mean ofpredictions from two different sets of parameters = theprediction produced by the average of the two sets;nonlinear makes parameter estimation more complicated)Nonlinear parameter space estimation: grid search, hill climbing, steepest descent, simplex constraints on parameters flat/local minima problemModel comparison techniquesWhen we have the optimal set of parameters for eachmodel, we can compare their fit to data using a varietyof techniques:1. If nested: use classic chi-square2. Bayesian model comparison (BIC: which of 2 models ismost likely to be correct given the sample data);provides a measure of evidence for model A comparedto model B, relative to # of free parameters3. AIC method: Selects the model that generates adistribution closest to the true distribution --> select themost likely generating model4. Minimum description length (equivalent to BIC for big N)Fit is not everythingThese techniques