PSYC 612, SPRING 2007Lecture 11: Choosing Statistical ProceduresApril 9, 2007Contents1 Part 1: A cursory review of the assigned readings 11.1 Tabachnick and Fidell (2007) - Chapter 2 . . . . . . . . . . . . . . . . . . . . 11.2 Chervany, Benson, and Iyer (1980) . . . . . . . . . . . . . . . . . . . . . . . 22 Part 2: Important aspects not covered in the readings 32.1 Simple minded or Muddle headed? . . . . . . . . . . . . . . . . . . . . . . . 32.2 Alternatives to simple minded decision trees . . . . . . . . . . . . . . . . . . 42.3 How do you learn what to use under different circumstances? . . . . . . . . . 43 Part 3: Some ugly details 51 Part 1: A cursory review of the assigned readingsThe two assigned readings were intentionally concrete so that you may see how many viewstatistics - as a deterministic process that may be reasonably automated through the useof formal decision aids. Before I comment on that approach, let me review the assignedreadings. I plan on coming back to that point shortly.1.1 Tabachnick and Fidell (2007) - Chapter 2Recall from last semester that Tabachnick and Fidell provided a decision tree for statisticaltechniques. That tree began with the “Major Research Question”, then directed you tothink about the nature of the data (e.g., number and type of dependent and independentvariables), covariates, and finally analytic strategy. Once you were through those aspects ofyour study, you finally arrived at the “Goal of Analysis.” I asked you to read this chapteragain for this lecture because I hope that you have learned something from the subsequentreadings that might lead you to think twice before using this decision tree. A decision treeoffers novice users some sense of guidance. That guidance, however, has many requirements.1For example, how might a novice conceptualize the “Major Research Question” that isconsistent with Tabachnick and Fidell’s “Degree of relationship among variables?” Perhapsthe novice might see this as an opportunity to check for relationships before running any“real” models. Thus, the novice would follow the decision tree and run bivariate correlationson all continuous variables or perhaps the novice might realize the fact that both the IV’sand DV’s are continuous and run a canonical correlation analysis. Neither of these analyseswould likely deliver what the novice really wanted.The novice is easy to discuss. Now let us consider the novice who now knows some basicstatistical terminology and can run many of the basic statistical procedures. We might referto this advance novice, an intermediate level student. Also, that desc ription may fit you. Ifit does, how might you use the decision tree? I suspect not in a very efficient manner. Why?Because you now know enough to start skipping from the major categories to the final columnwhere the goals of the analysis are listed. Once you achieve some level of understanding,you no longer stick with the tedium of a decision tree. Skipping those beginning parts mayspeed up the process but you may neglect some important aspects of either your data oryour question. Thus, over-riding a decision tree because you think you know more than thetree assumes, you may be making more mistakes than if you stuck to the tedious process.Experts may do well to heed the decision tree as well. We know that even though expertsmay have mastery over a content area, these “experts” still make mistakes. Decision treesoffer the user a method that prompts the user for each step and guides the decision-makingprocess in such a way that errors are (we hope) minimized. Where experts may find theTabachnick and Fidell’s decision tree cumbersome is in the circularity of the tree. For aspecific example, I refer to the last section of the tree on page 31. The “Major ResearchQuestion” entails a “[t]ime course of events” and the “Goal of Analysis” options all pertainto time as either the predictor or the outcome. If an expert knows the nature of the probleminvolves estimating the time course of events, then the exp e rt will not need to sift throughall the listed procedures. She will know the correct method. Moreover, the expert will knowthat only a limited subset of goals (i.e., the “Goal of Analysis”) may be relevant given aparticular data structure. In those cases, the expert ought to quickly see that a study thathas only time as an outcome may be a suitable dataset for only one analytic model - survivalanalysis.1.2 Chervany, Benson, and Iyer (1980)The article provides details of a method for teaching and understanding statistical reasoning.To summarize, the authors proposed a non-recursive hierarchical learning structure thatthey thought was both logical and natural. The article details a refinement of a previouslypublished manuscript detailing a three stage statistical reasoning process model. The stageswere as follows:• Comprehension: The individual formulates the problem, identifies key features thatmake that problem similar to other known exemplars, and adapting the recognized2problem to the relatively novel statistical language. This stage rests upon the indi-vidual’s ability to recognize reasonably unfamiliar topics, keywords, or concepts andcategorize them.• Planning and Execution: Applying the concepts and skills from the first stage tocircumscribed problems. This stage is largely mechanical.• Evaluation and Interpretation: The final stage is the cornerstone of statistical reason-ing. A competent analyst will be able to provide a coherent logic to the statisticalanalysis and interpret the results in a manner that anyone familiar with the topic canreadily follow.The authors go on to show how the three stage model may be taught and refined throughthe use of decision trees. In fact, the article details a course project where all students areresponsible for constructing a detailed decision tree. The authors reason that if a student isforced to think through the steps of the data analysis, those steps may be formulated intoconcrete steps via a decision tree.2 Part 2: Important aspects not covered in the read-ingsThere are many different ways to think about statistical reasoning and there is no best way- at least that I am aware of now . We may approach statistical reasoning as if there werea proper way to approach each problem. Tabachnick and Fidell and Chervany et. al.’sdecision trees make this assumption. Any deterministic process must hold to the tenets