PSYC 612, SPRING 2011Lecture 5: Scale Development (cont.)Lecture Date: 9/2 7/201 1Contents1 Preliminary Questions 12 Part I: Reivew of Assigned Readings (45 minutes; 5 minute break) 12.1 Classical test theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.2 Parallel Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Part II: Advanced Material - Reliability Expanded (45 minutes) 33.1 Test-retest reliability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33.2 Alternative form reliability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53.3 Split-halves reliability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53.4 Internal consistency reliability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 Part III: Advanced Optional Material (30 minutes) 71 Preliminary Questions•Have you read all the assigned reading for today?•Have you scheduled your module?2 Part I: Reivew of Assigned Readings (45 minutes; 5 minutebreak)2.1 Classical test theoryThere are many different theories in measurement. You have already been exposed to one in thislecture (True Score Theory or Classical Test Theory (CTT)) and that theory likely is the onlyone you’ll encounter during your graduate training. I say this not because it is the only plausibletheory but rather it is the one theory that every psychology graduate student learns. The othercompeting theories merely offer conditions under which CTT fails and offer promise on how toovercome those failures. It is impo r tant for you to realize that each measurement model we discuss1has some theoretical implication, just like ANOVA and MRC models. Measurement theories haveimplications as well and we will discuss those implications now.I will restrict my coverage of measurement theory to only CTT. Other measurement models thatfall under latent trait theory are beyond the scope of this class. It is best that you realize the limitedcoverage rather than believe that CTT is all that psychology has to offer in measurement. Actually,the one area t hat psychology has contributed the greatest to all of science is in measurement.Unfortunately CTT is not exactly the hallmark of success but for our discussions we will make dofor now with it.Classical test theory posits that there are two different sources of variability - t he true scoreavailable only to omniscient Jones and the observed score available t o all of us. Any differencebetween the two is considered to be error. These differences are not residuals, not inaccuracies buterror. Variance and covar iance are relevant to the model because the CTT model stipulates thatthe following equation holds for all psychological measurement:σ2Observed= σ2T rue+ σ2ErrorVariance lies at the heart of the model (i.e., the σ2’s) and the covariance is inherent in the model.Note that the relationship is merely a linear model with only one predictor (the true score) andone outcome ( t he observed score). Hence, the observed and true scores must co-vary to estimatethe model. CTT focuses on how to make sense out of the observed scores and how to relate whatwe observe to what we care to know. The relationship is a ssumed to be rectilinear as you can seefrom t he equation above and that relationship provides a convenience for computing other a spectsto characterize the nat ur e of the observed scores. Variance and covariance are essential componentsto t hat model.Where var iance comes in is in the formulation of reliability. The early psychometricians (e.g.,Lord) thought of reliability as the ability to capture the true score variance without much else.That logic lead to the following simple equation:ρ =σ2T rueσ2ObservedIf you consider the first equation that r elat es true and o bserved score variance, you will noticethat observed score variance will always exceed true score variance. When observed score varianceis high, the reliability (ρ) of that person x instrument interaction is less than what we would expectfrom a person x instrument interaction that only captured the true score. Hence, reliability is aratio of variances that expresses the error we might expect from the instrument and sample.2.2 Parallel MeasuresOne final part relevant to the discussion of classical test theory is the notion of pa r allel tests. Thereare ot her types of tests but the authors chose to discuss only parallel. Next week I intend to covertau-equivalent and congeneric measures as alt ernatives to the par allel measures discussed in theassigned readings for today.Parallel measures are measures that satisfy a very simple criterion. Two measures are par allel iftheir true scores variances and error variances are equal. That is to say that we have two measureswith the following CTT properties:2σ2O1= σ2T1+ σ2E1andσ2O2= σ2T2+ σ2E2whereσ2T1= σ2T2andσ2E1= σ2E2Of course, if these conditions are satisfied then the variance of the observed scores ought to beequal as well, right? Right. Parallel measures are instruments that purpor t to measure the sameunderlying construct a nd do so by producing the same variance. Thus, two measures are parallel ifand only if (iff for those of you who are philosophy buffs):σ2O1= σ2O2Why is this important to learn? Because parallel measures are often important for us to usewhen we know that testing may be a threat to validity. We might use two different forms of thesame instrument to reduce fa milia rity, boredom, or reactivity in our study. To use two differentinstruments that are not parallel would be a problem. Parallel tests ensure tha t t he varianceproduced by the use of the instrument remains constant and therefore does not introduce a confoundinto the study while trying to protect against another confound.Parallel measures are also important for us to learn so you can better understand the mechanismsfor estatiming reliability in classical test theory. The next part o f today’s lecture delves into thistopic further.3 Part II: Advanced Material - Reliability Expanded (45minutes)Since validity is not really accessible to us by any empirical means, classical test theory tends tofocus almost exclusively on reliability. What I mean by focus is that t he statistical tools in classicaltest theory are devoted to estimating reliability. The following discussion addresses four generalmethods for estimating reliability in standard social science instruments. Reliability