Reliability of MeasurementDegree to which a measuring instrument is consistent; that is, the same individual obtains similar scores in similar situations3 Ways to assess reliability (first thing you look at before you go on and look at something else)Test-retestAdminister the same test twice to the same individuals (see how much the results on the first test, match the results on the second test – ACT)Alternate formAdminister alternate forms of the test to the same individuals (ACT & SAT)Split-halfHalf of the time items are correlated with the other half (from a single testing session) (weakest form, look at scores on one half compared to the scores on the other half)You have to have reliability in measurementFactors that affect reliabilityWithin a sessionParticipant becomes upsetParticipant becomes illParticipant misreads a questionsParticipant guessesTest can’t be accurate if it changes with mood or something along those linesBetween sessionsAlternate forms may not be equivalentParticipant changesThings that affect between the testsPeople drop out or don’t come and take the next test it will affect reliabilityVideo on ReliabilityCrow sledding down roof with snowAnthropomorphize – when we give animals or objects human abilities like emotionPersonality questionnaires are so vague they can make anyone believe it is true about themReliability is like the foundation to your house- you have to haveYou should get about the same score on a test about every time you take itSplit-half – you should score about the same on even number questions as you would on the oddValidity of MeasurementDoes the defined measurement process actually measure the intended concept?Is the test measuring what it says its measuringSeveral ways to examine validityContent validityPerformance assessment based on information or skills to which participants were previously exposed (something we talked about in class)E.g., test questions based on information actually discussed in classMaking questions based off the areas on the study guide and no other areasFace validityPerformance assessment appears to be based on the general nature of the information or skills to which participants were previously exposedE.g., test questions based on the general category of information discussed in classWere the questions about what you were expectingIf the questions are so weird and surprising that isn’t face validityConcurrent validityPerformance assessment provides scores similar to established assessment instrumentsE.g., scores on a new depression scale are similar to those from the Beck Depression InventoryPredictive validity(criterion)Performance assessment provide scores that predict behavior in the futureE.g., scores on the ACT predict measures of success in collegeIn general high ACT score, high GPA – low ACT score, low GPAGPA in high school is more predictable for how well someone will do in college than the ACTConstruct validityMeasurement of a construct (concept) provides scores that can reliably identify individual differences and can successfully predict future performanceRequires converging evidence from several studiesE.g., intelligence, as measure by IQ scores, appears to differentiate individuals and appears to predict performance in a variety of arenasGive a test and then accurately be able to identify who’s extroverted and who’s introverted – or a test to accurately define high and low self-esteem, personalitySummaryGood measurement is essential in behavioral researchGood measurement requires specific operational definitions of constructs, identification on a particular scale of measurement, validity, and reliabilityMethods of Data CollectionObservation ActivitySwitching the remote between handsThe Nature of ObservationsThe importance of visual observations in behavioral researchElement of subjectivityTwo people can look at the same behavior and see two different observationsWays of observingParticipant vs. nonparticipant observationsScheduling observationsDefining the behavior to be observedOperational definitionsViolence on playground – what is your definition of violenceSpecific techniques for recording behaviorsFrequency methodHow often did something happen; how many times did she hit himDuration methodHow long did something happen; Exercise – how long did they run, could they run?Interval methodHow many times in a 15 min period in the library people walk around on the quiet floor; see if there is a difference on the quiet floor and normal floor how many times they get up and take a breakRecording more than one responseYou want to make sure your operational definition is good enough so your observations can’t be more than one thingWe attend to things that fit in to a pattern that reinforces into things we already thinkReliability of ObservationsFoundation of our measurementWould the behavior be observed the same way every time?Importance of concurrent and the independent observersNormally have more than one observing someone’s behaviorCan measure inter-observer agreementIf one person looks at violence behavior on a playground and sees like 8 acts of violence behavior and someone else observing the same behavior saw 20 acts – there is no inter-observer agreementInter-Observer AgreementHigh inter-observer agreement creates confidence that the behavior is well definedHigh inter-observer agreement makes it more likely that you will observe an effect of your independent variableMore confident on your results if you have High inter-observer agreement so you can get the same results from different peopleSteps for maintaining observer reliabilityEstablish objective criteriaPilot test procedures and assess inter-observer reliability prior to beginning the actual studyIf reliability is low, reassess definitions, criteria, and trainingLots of training when observingIf reliability is high, begin study and use periodic checksPeriodic retraining of observersIf possible, use “blind” observersMeasuring the Reliability of Observational DataPercentage agreement among observersThe reliability coefficientHow strongly are these two associatedRecordings by EquipmentCommon for physiological measuresOften increases precision for behavioral measuresNeed to calibrate equipmentSometimes video behavior and go back until they get reliable behaviorPublic RecordsExamples:Census dataCrime statisticVoting patternsNational surveysResearch questions usually focus on relationships among variablesNeed to