Criterion-related Validity• About asking if a test “is valid”• Criterion related validity types– Predictive, concurrent, & postdictive– Incremental, local, & experimental• When to use criterion-related validity• Conducting a criterion-related validity study– Properly (but unlikely)– Substituting concurrent for predictive validity– Using and validating an instrument simultaneously• Range restriction and its effects on validity coefficients• The importance of using a “gold standard” criterion“Is the test valid?”Jum Nunnally (one of the founders of modern psychometrics) claimed this was “silly question”! The point wasn’t that tests shouldn’t be “valid” but that a test’s validity must be assessed relative to…• the construct it is intended to measure • the population for which it is intended (e.g., age, level)• the application for which it is intended (e.g., for classifying folks into categories vs. assigning themquantitative values)So, the real question is, “Is this test a valid measure of this construct for this population in this application?” That question can be answered!Criterion-related Validity - 5 kinds• does test correlate with “criterion”? -- has three major types• predictive -- test taken now predicts criterion assessed later• most common type of criterion-related validity• e.g., your GRE score (taken now) predicts how well you will do in grad school (criterion -- can’t be assessed until later)• concurrent -- test “replaces” another assessment (now)• often the goal is to substitute a “shorter” or “cheaper” test• e.g., the written drivers test is a replacement for driving around with an observer until you show you know the rules• postdictive -- least common type of criterion-related validity• can I test you now and get a valid score for something that happened “earlier” -- e.g., adult memories of childhood feelings•incremental, local, & experimental validity will be discussed belowThe advantage of criterion-related validity is that it is a relatively simple statistically basedtype of validity!• If the test has the desired correlation with the criterion, then you have sufficient evidence for criterion-related validity.There are, however, some limitations to criterion-related validity…• It is dependent upon your having a criterion•Sometimes you don’t have a criterion variable to use --e.g., first test of construct that is developed• It is dependent upon the quality of the criterion variable•Sometimes there are “limited” or “competing” criteria• “Correlation is not equivalence”•your test that is correlated with the criterion might also becorrelated with several other variables -- what does it “measure” ? Conducting a Predictive Validity Studyexample -- test designed to identify qualified “front desk personnel” for a major hotel chain -- 200 applicants - and 20 position openingsConducting the “proper study”• give each applicant the test (and “seal” the results)• give each applicants a job working at a front desk• assess work performance after 6 months (the criterion)• correlate the test (predictor) and work performance (criterion)Anybody see why the chain might not be willing to apply this design?Here are two designs often substituted for this proper design.Substituting concurrent validity for predictive validity• assess work performance of all folks currently doing the job• give them each the test• correlate the test (predictor) and work performance (criterion)Problems?• Not working with the population of interest (applicants)• Range restriction -- work performance and test score variability are “restricted” by this approach• current hiring practice probably not “random”• good workers “move up” -- poor ones “move out”• Range restriction will artificially lower the validity coefficient (r)Applicant pool -- target populationSelected (hired) folks • assuming selection basis is somewhat reasonable/functionalSample used in concurrent validity study • worst of those hired have been “released”• best of those hired have “changed jobs”What happens to the sample ...Criterion -job performancePredictor -- interview/measureWhat happens to the validity coefficient -- rApplicant poolr = .75Sample used in validity studyr = .20Hired FolksUsing and testing predictive validity simultaneously• give each applicant the test• give those applicants who “score well” a front desk job• assess work performance after 6 months (the criterion)• correlate the test (predictor) and work performance (criterion)Problems?• Not working with the population of interest (all applicants)• Range restriction -- work performance and test score variability are “restricted” by this approach• only hired good those with “better” scores on the test• (probably) hired those with “better” work performance• Range restriction will artificially lower the validity coefficient (r) • Using a test before its validated can have “legal ramifications”Other kinds of criterion-related validity…Incremental ValidityAsks if the test “improves on” the criterion-related validity of whatever tests are currently being used.Example. I claim that scores from my new structured interview will lead to more accurate selection of graduate students. I’m not suggesting you stop using what you are using, but rather that you ADD my interview.Demonstrating Incremental Validity requires we show that the “new test + old tests” do better than “old tests alone”. R²Δ test…R²grad. grea, grev, greq= .45R²grad. Grea, grev, greq, interview= .62Incremental validity is .17 (or 38% increase)Local ValidityExplicit check on validity of the test for your population and application. Sounds good, but likely to have the following problems• Sample size will be small (limited to your “subject pool”)• Study will likely be run by “semi-pros”• Optimal designs probably won’t be used (e.g., predictive validity)• Often (not always) this is an attempt to “bend” the use of an established test to a population/application for which it was not designed nor previously validated Experimental ValidityA study designed to show that the test “reacts” as it should to a specific treatment.In the “usual experiment,” we have confidence that the DV measures the construct in which we are interested,