1EDMS 623 Applied Measurement: Issues and Practices Spring 2006 Room 0206 Benjamin Instructor: Amy B. Hendrickson, Ph.D. Office: 1230B Benjamin Phone: (301) 405-3627 E-mail: [email protected] Office Hours: By appointment Course Objective This course will provide an introduction to both classical and modern techniques that have been developed to measure inherently unobservable (latent) traits such as ability, preference, and attitude. Students will learn to apply test theory to construct, evaluate and modify tests and measures used in education and the social sciences. Both classical and modern measurement literature will be discussed. The course is designed for students who will be pursuing careers that involve psychometrics, measurement theory or assessment. Course Materials Required Texts: Crocker, L. & Algina, J. (1986). Introduction to classical and modern test theory. Fort Worth: Holt, Rinehart and Winston Joint Committee on Standards for AERA, APA, and NCME. (1999). Standards for educational and psychological testing. Washington, DC: AERA. Class notes are available as a Course Pack at the University Book Store Required and recommended supplemental articles and chapters are listed later in the syllabus. Students are responsible for obtaining these articles, but two copies of each article will be available at the ETS library. There will be NO CLASS on April 10th due to the AERA/NCME conference2Readings are given below each topic. Students should read assignments prior to lectures. Coverage dates are not given due to the fact that topic presentation time may vary. However, topics will be covered in the order given below. Topics and Readings Section 1: Introduction: Scaling, Testing and Measurement Readings: Text 1-44; Standards 1-6 Section 2: Scaling Readings: Text 45-65; Green (1954a and 1954b); Lord (1953); Thurstone (1927) Section 6: Test and Questionnaire Construction Student presentation on standards regarding test construction Student presentation on standards regarding educational testing Readings: Text 66-84; 311-336; Standards 37-48, 137-150; Millman & Greene (1989); Spector (1992; pp. 18-46) Recommended: Likert (1932); Thurstone (1928) Section 3: Classical Test Theory Student presentation on standards regarding test user responsibility Readings: Text 87-128, 131-154, 192-212; Standards 111-118; Dudek (1979); Feldt and Brennan (1989); Harvill (1991) Section 4: Reliability via Generalizability Theory Student presentation on standards regarding reliability Readings: Text 157-189; Standards 25-36; Brennan (1992), Lane et al. (1996) Section 5: Validity Student presentation on standards regarding validity Readings: Text 217-239; Text 287-306; Standards 9-24;Campbell and Fiske (1959); Crehan (2001), Messick (1989), Reckase (1998) Recommended: Cizek (2001) Section 7: Setting Standards and Cutoffs Student presentation on standards regarding testing individuals with diverse language backgrounds and with disabilities Readings: Text 197-212, 410-429; Standards 91-108, Cizek (1996); Reckase (2000), Cizek, Bunch, and Koons (2004) Section 8: Introduction to Item Response Theory Readings: Readings: Text 339-371; Hambleton, Swaminathan, and Rogers (1991; Chapters 6 &7); Hambleton and Jones (1993); Harris (1989); Mills (1999) Section 9: Test Equating Student presentation on standards regarding norms and comparability Readings: Text 431-479; Standards 49-66; Brennan and Kolen (1987); Kolen (1988); Section 10: Bias and Differential Item Functioning (DIF) Student presentation on standards regarding test fairness Readings: Text 376-392; Standards 73-84; Clauser and Mazor (1998); Hambleton, Swaminathan, and Rogers (1991; Chapter 8); Skaggs & Lissitz (1992)3Supplemental Readings Brennan, R. L. (1992). Generalizability theory. Educational Measurement: Issues and Practice, 11(4), 27-34. Brennan, R. L., & Kolen, M. J. (1987). Some practical issues in equating. Applied Psychological Measurement, 11, 279-290. Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105. Cizek, G. J. (1996). Standard-setting guidelines. Educational Measurement: Issues and Practice, 15(1), 13-21. Cizek, G. J. (2001). More unintended consequences of high-stakes testing. Educational Measurement: Issues and Practice, 20(4), 19-27. Cizek, G. J., Bunch, M. B. & Koons, H. (2004). Setting performance standards: Contemporary Methods. Educational Measurement: Issues and Practice, 23(4), 31-50. Clauser, B. E., & Mazor, K. M. (1998). Using statistical procedures to identify differentially functioning test items. Educational Measurement: Issues and Practice, 17(1), 31-44. Crehan, K. D. (2001). An investigation of the validity of scores on locally developed performance measures in a school assessment program. Educational and Psychological Measurement, 61, 841-848. Dudek, F.J. (1979). The continuing misinterpretation of the standard error of measurement. Psychological Bulletin, 86, 335-337. Feldt, L. S., & Brennan, R. L. (1989). Reliability (Chapter 3, 105-111). In R. L. Linn, Educational Measurement, 3rd edition, New York: Macmillan. Green, B. (1954a). Paired comparison scaling procedures. Handbook of Social Psychology, 1, 344-347. Green, B. (1954b). The method of successive intervals. Handbook of Social Psychology, 1, 347- 350. Hambleton, R.K., Swaminathan, H. & Rogers, H. J. (1991). Fundamentals of item response theory (Chapters 6, 7, & 8). Newbury Park: Sage. Hambleton, R. K., & Jones, R. W. (1993). Comparison of classical test theory an ditem response theory and their applications to test development. Educational Measurement: Issues and Practice, 12(3), 253-262. Harris, D. (1989). Comparison of 1-, 2-, and 3-Parameter IRT Models. Educational Measurement: Issues and Practice, 8(1), 157-163. Harvill, L.M. (1991). Standard error of measurement. Educational Measurement: Issues and Practice, 10(4) 181-189.4Kolen, M. J. (1988). Traditional equating methodology. Educational Measurement, Issues and Practice, 115-122. Lane, S., Liu, M., Ankenmann, R. D. , & Stone, C. A. (1996). Generalizability and Validity of a Mathematics Performance Assessment, Journal of Educational Measurement, 33, 71-92. Likert, R. (1932). A technique for the measurement of attitudes. Archives of Psychology. No. 140, 5-53. Lord, F. (1953). On the statistical treatment of football numbers.