22S:138 Bayesian Statistics-Term ProjectBayesian IRT Analysis of a Standardized TestGiven under Nonstandard Conditions Bayesian Statistics Project Final ReportChen, Qi, and Chris December 4, 2009I. IntroductionIn the wake of the No Child Left Behind (NCLB) legislation, passed in the administrationof President G. W. Bush, the concept of “accountability” of schools has risen inimportance. Standardized tests have become pervasive across the United States. Thesetests are high-stakes; sanctions for individual schools and school districts are included inthe legislation. While the driving force behind the legislation is improvement of studentachievement, it is not clear by what mechanism this would occur. It seems to be apolitical and public “given” that the required standardized tests will somehow catalyzeimprovement. The classroom teacher, placed securely at the muzzle end of the delivery of instruction, isperhaps least equipped to respond intelligently to the call for improved achievement, readby many classroom teachers as a veiled call for improved “instruction.” The classroomteacher lives in a world of tests created by themselves, or possibly their departments. Anexam created by someone else, especially in a multiple-choice format, puts teachers in anunenviable position. The high stakes tests are given (usually) once a year, the items aresecure, and the summary reports generated by the testing authority are for the most partincomprehensible, due to teachers’ minimal understanding of descriptive statistics such aspercentiles and standard scores. (Matthews, et al. 2007) Even if such reports wereunderstood, it is not clear how teachers should respond. Using the Iowa Tests of BasicSkills as an example, an elementary teacher may see a low summary score for“Punctuation.” But armed with only that information, how is a teacher to respond? Aretheir students comma happy? Do they not understand the requirements for endingpunctuation in sentences? Without specific knowledge, specific targeting strategiescannot be mounted in the classroom.One possible strategy for teachers is to give “released” exams to their students. Not onlywill students be more aware of what might typically be expected of them in terms ofitems, teachers could use the information from the test as a whole and individual items asfeedback in a systematic formative assessment strategy. Well-designed multiple-choicetests will carry diagnostic information in their distracters; not only could teachers see whattheir students missed, but they might be able to identify individual and/or groupmisunderstandings.122S:138 Bayesian Statistics-Term ProjectA possible flaw in such a strategy is that teachers may not be able to give the tests underthe standardized conditions, most probably due to time requirements. For our project we analyzed the test results of students who took a released standardizedtest as preparation for the “real” test. These tests were given for the express purpose ofidentifying problem areas for concentration of review before the “real” test. For thisstrategy to be viable, it is important that teachers are able to “trust” the results of theclassroom administered test. The validity and reliability of the results rest in the statisticalqualities of the individual items, and we undertook to analyze test items under an ItemResponse Theory (IRT) framework. Of particular interest are the existence of acceptableitem parameters.The 2007 Advanced Placement Statistics Exam (College Board, 2008) was released forteacher use in 2008. One of us (CO) offers as a service an analysis of test results for APStatistics teachers who give the exam. In 2008-9, 750+ student results were analyzed andreports sent to 12-15 teachers. We used these data, with all identifying informationstripped, in our analysis. II. General background of Item Response TheoryIn the early 20th century the intelligence testing movement began with the work of Binetand Simon (1916) in Paris, and continued in the United States at Stanford University(Terman, 1916; Terman and Merrill, 1937). The “test score” was the basic unit ofmeasurement and the theory of test development subsequently developed is known as the“classical” test theory. Both theoreticians and practitioners became dissatisfied with thetest score as a basic unit as conceptions of intelligence (and other traits) became morecomplex. Attention has shifted from whole test scores to individual items, anddevelopment of the theory and methods of Item Response Theory (Lord & Novick, 1968).IRT provides a theoretical framework for evaluating whole assessments (“tests”) as wellas individual questions (“items.”) IRT is used not only to develop and refine tests andmaintain item banks, but also to “link,” or equate exams so that results across tests can beinterpreted. As an example, a score of 700 on the Graduate Record Exam will indicate thesame achievement level of a student, even if the tests were taken in different years, anddid not contain the same items.IRT uses mathematical models to analyze the outcomes of standardized testing, based onthe idea that the probability of a student’s correct response to an individual test item is afunction of both person and item parameters. The statistical performance of an item isdescribed by its item characteristic curve (ICC). The ICC models the probability that aperson with a given (or estimated) ability level will answer the item correctly. Personswith lower ability are assumed to have less of a chance of answering an item correctlythan persons with high ability. The person and the item may be said to interact; therefore,a given IRT model describes the probability of a correct response to the item as a function222S:138 Bayesian Statistics-Term Projectof an individual’s achievement (“ability”) which probability depends on one or more itemparameters which model the particular testing situation. We analyzed our data using boththe 2- and 3-parameter logistic models, and compared our results with estimates ofparameters for the 2PL model.The 2PL ModelThe 2PL model contains parameters for the difficulty and discrimination of an
View Full Document
Unlocking...