UNC-Chapel Hill PSYC 840 - Item Response Theory Score Approximation Software

Unformatted text preview:

L.L. Thurstone Psychometric LaboratoryElectronic Research Memorandum2002-1User’s Guide for IRTSCORE:Item Response Theory Score Approximation SoftwareDavid FloraDavid Thissen1User’s Guide for IRTSCORE:Item Response Theory Score Approximation SoftwareDavid Flora & David ThissenL.L. Thurstone Psychometric LaboratoryThe University of North Carolina at Chapel HillIRTSCORE is a console-based program for scoring tests comprising graded-type items (e.g., Likert-type items) with item response theory (IRT). IRTSCOREalso has a Monte Carlo capability that, given a set of IRT parameters,provides simulation-based estimates of reliability and mean squared errorfor the various types of scores it produces. In addition to providing IRT-scalescores based on summed scores (i.e, EAP|SS), IRTSCORE implements analgorithm for calculating a linear approximation for IRT-based response-pattern scale scores. The approximation is essentially a weighted sum of theIRT-calculated EAPs for each individual item response. This scoringalgorithm has been described by Steinberg and Thissen (2001) and Thissen,Flora, Reeve, & Vevea (2000). Thissen et al. (2000) showed that scorescalculated with the IRTSCORE algorithm (i.e., LinEAP) tend to be more reliableand have lower mean squared error (MSE) than EAP|SS scores.Preparation for Running IRTSCOREIRTSCORE scores tests comprising graded-type items or two-parameterlogistic (2PL) items. All items in the test must have the same number ofresponse categories. For a given test to be scored and analyzed usingIRTSCORE, the items must first be calibrated using software external toIRTSCORE, such as MULTILOG (Thissen, 1991). This calibration produces a fileof item parameters that will serve as input for IRTSCORE. If the items arecalibrated with MULTILOG , the >SAVE command will create a file with theextension .SAV that can be used as input for IRTSCORE. IRTSCORE also acceptsitem parameter files in tab- or space-delimited column format, where thefirst column contains the items’ a parameters and subsequent columnscontain the relevant number of b parameters for each item. The itemparameter file should be in the same directory as IRTSCORE.exe.Preparation for Running IRTSCORE...................................................1Running IRTSCORE......................3Major Functions of IRTSCORE......9Keyboard Entry.....................9Simulee-Simulation............10Results...............................122The particular test serving as an illustrative example in this guide is theNorth Carolina End-of-Grade 3 Social Studies exam from 1993, as describedby Thissen, Pommerich, Billeaud, and Williams (1995) in their article on IRTfor tests including items with ordinal responses. Because all demonstrationsin this guide are based on this test, the Thissen et al. (1995) article is auseful co-reference for this guide. This social studies test consists of open-ended items that were rated on a 4-point scale, thus generating ordinal itemresponses. Any test comprising items with ordinal responses that arecalibrated using Samejima’s (1969) graded IRT model are appropriate forIRTSCORE. Thus, both judged ratings, such as those from this social studiestest, and Likert-type item responses, such as may be found on apsychological questionnaire, generate appropriate item response data foruse with IRT score.The actual values given to observations on an ordinal scale are arbitrary,so long as the proper ordering is preserved. In educational and othersettings, it is conventional to assign a value of zero to observations falling inthe lowest possible category of an ordinal scale. On the other hand,researchers using Likert-type scales typically assign a value of one, ratherthan zero, to observations falling in the lowest possible category.IRTSCORE assumes that observations falling in the lowest possible havebeen assigned the value zero, observations in the next-lowest category areassigned the value one, the next category is associated with the value two,and so on. Given this convention, the range of summed scores is from zeroto the summed numbers of response categories for the items, each less one.That is, the summed scores for 10 binary items range from 0 to 10; or, thesummed scores for 6 four-alternative Likert-type items range from 0 to 18.ReferencesSamejima, F. (1969). Estimation oflatent ability using a response patternof graded scores. PsychometricMonograph, No. 17, 34, Part 2.Steinberg, L., & Thissen, D. (2001).Alternative methods to computescores based on item response theory:Illustrations with measures of adultattachment.Thissen, D. (1991). MULTILOG user’sguide. Chicago, IL: Scientific Software.Thissen, D., Flora, D., Reeve, B., & Vevea,J. L. (2000, July). LinearApproximations for Item ResponseTheory Response Pattern Scores. Paperpresented at the annual meeting ofthe Psychometric Society, Vancouver,BC, Canada.Thissen, D., Pommerich, M., Billeaud, K.,& Williams, V. S. L. (1995). Itemresponse theory for scores on testsincluding polytomous items withordered responses. AppliedPsychological Measurement, 19, 39-49.3Running IRTSCOREStart IRTSCORE.exe by double-clicking on the application. A console windowappears:IRTScore: Score Approximation with Item Response TheoryEnter filename for output [CR for screen output]: <return>Here, the user has the option of specifying whether the output generated byIRTSCORE is to be saved in a file (which the user names at this prompt), orsimply written to the console window (which happens when the user entersa carriage return, “<return>” without a filename).In this guide, illustrative userresponses are in bold.The console windows in thisguide are from Mac OS X;windows have different cosmeticfeatures on other platforms.4Next, IRTSCORE asks for information about the external file containing theitem parameters.IRTScore: Score Approximation with Item Response TheoryEnter input file format (1=Multilog, 2=Columns) [1]: <return>Enter input file name: test.sav<return>Here, the user has specified that the item parameter file is an output filefrom MULTILOG by simply entering a carriage return for the first prompt, thusaccepting the default option, “1=Multilog.” Next, the user has specified thatthe name of the item parameter file is test.sav. 0001 2 4 1 2 3 4 0.187E+01 0.649E+00 0.197E+01 0.314E+01 0001 2 4 5 6 7 8 0.266E+01 0.122E+00 0.157E+01 0.269E+01 0001 2 4 9 10 11 12 0.124E+01 0.778E-01 0.203E+01 0.430E+01GRP10004 2 21998


View Full Document

UNC-Chapel Hill PSYC 840 - Item Response Theory Score Approximation Software

Download Item Response Theory Score Approximation Software
Our administrator received your request to download this document. We will send you the file to your email shortly.
Loading Unlocking...
Login

Join to view Item Response Theory Score Approximation Software and access 3M+ class-specific study document.

or
We will never post anything without your permission.
Don't have an account?
Sign Up

Join to view Item Response Theory Score Approximation Software 2 2 and access 3M+ class-specific study document.

or

By creating an account you agree to our Privacy Policy and Terms Of Use

Already a member?