DOC PREVIEW
irt07abim_lecture08

This preview shows page 1-2-3-4-28-29-30-31-57-58-59-60 out of 60 pages.

Save
View full document
View full document
Premium Document
Do you want full access? Go Premium and unlock all 60 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 60 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 60 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 60 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 60 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 60 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 60 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 60 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 60 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 60 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 60 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 60 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 60 pages.
Access to all documents
Download any document
Ad free experience

Unformatted text preview:

Test Reliability & Dl tUiIRTDevelopment Using IRTAmerican Board of Internal MedicineItem Response Theory CourseOverviewOverview•Reliability with IRT•Reliability with IRT–Item and Test Information Functions•ConceptsConcepts• EquationsUdEl•Uses and Examples•Optimal Test DesignpgReliability with IRTReliability with IRT•We all know that reliability (precision) is aWe all know that reliability (precision) is a desirable property for an assessment.•The more reliable a test is the more•The more reliable a test is, the more precisely we can measure the construct.F li d (IRT CTT)•For any scaling procedure (IRT or CTT), as reliability goes up, the standard error of tdmeasurement goes down.Reliability with IRTReliability with IRT•In CTT, reliability is a one-number summary ofIn CTT, reliability is a onenumber summary of test precision, and there is a corresponding single standard error of measurement that is used for any test score.• In IRT, test precision is conceptualized as something called Information, which is conditional on the trait level being measured.Stt ld tititll–Some tests could measure certain trait levels very well but measure others poorly…Reliability with IRTReliability with IRT•A further advantage of IRT with respect toA further advantage of IRT with respect to evaluating reliability is that we can consider the amount ofInformationanconsider the amount of Informationan item and/or a test provides. •In CTT measures of item quality exist but•In CTT, measures of item quality exist, but these are only indirectly related to what the reliability of the test will bethe reliability of the test will be.Item Information FunctionItem Information Function•“Item Information”indicates an item’s•Item Information indicates an item s usefulness for assessing ability.• By “usefulness” we basically mean how good an item is at distinguishing gggexaminees with lower ability levels from those with higher ability levelsfrom those with higher ability levels.• Information Æ Precision.0.81.0040.6(u = 1 | θ)0.20.4P (0.0-3-2-10123Ability (θ)0.81.00.6Info(θ)0.20.4I0.0-3-2-10123Ability (θ)Item Information FunctionItem Information Function•Items are basically more informativeItems are basically more informative where the slope of the ICC is steepest which happens whensteepest, which happens when…bjis relatively close to θi,ais relatively high andajis relatively high, andcjis relatively lowIf0i idi•If cj= 0, an item provides its maximum information when θi= bj1.0a=100.8a = 1.0c=0.00.6 1 | θ)c 0.0b = 1.0 or 2.00.4P (u =0.20.0-3 -2 -1 0 1 2 3Ability (θ)Ability (θ)1.0a=100.8a = 1.0c=0.00.6o(θ)c 0.0b = 1.0 or 2.00.4Info0.20.0-3 -2 -1 0 1 2 3Ability (θ)Ability (θ)1.00.8b=-100.6 1 | θ)b 1.0c = 0.20.4P (u =a = 1.0 or 0.50.20.0-3 -2 -1 0 1 2 3Ability (θ)Ability (θ)1.00.8b=-100.6o(θ)b 1.0c = 0.20.4Infoa = 1.0 or 0.50.20.0-3 -2 -1 0 1 2 3Ability (θ)Ability (θ)1.0a=100.8a = 1.0b=0.00.6 1 | θ)b 0.0c = 0.0 or 0.20.4P (u =0.20.0-3 -2 -1 0 1 2 3Ability (θ)Ability (θ)1.0a=100.8a = 1.0b=0.00.6o(θ)b 0.0c = 0.0 or 0.20.4Info0.20.0-3 -2 -1 0 1 2 3Ability (θ)Ability (θ)Item Information FunctionItem Information Function•IMPORTANT: information is afunctionofIMPORTANT: information is a function of θ, which means that an item could be very informative for some ability levels andinformative for some ability levels and relatively uninformative for others.•Example: difficult items are informative for•Example: difficult items are informative for higher ability levels, but don’t tell us much about lower ability levels (because theyabout lower ability levels (because they mostly get all those items wrong!).1.0c=0150.8c = 0.15a=1.2 or 0.80.6 1 | θ)a 1.2 or 0.8b = 1.0 or 0.00.4P (u =0.20.0-3 -2 -1 0 1 2 3Ability (θ)Ability (θ)1.0c=0150.8c = 0.15a=1.2 or 0.80.6o(θ)a 1.2 or 0.8b = 1.0 or 0.00.4Info0.20.0-3 -2 -1 0 1 2 3Ability (θ)Ability (θ)Item Information FunctionItem Information Functionfor the 3-PL'2[()]jPθ[()]()() ()jjjjIPQθθθ=22(1 )jjjjDa c−() ()2[][1]jj jjjjDa b Da bjce eθθ−−−=++jNotes on IIFNotes on IIF•The roles ofaandcare easy to see•The roles of ajand cjare easy to see.–as ajincreases, information increases–as cjincreases, information decreases•As ability moves away frombj(+or-)As ability moves away from bj( or ) the denominator increases, so information approaches zeroinformation approaches zero.Maximum InformationMaximum Information1⎡⎤max1ln 0.5(1 1 8 )jjjbcDaθ⎡⎤=+ ++⎣⎦If cj = 0, then Information is maximized at bjjjjIf cj > 0, then Information is maximized at an ability level slightly greater thanban ability level slightly greater than bjTest Information FunctionTest Information Function•Just like we add up ICCs to get a•Just like we add up ICCs to get a TCC, we add up IIFs to get a TIF.• Information will continue to increase as we add test items, therefore increasing precision.•All things equal longer tests provide•All things equal, longer tests provide increased measurement precision.Test Information FunctionTest Information Function•Defined for a set of items at each•Defined for a set of items at each point along the ability (θ) scale.• Test information is influenced by the ‘quality’ and the number of test items:qy() ()nIIθθ∑1() ()jjIIθθ==∑1j=1.00.80.6 1 | θ)0.4P (u =0.20.0-3 -2 -1 0 1 2 3Ability (θ)Ability (θ)86745| θ)34E(X 120-3-2-10123Ability (θ)Ability (θ)1.00.80.6o(θ)0.4Info0.20.0-3 -2 -1 0 1 2 3Ability (θ)Ability (θ)432o(θ)2Info10-3-2-10123Ability (θ)Ability (θ)432o(θ)2Info10-3-2-10123Ability (θ)Ability (θ)Conditional Error forConditional Error for Maximum Likelihood Estimates• One of the great benefits of IRT li i th tIRT scaling is that measurement precision andmeasurement precision and error can now be considered conditional on θ.Conditional Error forConditional Error for Maximum Likelihood Estimates• Standard error of an MLE is determined by:determined by:1ˆ()θ1()ˆSEθ=()IθConditional Standard ErrorConditional Standard Error•Theimprecisionof ability estimationThe imprecisionof ability estimation is therefore inversely related to the amount ofInformationwith respectamount of Informationwith respect to ability that is available.•Since Information increases with the•Since Information increases with the quality and number of items, the SE conversely decreases whichconversely decreases…which hopefully makes some


irt07abim_lecture08

Download irt07abim_lecture08
Our administrator received your request to download this document. We will send you the file to your email shortly.
Loading Unlocking...
Login

Join to view irt07abim_lecture08 and access 3M+ class-specific study document.

or
We will never post anything without your permission.
Don't have an account?
Sign Up

Join to view irt07abim_lecture08 2 2 and access 3M+ class-specific study document.

or

By creating an account you agree to our Privacy Policy and Terms Of Use

Already a member?