KIN 4310 1nd Edition Lecture 7 Outline of Last Lecture I Workshop II Excel Functions III Objectives Measurement Theory IV Measurement V Terms Measurement VI Measurement Challenges VII Variables of Interest VIII Classical Test Theory O T E Outline of Current Lecture I Terms True Score II Terms Observed Score III Terms Error IV Levels of Measurement V Nominal Scales VI Ordinal Scales VII Interval Scales VIII Ratio Scales IX Levels of Measurement These notes represent a detailed interpretation of the professor s lecture GradeBuddy is best used as a supplement to your own notes not as a substitute X Objectives Reliability XI Reliability XII Reliability XIII Reliability XIV Reliability XV Test Retest Reliability XVI Interrater Reliability XVII Internal Consistency Reliability XVIII Parallel Forms Consistency XIX What is Reliable XX Reliability Example XXI Reliability Example cont XXII Improving Reliability Current Lecture CORRECTION variability will be NOT be on Exam 1 I Terms True Score a True score the actual amount of the attribute you want to measure e g true self esteem true dietary intake etc b Assumption the construct is real and exists much like a blood level or atomic weight if only we could measure it accurately II Terms Observed Score a Observed score is what results from your measurement b We want our observed score to be as close as possible to the true score but there is often some discrepancy between what we observe and what is actually there III Terms Error a Error anything you did not intend to measure that affected score b Systematic error recurs on repeated measurements and affects scores predictability i ex scale putting clock ahead c Nonsystematic error is unpredictable and varies with every measurement i Hard to deal with but can be done ii Can deal with this by doing repeated measurement IV Levels of Measurement a Nominal i Worst out of the all b Ordinal c Interval d Ratio i Can use all statistical terms but ratio is the best one V Nominal Scales a Mutually exclusive unordered categories b For example i Gender ii Political affiliation iii Country of origin iv Movie genre VI Ordinal Scales a Characteristics that can be put in order but there is no consistent difference between adjacent scores b For example i Rank 3rd place ii Olympic Medal Color gold silver bronze iii Movie Ratings VII Interval Scales a Variables refer to quantities of units on a continuum b Distance between variables is meaningful c For example i Temperature in Fahrenheit or Celsius ii Dates e g year iii Intelligence Quotient IQ d Generally well use units e You can get into the negative interval scale like before year 0 for example f There s no absolute zero g Can only subtract here VIII IX Ratio Scales a Variables refer to quantities of units on a continuum b Has an absolute zero c For example i Annual salary ii Distance traveled in the 12 minute run test iii Body fat percentage iv Stature d Can be ratio s like he makes twice as much money as me e Can only divide here Level of Measurement a X Objectives Reliability and Validity a Define reliability b Define validity and identify different types purposes c Apply these concepts to the development of tests and surveys d Will wait to discuss validity in section two XI Reliability a The degree to which scores are i Free from errors of measurement ii Consistent or stable across a variety of conditions XII Reliability a Types of reliability i Test retest reliability ii Interrater reliability iii Internal consistency reliability iv Parallel forms reliability b Refer to book chapter 6 for this to get more in depth XIII Reliability a Measures of reliability are correlations b Correlations represent consistency over time across different raters across different raters across different assessment forms or consistency of items within the measure c This is how we tie r linear correlation coefficient to reliability XIV Reliability a There are 4 types that we ll go over b You can look at the line y x XV Test Retest Reliability a Correlation of scores on a test given at two separate times b Longer times require greater stability c Affected by Change carry over effects i Carry over effects are like when you take a test for the first time and then again a second time and do better the second time just because you re more familiar with the test structure d Alternate Forms or Split Half Reliability can address some of the problems i Don t worry about this XVI Interrater Reliability a Correlation of scores measure by two different observers or raters b Example i Reading an ambiguous scale ii Subjective assignment of quantitative scores judging gymnastics diving figure skating because there is subjective error iii Operating a stopwatch XVII Internal Consistency Reliability a When a test consists of multiple items do all items assess the same dimension b A function of the relationship between items on a scale and the number of items c Cronbach s Alpha i Won t test over this It s a number like r d Ex Presidents Test in Public School i Sit ups ii Pushups iii Flexibility iv Mile run v Etc vi Its supposed to measure fitness e Ex field sobriety test XVIII Parallel Forms Reliability a Correlation of scores between two different versions of a test b Example i IQ Tests ii Exam 3A versus Exam 3B c Here it shouldn t matter what version of the test you get there should be the same level of difficulty XIX What is Reliable a Reliability coefficients should be positive i 0 0 to 1 0 b General rules of thumb i Test retest 0 60 r equalto 1 00 ii Interrater 85 agreement iii Parallel forms 0 70 r equalto 1 00 iv Don t have to remember these numbers c High reliability DOES NOT alone constitute quality or validity i Will talk about this later XX Reliability Example a Tinettie Balance Assessment Tool i PT s use this b Two PT s assess the same group of patients i r 0 98 ii Very good interrrater reliability c One PT assesses each patients twice in 48 hours i r 0 98 ii Very good test retest reliability XXI Reliability Example cont a Self Efficacy Assessment b 20 students write two different self efficacy tests The correlation coefficient turns out to be r 0 32 c Poor parallel test reliability d This result is not statistically significant according to a two tailed test with significance of 0 05 XXII Improving Reliability a Make sure instructions are standardized across all settings i Controlled environment b Increase number of items or observations c Delete unclear items d Moderate easiness