Research PaperNOTEWorks CitedKatz & LautenschlagerENG 401(3)June/30/2003Iri-Noda-San(IRINODA, Katsutoshi)Research PaperCan you correctly answer the reading comprehension test in SATwithout reading the passage?Reviewing the debate between Katz & Lautenschlager (The University of Georgia)and Freedle & Kostin (Educational Testing Service) The author’s attitude toward children appears to be one of (A) concern for the development of their moral integrity(B) idealization of their inexperience and vulnerability(C) contempt for their inability to accept unpleasant facts(D) exaggerated sympathy for their problems in daily life(E) envy of their willingness to learn about morality(College Board, 1983) This is one of the test items in the past SAT. It comes with a particular readingpassage and the test-takers are asked to choose the best answer. In the example above,however, the whole reading passage is omitted. Nonetheless, isn’t it true that you arelikely to get the answer correctly just reading the stem and choices, is it? Which willyou choose? In an experiment (Karz, et al., 1990), without the passage, 83% of examinees choose1Katz & Lautenschlager(A), while no more than 7% choose any of the incorrect choices. The correct choice is(A). I could choose (A) myself and I am very sure that you have chosen the same one.How could I do that? My answer is, the others are simply unlikely (1). In the first paper for this course, I have reviewed major journals in languagetesting. With the growth of the field for these decades, there have appeared a hugenumber of issues there. As a case study, in this paper I am going to deal with the debatein the 1990s between Stuart Katz and Gary J. Lautenschlager of University of Georgiaand Roy Freedle and Irene Kostin of Educational Testing Service (ETS). The debateitself centers around the validity of reading comprehension tests, and it starts when theformer scholars criticized the SAT on the basis of their findings that many of thequestions could be answered without reading the passages at all. How it can be true ofcourse interests us, so it would be worthwhile to introduce this debate. If we can answer the questions without reading the passage, then what is thevalue of the omitted reading passage itself? Is the SAT a valid test for measuringreading comprehension? This is the very question that Karz & Lautenschlager havebeen studying over a decade. Katz, Lautenschlager, Blackburn, & Harris (1990) raise the issue in the researcharticle “Answering reading comprehension items without passages on the SAT” inPsychological Science. They conduct the two experiments in this study. In the firstexperiment, the examinees in the control condition (P Group) score 69.6 on the average(out of 100), whereas the other two groups who take the test without the readingpassage score 45.8 (NP/C Group; No Passage/Coaching) and 46.6 (NP Group; No2Katz & LautenschlagerPassage) respectively. Since the RC question has five choices, the chance level of thescore should be one-fifth of the full score, which is 20 in this case, or less; for lowlevels of ability, performance on well-constructed items may fall below chance, becausethe examinee succumbs to distractors (Donlon, 1984; Lord, 1974, 1980. Italics in theoriginal). Pearson product-moment correlations between their scores in the experimentand their original SAT Verbal scores are also high; r(16) = .68 (p < .005), r(15) = .56 (p< .025), and r(27) = .51 (p < .005), respectively (2). The item analysis reveals that 69and 72 of the 100 test items in the NP/C and NP conditions, respectively, have p values(proportion correct) exceeding .3, while nearly half have p values exceeding .5. Better-than-chance performance of individual items, therefore, is a frequent occurrence whenthe passages are missing. In the second experiment, where they employ a larger number of participantswho are much closer to the average of the normative samples in their SAT-Verbalscores, they obtain a similar result. The examinees in the control condition score 56.8(out of 100), while those in the NP Group score 37.6. Pearson product-momentcorrelations between their scores in the experiment and their original SAT Verbal scoresare also high; r(51) = .88 (p < .001), r(73) = .72 (p < .001), respectively. The itemanalysis reveals that 61 of the 100 test items in the NP condition have p values(proportion correct) exceeding .3 (30%), and nearly a third have p values exceeding .5(50%). To sum up, these findings indicate that though the examinees who have accessto the reading passage naturally score higher than those who have no access, the lattersstill get about twice as high scores as a mere chance (20 out of 100). All of theexaminees show strong correlations between the scores when they take the test with the3Katz & Lautenschlagerreading passage and those without. The majority of the items are answered correctlywithout the passage. Karz et al. conclude that “performance on the RC task, therefore,would appear to depend substantially on factors having nothing to do withunderstanding the prose passages normally accompanying test questions (pp. 125-126)”. This is a serious problem because “ETS instructs examinees to answer questionson the basis of passage content, not personal opinion or knowledge. Despite theseinstructions, our findings show that proscribed information can often be used toadvantage, and prescribed information often ignored without disadvantage (p. 126)”. Katz, Blackburn, and Lautenschlager (1991) is designed to supplement thefindings above. In case that examinees used information from other items (“cognates”)belonging to the same passage, they make the comparison between the NP conditionand the Quasi-randamized (NP/QR) condition, where the test items are presented in aquasi-randamized order to prevent the use of overlapping information among cognates;no more than two items from the same passage appear on the same exam battery. Theadjusted means for the NP and NP/QR conditions, respectively, are 35.7 and 34.1.Pearson product-moment correlations between their original SAT-Verbal scores and thescores in this experiment are high; r(62) = .64, p < .001 for the NP