Demonstration of a Reading Coach that ListensJack Mostow, Alexander G. Hauptmann, and Steven F. RothProject LISTEN, 215 Cyert Hall, 4910 Forbes AvenueCarnegie Mellon University Robotics Institute, Pittsburgh, PA 15213-3890(412)[email protected]://www.cs.cmu.edu/afs/cs.cmu.edu/user/hcii/www/projectlisten.htmlCOPYRIGHT NOTICEThis publication and its companion video are copyrighted:1. J. Mostow. A Reading Coach that Listens: Project LISTEN (4-minute video). UIST ’95 Video Proceedings (EighthAnnual Symposium on User Interface Software and Technology), Sponsored by ACM SIGGRAPH and SIGCHI in coopera-tion with SIGSOFT, Pittsburgh, PA, November, 1995, pp. 52:34 - 56:40.2. J. Mostow, A. Hauptmann, and S. Roth. Demonstration of a Reading Coach that Listens. Proceedings of the EighthAnnual Symposium on User Interface Software and Technology, Sponsored by ACM SIGGRAPH and SIGCHI in coopera-tion with SIGSOFT, Pittsburgh, PA, November, 1995.Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without feeprovided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and thefull citation on the first page. Copyrights for components of this work owned by others than ACM must be honored.Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers or to redistribute to lists, requiresprior specific permission and/or a fee.Demonstration of a Reading Coach that ListensJack Mostow, Alexander G. Hauptmann, and Steven F. RothProject LISTEN, 215 Cyert Hall, 4910 Forbes AvenueCarnegie Mellon University Robotics Institute, Pittsburgh, PA 15213-3890(412)[email protected]://www.cs.cmu.edu/afs/cs.cmu.edu/user/hcii/www/projectlisten.htmlABSTRACTThe coach assists motivation by responding to the child’sProject LISTEN stands for "Literacy Innovation thatreading with supportive spoken feedback, and by reducingSpeech Technology ENables." We will demonstrate athe frustration that unassisted reading poses for strugglingprototype automated reading coach that displays text on areaders.screen, listens to a child read it aloud, and helps whereFUNCTIONS OF SPEECH RECOGNIZERneeded. We have tested successive prototypes of the coachWe adapted CMU’s connected speech recognizer to listenon several dozen second graders. [1] reports implemen-to children read. The listening capability required by thetation details and evaluation results. Here we summarize itscoach differs from conventional speech recognition, wherefunctionality, the issues it raises in human-computer inter-the task is to guess what the speaker said. Instead, theaction, and how it addresses them. We are redesigning thecoach knows the text the speaker is supposed to read, andcoach based on our experience, and will demonstrate itsmust perform the following three related functions tosuccessor at UIST ’95.provide the reading assistance described above.KEYWORDS: Speech interfaces for children, continuousTo speak the correct word when the reader gets stuck, thespeech recognition, education, children, non-readerscoach must track the reader’s position in the known text.GOALS OF READING COACHThis task is complicated by the various speech phenomenaReading is taught orally in grades 1-3 to help children relatecharacteristic of disfluent readers, including substitution,printed English to the spoken language they have alreadydeletion, repetition, hesitation, sounding out, and otheracquired. Unfortunately, a shocking percentage of thetypes of insertion.nation’s children lag behind grade level in reading [2] andTo provide corrective feedback, the coach must detect read-grow up functionally illiterate, at an annual productivitying mistakes. We define mistakes as "important words thatcost measured in hundreds of billions of dollars [3]. Anthe reader failed to speak." We do not treat insertions,automated reading coach could give such children hundredsrepetitions, self-corrections, or hesitations as mistakes.of hours of individualized attention that teachers andparents cannot. Thus LISTEN’s eventual goal is to helpTo know when to respond, the coach must detect the end ofchildren learn to read better over time. The goal of thethe sentence. This is a special case of tracking the reader’scurrent coach is to help them read a given text.position. It is not sufficient to wait for a long pause, be-cause young readers often pause in mid-sentence. Nor is itThe coach is designed to provide a combination of readingsufficient simply to wait until the recognizer detects the lastand listening, in which the child reads whenever possible,word of the sentence, both because the reader may misreadand the coach helps whenever necessary, so as to provide ait, and because speech recognition errors may cause prema-pleasant, successful reading experience. The coach’s assis-ture detection.tance, modelled after expert reading teachers, is intended tosupport word identification, comprehension, and motiva-Thus the reading application places novel demands on thetion.speech recognizer. The coach’s listening task is easier thanconventional speech recognition in the sense that it just hasTo assist word identification, the coach speaks words thatto detect where the reader is and which words were missedthe child clicks on, gets stuck on, or misreads.-- it doesn’t have to identify what the speaker said instead.To assist comprehension, the coach rereads sentencesBut this task is also harder since it involves an infinitewhere the child had difficulties. It uses digitized human"vocabulary" of possible substitutions and insertions, bothspeech, both for its natural quality and to project a suppor-words and non-words, many of which are highly confusabletive personality.with the correct text. Moreover, mispronunciations anddialect are not considered reading mistakes, and must there-fore be tolerated.ROBUSTNESS: HOW TOLERATE MISRECOGNITION?Since speech recognition is less than 100% accurate, wemust balance the proportion of reading mistakes the coachdetects against the frequency of false alarms -- correctwords misclassified as incorrect. We opted to give the prompts the student to reread it, and then responds bystudent the benefit of considerable doubt. In the evaluation speaking the correct word, as described before. If thereported in [1], the coach accepted over 96% of the cor- reader had trouble with more