Improving Literacy in Developing Countries Using Speech Recognition-Supported Games on Mobile Devices Anuj Kumar1, Pooja Reddy2, Anuj Tewari3, Rajat Agrawal1, Matthew Kam1 1Human-Computer Interaction Institute, Carnegie Mellon University, Pittsburgh, PA, USA, 15213 2American Institutes for Research, Washington D.C., USA, 20007 3Computer Science Division and Berkeley Institute of Design, University of California, Berkeley, USA [email protected] ABSTRACT Learning to read in a second language is challenging, but highly rewarding. For low-income children in developing countries, this task can be significantly more challenging because of lack of access to high-quality schooling, but can potentially improve economic prospects at the same time. A synthesis of research findings suggests that practicing recalling and vocalizing words for expressing an intended meaning could improve word reading skills – including reading in a second language – more than silent recognition of what the given words mean. Unfortunately, many language learning software do not support this instructional approach, owing to the technical challenges of incorporating speech recognition support to check that the learner is vocalizing the correct word. In this paper, we present results from a usability test and two subsequent experiments that explore the use of two speech recognition-enabled mobile games to help rural children in India read words with understanding. Through a working speech recognition prototype, we discuss two major contributions of this work: first, we give empirical evidence that shows the extent to which productive training (i.e. vocalizing words) is superior to receptive vocabulary training, and discuss the use of scaffolding hints to “unpack” factors in the learner‟s linguistic knowledge that may impact reading. Second, we discuss what our results suggest for future research in HCI. Author Keywords Educational games; developing countries; information and communication technology and development (ICTD); literacy; mobile learning; speech recognition ACM Classification Keywords H.5.m [Information Interfaces and Presentation]: Miscellaneous; General Terms Design; Human Factors INTRODUCTION More than half of the world speaks at least two languages [9], to the extent that “bilingualism and multilingualism are a normal and unremarkable necessity to everyday life for the majority of the world‟s population” [12]. Multilingualism arose because of reasons that include colonialism, diglossia, and efforts to promote national identity [12, 32]. Consequently, many children in both industrialized and developing nations grow up having been exposed to, and even learning to read in, at least two languages, neither of which may be spoken at home. However, most languages in multilingual societies do not share equal status; a “global” or “national” language (e.g., English, French, Mandarin, or Spanish) may co-exist with the vernacular languages, with the former privileged as the medium of instruction or official language of business. Sadly, many children in the developing world stand to lose significant opportunities in life when their schools struggle to provide high-quality literacy instruction, sometimes for a vernacular language and more often in a second language (e.g., English), which teachers themselves lack proficiency in. The British Council estimates wage differences between salaried professionals with and without English skills to be between 20% and 30% in Bangladesh, Cameroon, Nigeria, Pakistan and Rwanda [32]. In postcolonial Morocco, where French is the official language of business, switching the medium of instruction from French to [the local language] Arabic is associated with a 50% reduction in the economic returns to schooling [2]. In India, from surveys with poor parents [27], English is one of the two most sought-after skills. In the city of Mumbai in India, for instance, “schooling in [the local language] Marathi channels the child into working class jobs, while more expensive English education significantly increases the likelihood of obtaining a coveted white-collar job” [24], such that English speakers experience returns on investment in schooling between 24% and 27%, while the returns for non- speakers are 10%. Word reading skills, which includes vocabulary knowledge, is a significant predictor of the ability to read and comprehend longer passages of texts [30]. Studies have shown successes on the cellphone [18] and desktop Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. CHI’12, May 5–10, 2012, Austin, Texas, USA. Copyright 2012 ACM 978-1-4503-1015-4/12/05...$10.00.computer [29] in developing countries with receptive vocabulary training (i.e., recognizing the meaning of a word when the reader sees it). Drawing on the research literature that we will discuss below, productive vocabulary practice (i.e., recalling the word for expressing a meaning and vocalizing aloud) is likely to bring about higher gains in vocabulary knowledge. We therefore hypothesize that productive vocabulary training, which language learning software applications can support via speech recognition (which checks that the learner is saying the correct word), can yield stronger literacy gains. While speech recognition remains a computationally difficult problem, speech in the niche domain of vocabulary learning is significantly more tractable when recognition is isolated to individual words (i.e., no need to locate boundary words, when recognizing phrases and longer sequences of words in the speech signal) and small vocabularies (which is usually feasible in practice because the words to be taught can be organized into short vocabulary lists in most situations). This paper therefore proceeds as follows: first, based on the literature in second language learning and the psychology of