TESSA, a system to aid communication with deaf peopleStephen Cox, MichaelLincoln and JudyTryggvasonSchool of InformationSystems,University of East Anglia,Norwich NR4 7TJ, U.K.Melanie NakisaRoyal National Institute forDeaf People,19–23 Featherstone Street,London EC1Y 8SL U.K.Mark Wells, Marcus Tuttand Sanja AbbottTelevirtual Ltd.Anglia House,Agricultural Hall Plain,Norwich, U.K.ABSTRACTTESSA is an experimental system that aims to aidtransactions between a deaf person and a clerk in a PostOffice by translating the clerk’s speech to sign language.A speech recogniser recognises speech from the clerk andthe system then synthesizes the appropriate sequence ofsigns in British Sign language (BSL) using a specially-developed avatar. By using a phrase lookup approach tolanguage translation, which is appropriate for the highlyconstrained discourse in a Post Office, we were able tobuild a working system that we could evaluate. Wesummarise the results of this evaluation (undertaken bydeaf users and Post office clerks), and discuss how thefindings from the evaluation are being used in thedevelopment of an improved system.KeywordsInteractive systems, translation systems, Aids for theDeaf, speech recognition, avatars.INTRODUCTIONThere has recently been considerable research activity indeveloping automatic systems which can understand andoutput speech to provide information services or toperform transactions with customers [10]. Most of thesesystems have been developed for use over the telephonenetwork with the goal of replacing completely or assistinga human operator [5]. A key aspect of them is that theyoperate in a rather restricted domain of discourse (e.g.train timetable enquiries [9], e-mail access [17] directoryenquiries [18]) and this gives them some robustness to thedifficult problems of variability and “noise” in thelanguage used by the speakers, the speech signal and thetelephony channel. There has also been work oninteractive speech-to-speech translation systems e.g. [12]. These systems are designed to provide translation ofconversational speech between languages with apotentially very large vocabulary [13]. We have beendeveloping a system which combines aspects of bothkinds of systems mentioned above. It is an interactivetranslation system but it operates in a very restricteddomain and is designed to assist in the completion of atransaction between a Post Office (PO) clerk and a deafcustomer. The system translates the clerk's speech intoBritish Sign Language (BSL) and displays the signs usinga specially-developed avatar. A comprehensive approachto the task of enabling humans who cannot sign tocommunicate using sign-language would clearly requirethe development of a general purpose speech to sign-language converter. This in turn requires the solution ofthe following problems:1. automatic speech to text conversion (speechrecognition);2. automatic translation of arbitrary English textinto a suitable representation of sign language;3. display of this representation as a sequence ofsigns using computer graphics techniques.However, recent research into "formulaic" language hasshown that cross-language communication is possibleusing a limited set of pre-defined phrases, provided thatthe discourse between the participants is highlyconstrained in its topics and scope [16]. We chose todevelop a system for use in a PO because many of thetransactions are highly predictable and hence much of theassociated language can be pre-defined. This enables usto sidestep or simplify many of the difficult problemsmentioned above by defining a limited set of phrases thatcan be recognised and displayed in sign-language.Although this imposes restrictions on what can be“translated”, it is still likely to form a useful systembecause the task underlying the translation is a narrowone. Our aim in developing the current system was toprovide a speech-to-sign translation system which hadlimited generality, but which could be used to accomplishtransactions and hence whose benefit to the deafcommunity could be objectively evaluated. Theinformation obtained by deploying and evaluating asimple working system should be invaluable in thedevelopment of more complex and sophisticatedtranslation systems that may be of higher generality.The system has been developed with the collaboration ofConsignia (the UK Post Office), and research continuesas part of the European Union fifth framework project,ViSiCAST [2], which aims to benefit deaf citizens byfacilitating access to information and services by usingsign language.OVERVIEW OF THE SYSTEMDesign philosophyOur goal was to develop a system to enable a Post Officecounter-clerk to communicate with a deaf customer usingautomatically-generated sign-language, and hence to aidcompletion of a transaction. A priori, it might seem thatrecognising the clerk's speech and displaying it as text tothe deaf customer would be adequate. However, formany people who have been profoundly deaf from ayoung age, signing is their first language so they learn toread and write English as a second language [4]. As aresult, many deaf people have below-average readingabilities for English text and prefer to communicate usingsign language [15].Two variants of sign language are possible forcommunication with deaf people in the UK: British SignLanguage (BSL) and Sign-supported English (SSE). BSLis a fully developed language with it own syntactical andsemantic structures [3], whereas SSE uses the same (orvery similar) signs for words as BSL, but uses Englishlanguage word order [8]. Using pre-stored SSE “words”would enable sentences to be translated into signlanguage, but SSE is not popular with the deafcommunity and it is very important that the system isacceptable to deaf users. Another approach is to usewhole BSL phrase units rather than words. This ispossible only if a small number of phrases is required,and these phrases can be recorded in BSL rather thanSSE. If recording of the signs is done correctly, phrasescan be concatenated to a certain extent e.g. amounts ofmoney can be slotted into a carrier phrase such as “Thecost is...”. Although this approach imposes considerablerestrictions on the meanings that can be conveyed in BSL,and hence on the dialogue, the limited nature of thetransactions in the Post Office should mean that mosttransactions could be completed in this way.System componentsFigure 1 is a diagram showing the structure of the system.The Post
View Full Document
Unlocking...