New version page

patel-06-iva

Upgrade to remove ads

This preview shows page 1-2-3-4 out of 12 pages.

Save
View Full Document
Premium Document
Do you want full access? Go Premium and unlock all 12 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 12 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 12 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 12 pages.
Access to all documents
Download any document
Ad free experience

Upgrade to remove ads
Unformatted text preview:

Dealing with Out of Domain Questions inVirtual CharactersRonakkumar Patel, Anton Leuski, and David TraumInstitute for Creative TechnologiesUniversity of Southern CaliforniaMarina del Rey, CA 90292, USA{patelr, leuski, traum}@ict.usc.eduAbstract. We consider the problem of designing virtual characters thatsupport speech-based interactions in a limited domain. Previously wehave shown that classification can be an effective and robust tool forselecting appropriate in-domain responses. In this paper, we considerthe problem of dealing with out-of-domain user questions. We introducea taxonomy of out-of-domain resp onse typ es. We consider three classi-fication architectures for selecting the most appropriate out-of-domainresponses. We evaluate these architectures and show that they signifi-cantly improve the quality of the response selection making the user’sinteraction with the virtual character more natural and engaging.1 IntroductionPrevious work has shown that limited domain virtual humans that use spokeninteraction can be quite successful in terms of delivering quality answers toin-domain questions [2, 3]. Question-answering characters can serve a numberof purposes, including entertainment, training, and education. For a question-answering character, a key point is to give human-like responses to questionswhen no answer is available. The character should act like a person who eitherdoes not know or does not want to reveal the answer: recognizing explicitly thatsomething is “off-topic” and giving a response indicating this recognition is betterthan providing an inappropriate in-domain answer. While a character could beconstructed to always reply with something generic like “I don’t know”, this canlead to repetitive behavior that breaks a sense of immersion. Having a set ofsuch answers allows the character to seem more engaging, by producing somevariety in his responses. Thus we have constructed a set of off-topic responsesfor our characters to choose from.We have found, however, that not all off-topic responses are equally satisfac-tory as replies to each of a range of off-topic questions. In this paper we explorewhether the general category “off-topic” can be broken down into appropriatesub-categories to achieve higher performance. We use the SGT Blackwell char-acter [2, 3], as a testb e d for this exploration, and create a taxonomy of typ es ofoff-topic areas, a set of replies for the SGT Blackwell character for each area.We further evaluate performance of several classification-based architectures thatuse the off-topic taxonomy, as to how satisfactory the answers are. The resultsshow that the best architecture significantly out-performs the baseline character,– which does not use the taxonomy, – on both on-topic and off-topic questions.In the next section we give an overview of the SGT Blackwell characterand the baseline question-understanding/response. In Section 3 we discuss ataxonomy of off-topic response classes , which we hope can reduce the numberof inappropriate off-topic responses. In Section 4 we describe three differentclassification-based architectures, which are intended to improve the baselineclassifier, using the off-topic taxonomy. In Section 5, we present the results ofevaluating the three architectures with respect to the quality of answers given.Finally, in Section 6 we summarize our results and outline some directions forfuture work.Fig. 1. SGT Blackwell2 The baseline SGT Blackwell SystemSGT Blackwell, shown in Figure 1, is a life-sized character projected on a trans-parent screen. He is meant to answer questions from a user acting as a reporterinterviewing him about his role in the Army and the technology at the Institutefor Creative Technology that created him. A user talks to SGT Blackwell usinga head mounted microphone. For speech recognition, we use a hybrid limiteddomain/general language model [4], built using the SONIC system [5]. A classi-fier [3] then analyzes the text output and selects the highest scoring answer. Theanswers are pre-recorded audio clips linked with animation, which are playedthrough the game engine to s how SGT Blackwell providing the response. SGTBlackwell’s responses include spoken lines ranging from one word to a coupleparagraphs. There are 55 content answers with domain information. When SGTBlackwell detects a question that cannot be answered with one of the content(on-topic) answers, he picks a random answer from a pool of 17 off-topic answers.The classifier is based on statistical language m odeling techniques used incross-lingual information retrieval. It represents a text string with a languagemodel – a probability distribution over the words in the string. The classifierviews both questions and answers as samples from two different “languages” thelanguage of questions and the language of answers. Given an input question fromthe user, the classifier calculates the language model of the most likely answerfor the question, – it uses the training data as a dictionary to “translate” thequestion into an answer, – then it compares that model to the language model ofindividual answers, and selects the best matching response. We showed that thistechnique outperforms traditional text classification approaches such as support-vector machines for tasks that have a large number of response classes [3].BaselineClassifierQuestionOn-topic Answer: Class 1On-topic Answer: Class 2On-topic Answer: Class 55Off-topic Answer: Class 56...Fig. 2. Baseline classifier architectureIn order to train the classifier, we have created a training corpus of questionslinked to either one of the 55 content classes or the “off-topic” class. Questionsand answers were created using a multi-stage process, including scripted ini-tial questions, manual paraphrases, and collected questions from a Wizard ofOz study, in which naive users were allowed to ask whatever they wanted, af-ter a brief description of the intended domain. We also used human coders tolink questions to appropriate answers using the question-answer quality rankingscheme suggested by Gandhe and his colleagues [6], described in Section 5. Thebaseline training set included 1572 questions linked to the 56 answer classes.Figure 2 shows the design of the question-response part of the initial system,which serves as the baseline for our study. It has one classifier, which can deliveron-topic or off-topic answers based the input question. This design assumes thatall


Download patel-06-iva
Our administrator received your request to download this document. We will send you the file to your email shortly.
Loading Unlocking...
Login

Join to view patel-06-iva and access 3M+ class-specific study document.

or
We will never post anything without your permission.
Don't have an account?
Sign Up

Join to view patel-06-iva 2 2 and access 3M+ class-specific study document.

or

By creating an account you agree to our Privacy Policy and Terms Of Use

Already a member?