Evaluation of a Practical Interlingua for Task-Oriented DialogueInterchange Format DesignExpressivity vs SimplicityTask Oriented SentencesDomain Actions: Extended, Domain-Specific Speech ActsComponents of the Interchange FormatExamplesNot Covered or Not Represented in IFScope of the IFExpressivity: Coverage ExperimentCoverage Experiment: Development and Test DataThe Interchange Format DatabaseCoverage of Top 10 Dialogue Acts in Development DataCoverage of Top 10 Speech Acts in Development DataCoverage of Top 10 Dialogue Acts in Test DataCoverage of Top 10 Speech Acts in Test DataSimplicity: Consistency of Use Across SitesInter-coder Agreement ExperimentInter-Coder Agreement ResutsInter-Coder Agreement Error Analysis of 33 SentencesCross-Site EvaluationIntra-Site EvaluationCross Site Evaluation DataEvaluation ScoringEnd-to-End Evaluation ResultsSlide 26ConclusionsCurrent WorkSIG IL 2000Evaluation of a Practical Interlingua for Task-Oriented DialogueLori Levin, Donna Gates, Alon Lavie, Fabio Pianesi, Dorcas Wallace, Taro Watanabe, Monika WoszczynaSIG IL 2000Interchange Format DesignThe CSTAR II Interchange Format was designed and developed by all of the CSTAR II partners: CMU, IRST, ETRI, UKA, CLIPS++ and ATR.www.c-star.orgSIG IL 2000Expressivity vs Simplicity•If it is not expressive enough, components of meaning will be lost.•If it is not simple enough, it can’t be used reliably across sites.•If it is not simple enough, it will not be quickly portable to new domains.SIG IL 2000Task Oriented Sentences•Perform an action in the domain.•Are not descriptive.•Contain fixed expressions that cannot be translated literally.SIG IL 2000Domain Actions: Extended, Domain-Specific Speech ActsExamples: c:request-information+availability+room a:give-information+personal-data c:give-information+temporal+arrivalSIG IL 2000Components of the Interchange Formatspeaker a: a: (agent)speech act give-informationgive-informationconcept* +availability+room+availability+roomargument* (room-type=(single & double), (room-type=(single & double), time=md12)time=md12)SIG IL 2000Examplesno that’s not necessaryc:negatec:negateyes I amc:affirmc:affirmand I was wondering what you have in the way of rooms available during that timec:request-information+availability+roomc:request-information+availability+roommy name is alex waibelc:give-information+personal-data (person-name=(given-name=alex, family-c:give-information+personal-data (person-name=(given-name=alex, family-name=waibel))name=waibel))and how will you be paying for thisa:request-information+payment (method=question)a:request-information+payment (method=question)I have a mastercardc:give-information+payment (method=mastercard)c:give-information+payment (method=mastercard)SIG IL 2000Not Covered orNot Represented in IF•Relative clauses•Comparatives (in general)•Tense•Number (but quantity is represented)SIG IL 2000Scope of the IFMay 1999 Speech acts 54  Concepts 84 Arguments 118SIG IL 2000Expressivity: Coverage Experiment•Development data was tagged with interlingua representations by human experts. •Sentences that are not intended to be covered by the interlingua (as judged by human experts) were given the tag “no-tag.”•Test data was tagged by human experts.SIG IL 2000Coverage Experiment:Development and Test DataLanguages Dialogue Type Number of DAUnitsDevelopment Data:English monolingual 2698Italian monolingual 234Korean bilingual (onlyKorean utterancesare included)1142Test Data:Japanese-English bilingual 6069SIG IL 2000The Interchange Format Database61.2.3 olang I lang I Prv IRST “telefono per prenotare delle stanze per quattro colleghi”61.2.3 olang I lang E Prv IRST “I’m calling to book some rooms for four colleagues”61.2.3 IF Prv IRST c:request-action+reservation+features+room (for-whom= (associate, quantity=4))61.2.3 comments: dial-oo5-spkB-roca0-02-3d.u.sdu olang X lang Y Prv Z “sdu-in-language-Y on one line”d.u.sdu olang X lang E Prv Z “sdu-in-English on one line”d.u.sdu IF Prv Z dialogue-act-on-one-lined.u.asdu comments: your commentsd.u.asdu comments: go hereSIG IL 2000Coverage of Top 10 Dialogue Acts in Development DataCumulative % Percent Count DA5.9 244 no-tag15.7 15.7 652 acknowledge19.8 4.1 172 affirm23.3 3.4 143 thank26.0 2.7 113 introduce-self28.0 2.0 85 give-info+price30.1 2.0 85 greeting31.9 1.9 78 give-info+temp33.7 1.8 75 give-info+num35.5 1.8 73 give-info+price+room37.2 1.7 70 req-info+paymentSIG IL 2000Coverage of Top 10 Speech Acts in Development DataCumulative % Percent Count Speech Act30.1 30.1 1250 give-information45.8 15.7 655 acknowledge57.7 11.9 493 req-information62.7 5.0 209 req-verif-give-inf.67.6 4.9 203 request-action71.7 4.1 172 affirm75.1 3.4 143 thank77.9 2.7 113 introduce-self80.2 2.4 98 offer82.4 2.1 89 acceptSIG IL 2000Coverage of Top 10 Dialogue Acts in Test Data Cumulative % Percent Count DA4.6 263 no-tag15.6 15.6 885 acknowledge20.2 4.6 260 thank23.7 3.5 200 introduce-self27.0 3.4 191 affirm29.7 2.7 153 apologize32.3 2.6 147 greeting34.6 2.3 128 closing36.3 1.7 98 give-info+personal38.0 1.7 95 give-info+temp.39.5 1.6 89 give-info+priceSIG IL 2000Coverage of Top 10 Speech Acts in Test DataCumulative % Percent Count DA25.6 25.6 1454 give-information41.7 16.1 916 acknowledge53.6 11.9 677 req-information58.2 4.6 260 thank62.0 3.7 213 req-verif-give-info65.5 3.5 200 Introduce-self68.8 3.4 191 affirm72.0 3.2 181 request-action74.8 2.8 159 accept77.5 2.7 153 apologizeSIG IL 2000Simplicity:Consistency of Use Across Sites•Successful international demo.•After testing English-Italian and English-Korean, Italian-Korean worked without extra effort.•Inter-coder agreement experiment•Cross-site evaluation experimentSIG IL 2000Inter-coder Agreement Experiment•84 DA units from Japanese-English data•Some dialogue fragments and some isolated sentences•Coded at CMU and IRST•Results reported in percent agreementSIG IL 2000Inter-Coder Agreement ResutsSpeech Act 82.14Concept List 88.00Dialogue Act 65.48Argument List 85.79SIG IL 2000Inter-Coder Agreement Error Analysis of 33 Sentences•6 are equivalent due to ambiguity in the IF specification.•16 are similar enough to produce output with equivalent meaning.–offer-search+availability: Let me check the