Spoken Dialogue Systems Julia Hirschberg CS 4706 01 14 19 1 Today Some Swedish examples Controlling the dialogue flow State prediction Controlling lexical choice Learning from human human dialogue User feedback Evaluating systems 01 14 19 2 The Waxholm Project at KTH tourist information Stockholm archipelago time tables hotels hostels camping and dining possibilities mixed initiative dialogue speech recognition multimodal synthesis graphic information pictures maps charts and timetables Demos at http www speech kth se multimodal 01 14 19 3 The Waxholm system 01 14 19 There Information Information Information Which areWhen IIs Ilots Waxholm am Which think day This itWhere looking possible of Ido about about of want Where about boats is I hotels the the want Thank ais can Thank table hotels to the evening The for week the shown from to isto go are Ieat restaurants boats Waxholm you hotels find city of go tomorrow you is do Stockholm in in the to on shown boats too Waxholm hotels you Waxholm to Waxholm in boats this Waxholm Waxholm want depart in map inWaxholm to this toWaxholm go table is on a Friday Fromis At shown where shown whatin do time inthis you this do table want table you to want go to go 4 Today Some Swedish examples Controlling the dialogue flow State prediction Controlling lexical choice Learning from human human dialogue User feedback Evaluating systems 01 14 19 5 Dialogue control state prediction Dialog grammar specified by a number of states Each state associated with an action database search system question Probable state determined from semantic features Transition probability from one state to state Dialog control design tool with a graphic interface 01 14 19 6 Waxholm Topics TIME TABLE Task get a time table Example N r g r b ten When does the boat leave SHOW MAP Task get a chart or a map displayed Example Var ligger Vaxholm Where is Vaxholm located EXIST Task display lodging and dining possibilities Example Var finns det vandrarhem Where are there hostels OUT OF DOMAIN Task the subject is out of the domain Example Kan jag boka rum Can I book a room NO UNDERSTANDING Task no understanding of user intentions Example Jag heter Olle My name is Olle END SCENARIO Task end a dialog Example Tack Thank you 01 14 19 7 Topic selection FEATURES 01 14 19 TOPIC EXAMPLES TIME TABLE SHOW MAP FACILITY NO UNDER OUT OF STANDING DOMAIN OBJECT QUEST WHEN QUEST WHERE FROM PLACE AT PLACE 062 188 062 250 062 312 031 688 031 219 073 024 390 024 293 091 091 091 091 091 067 067 067 067 067 091 091 091 091 091 TIME PLACE OOD END HOTEL HOSTEL ISLAND PORT MOVE 312 091 062 062 062 062 333 125 875 031 200 031 031 031 031 556 750 031 024 500 122 024 488 122 062 244 098 091 091 091 091 091 091 091 091 091 067 067 933 067 067 067 067 067 067 091 091 091 909 091 091 091 091 091 p ti F argmax i END 8 Topic prediction results Errors 15 12 9 8 8 10 5 12 7 8 5 All no understanding excluded 3 1 2 9 0 complete parse 01 14 19 raw data no extra linguistic sounds 9 Today Some Swedish examples Controlling the dialogue flow State prediction Controlling lexical choice Learning from human human dialogue User feedback Evaluating systems 01 14 19 10 User answers to questions The answers to the question What weekday do you want to go Vilken veckodag vill du ka 22 11 11 7 6 01 14 19 Friday fredag I want to go on Friday jag vill ka p fredag I want to go today jag vill ka idag on Friday p fredag I want to go a Friday jag vill ka en fredag are there any hotels in Vaxholm finns det n gra hotell i Vaxholm 11 Examples of questions and answers Hur ofta ker du utomlands p semestern Hur ofta reser du utomlands p semestern jag reser en g ng om ret utomlands jag ker en g ng om ret kanske jag reser inte ofta utomlands p semester det blir mera i jag ker ganska s llan utomlands p semester jag ker n stan alltid utomlands under min semesterarbetet jag reser reser utomlands p semestern vartannat r jag reser utomlands en g ng per semester jag ker ungef r 2 g nger per r utomlands p jag reser utomlands p semester ungef r en g ng per r semester jag brukar resa utomlands p semestern tminst ne en g ng jag ker utomlands n stan varje r i ret jag ker utomlands p semestern varje r en g ng per r kanske jag ker utomlands ungef r en g ng om ret en g ng vart annat r jag r n stan aldrig utomlands varje r en eller tv g nger om ret vart tredje r ungef r en g ng per semester nu f r tiden inte s ofta kanske en g ng per r varje r brukar jag ka utomlands ungef r en g ng per r tminst ne en g ng om ret n stan aldrig 01 14 19 12 Results no no reuse 4 2 answer other 24 reuse 52 18 ellipse 01 14 19 13 Today Some Swedish examples Controlling the dialogue flow State prediction Controlling lexical choice Learning from human human dialogue User feedback Evaluating systems 01 14 19 14 The August system 01 14 19 Yes Strindberg When itWhat Do You do might you Thank Good you were are islive was like be your do welcome you bye that born you for itdepartment name born we ameet inliving will 1849 People IIWhat IYes call Over can The How come Strindberg The Perhaps myself answer that information who amany Royal from million was we Strindberg questions people was the Institute ain will people smart married glass ishere live shown thing of about houses live but in I Strindberg ofdon t Speech should in the really three Technology Stockholm on soon not Stockholm KTH Music to the have throw say times again map and and a surname stones Stockholm area Hearing 15 Evidence from Human Performance Users provide explicit positive and negative feedback Corpus based vs laboratory experiments do these tell us different things 01 14 19 16 Adapt demonstration of complete system 01 14 19 17 Feedback and Grounding Bell Gustafson 00 Positive and negative Previous corpora August system 18 of users gave pos or neg feedback in subcorpus Push to talk Corpus Adapt system 50 dialogues 33 subjects 1845 utterances Feedback utterances labeled w Positive or negative Explicit or implicit Attention Attitude Results 18 of utterances contained feedback 94 of users provided 01 14 19 18 65 positive 2 3 explicit equal amounts of attention vs attitude Large variation Some subjects provided at almost every turn Some never did Utility of study Use positive feedback to model the user better preferences Use negative feedback in error detection 01 14 19 19 The HIGGINS domain This is a 3D test environment The primary domain of HIGGINS is city navigation for pedestrians Secondarily HIGGINS is …