Speech Processing 15-492/18-492Spoken Dialog SystemsSDS componentsSpoken Dialog SystemsMore than just ASR and TTSMore than just ASR and TTSRecognitionRecognitionParsingParsingManipulation of utterancesManipulation of utterancesGeneration of new informationGeneration of new informationText generationText generationSynthesisSynthesisSDS ArchitectureSDS InternalsParserParserFrom words to structureFrom words to structureDialog ManagerDialog ManagerState of dialog (who is talking)State of dialog (who is talking)Direction of dialog (what next)Direction of dialog (what next)References, user profile etcReferences, user profile etcInteraction of database/internetInteraction of database/internetLanguage GenerationLanguage GenerationFrom structure to wordsFrom structure to wordsParsingParsing of SPEECH not TEXTParsing of SPEECH not TEXTEh, I Eh, I wannawannago, go, wannawannago to Boston tomorrowgo to Boston tomorrowIf its not too much trouble I’d be very grateful if If its not too much trouble I’d be very grateful if one might be able to aid me in arranging my one might be able to aid me in arranging my travel arrangements to Boston, Logan airport, travel arrangements to Boston, Logan airport, at sometime tomorrow morning, thank you.at sometime tomorrow morning, thank you.Boston, tomorrowBoston, tomorrowParsing: Output structure““I I wannawannago to Boston, tomorrow”go to Boston, tomorrow”Destination: BOSDestination: BOSDeparture: 20081028, AMDeparture: 20081028, AMAirline: unspecifiedAirline: unspecifiedSpecial: unspecifiedSpecial: unspecifiedConvert speech to structureConvert speech to structureSufficient for further processing/querySufficient for further processing/queryPhoenix Parser7[Place](carnegie mellon university)(downtown)(robinson towne center)(the airport)(south hills junction)(mount oliver)(the south side)(oakland)(bloomfield)(polish hill)(the strip district)(the north side);[NextBus](*WHEN_IS *the next *BUS)(*WHEN_IS *the BUS after that *BUS)WHEN_IS(when is)(when's)BUS(bus)(one);Phoenix ParserParse what is importantParse what is importantIgnore other partsIgnore other partsMap know parts to usually informationMap know parts to usually informationParsing vs Language ModelLanguage ModelLanguage ModelModel what actually gets saysModel what actually gets saysParsing Parsing Extract the information you wantExtract the information you wantModels *can* be sharedModels *can* be sharedOnly accept things in the grammarOnly accept things in the grammarCan be over limitingCan be over limitingDialog ManagerMaintain stateMaintain stateWhere are we in the dialogWhere are we in the dialogWhose turn is itWhose turn is itWaiting for speakerWaiting for speakerWaiting for database query (stall user)Waiting for database query (stall user)Deal with bargeDeal with barge--ininLanguage GenerationQuery for flights to BostonQuery for flights to BostonTemplate fill Template fill answer(sanswer(s))The next flight to DEST leaves at The next flight to DEST leaves at DEPART_TIME arriving at ARRIVE_TIME.DEPART_TIME arriving at ARRIVE_TIME.Templates may be much more complexTemplates may be much more complexLanguage GenerationChoose which template to useChoose which template to useBased on state, answer typeBased on state, answer typeNatural variationNatural variationStatistical variationStatistical variationInclude <Include <ssmlssml> tags to help synthesis> tags to help synthesisCan <Can <emphemph>emphasize</>emphasize</emphemph> parts> partsCan identify dates, numbers etc.Can identify dates, numbers etc.Humans like variation in the outputHumans like variation in the outputIt is rare for a human to repeat things exactlyIt is rare for a human to repeat things exactlyLanguage GenerationFrames structures to (marked up) textFrames structures to (marked up) textSTART: PittsburghSTART: PittsburghEND: BostonEND: BostonDATE: 20081028DATE: 20081028TIME: 07:45TIME: 07:45FLIGHT: US075FLIGHT: US075Can generationCan generationI have US 075 leaving at 07:45 tomorrowI have US 075 leaving at 07:45 tomorrowUS Airways has a flight departing tomorrow at 07:45US Airways has a flight departing tomorrow at 07:45Standardized thingsHelpHelpUser should be able to get help at any timeUser should be able to get help at any timeExplain where they are and what they are Explain where they are and what they are expected to say (with explicit examples)expected to say (with explicit examples)ErrorsErrors“I didn’t understand” …“I didn’t understand” …ConfirmationConfirmationDid you say “Boston”?Did you say “Boston”?ConfirmationExplicit confirmation Explicit confirmation Where are you traveling to ?Where are you traveling to ?BostonBostonBoston, did I get that right?Boston, did I get that right?YesYesConfirmationImplicit confirmationImplicit confirmationWhere are you traveling to?Where are you traveling to?BostonBostonBoston, where …Boston, where …<can barge in><can barge in>ConfirmationExplicit confirmationExplicit confirmationSafe but slowSafe but slowImplicit confirmationImplicit confirmationNatural, but requires good support for bargeNatural, but requires good support for barge--in inGroundingShowing evidence the system understandsShowing evidence the system understandsWhere are you traveling to?Where are you traveling to?Boston.Boston.Right. Where ….Right. Where ….Boston, right. Where ….Boston, right. Where ….Designing PromptsConstrain your questions:Constrain your questions:How may I help you?How may I help you?Long story replyLong story replyWhat bus number would like schedules for?What bus number would like schedules for?Expect bus number repliesExpect bus number