Speech Processing 15-492/18-492MultilingualitySPICE: making it easierDealing with *all* LanguagesOver 6000 LanguagesOver 6000 LanguagesMaybe not all commercially interesting … nowMaybe not all commercially interesting … nowMajor languages (economic)Major languages (economic)Cell phone manufacturers list 46 languagesCell phone manufacturers list 46 languagesBut even those not all coveredBut even those not all covered ComputerizationComputerization: Speech is key technology: Speech is key technologyMobile Devices, Ubiquitous Information AccessMobile Devices, Ubiquitous Information AccessGlobalizationGlobalization: : MultilingualityMultilingualityMore than 6000 Languages in the world More than 6000 Languages in the world Multiple official languagesMultiple official languagesEurope has 20+ official languagesEurope has 20+ official languagesSouth Africa has 11 official languagesSouth Africa has 11 official languages⇒⇒Speech Processing in multiple LanguagesSpeech Processing in multiple LanguagesCrossCross--cultural Humancultural Human--Human InteractionHuman InteractionHumanHuman--Machine Interface in mother tongueMachine Interface in mother tongueMotivationChallengesAlgorithms language independent but require dataAlgorithms language independent but require dataDozens of hours audio recordings and corresponding transcriptionDozens of hours audio recordings and corresponding transcriptionssPronunciation dictionaries for large vocabularies (>100.000 wordPronunciation dictionaries for large vocabularies (>100.000 words)s)Millions of words written text corpora in various domains in queMillions of words written text corpora in various domains in questionstionBilingual aligned text corporaBilingual aligned text corporaBUT: Such data only available in very few languagesBUT: Such data only available in very few languagesAudio dataAudio data≤≤4040languages,languages,Transcriptions take up toTranscriptions take up to40x 40x real timereal timeLarge vocabulary pronunciation dictionariesLarge vocabulary pronunciation dictionaries≤≤2020languageslanguagesSmall text corporaSmall text corpora≤≤100 100 languages,languages,large corpora large corpora ≤≤30 30 languageslanguagesBilingual corpora in very few language pairs, pivot mostly EngliBilingual corpora in very few language pairs, pivot mostly EnglishshAdditional complications:Additional complications:Combinatorical explosionCombinatorical explosion(domain, speaking style, accent, dialect, ...)(domain, speaking style, accent, dialect, ...)Few native speakers at hand for minority (endangered) languagesFew native speakers at hand for minority (endangered) languagesLanguages without writing systemsLanguages without writing systemsSolution: Learning Systems⇒⇒Systems that learn a language from the userSystems that learn a language from the userEfficientEfficientlearning algorithms for speech processinglearning algorithms for speech processingLearning:Learning:Interactive learning with user in the loopInteractive learning with user in the loopStatistical modeling approachesStatistical modeling approachesEfficiency:Efficiency:Reduce amount of dataReduce amount of data(save time and costs): by a factor of 10(save time and costs): by a factor of 10Speed up development cycles:Speed up development cycles:days rather than monthsdays rather than months⇒⇒Rapid Language Rapid Language Adaptation from universal modelsAdaptation from universal modelsBridge the gap: language and technology expertsBridge the gap: language and technology expertsTechnology experts do not speak all languages in questionTechnology experts do not speak all languages in questionNative users are not in control of the technologyNative users are not in control of the technologySharing data between modulesLexstLMtWord s ↔Word t N-gramsAMtDicttWord →phone sequenceLMtN-gramsAMsDictsWord →phone sequenceLextsWord s ↔Word t LMsN-gramsAMsDictsLMsWord →phone sequenceN-gramsAMtDicttWord →phone sequenceInput LsInput LtOutput LsSpeech-to-Speech TranslationLsourceLtargetLsourceLtargetSPICESpeech Processing: Interactive Creation and Evaluation toolkit• National Science Foundation, Grant 10/2004, 3 years• Principle Investigators Tanja Schultz and Alan Black • Bridge the gap between technology experts → language experts• Automatic Speech Recognition (ASR), • Machine Translation (MT),• Text-to-Speech (TTS)• Develop web-based intelligent systems• Interactive Learning with user in the loop• Rapid Adaptation of universal models to unseen languages• SPICE webpage http://cmuspice.orgSpice Project PageInput: SpeechSpeech Processing SystemsPronunciation ruleshi /h//ai/you /j/u/we /w//i/hi youyou areI am AMLex LMOutput: Speech & TextHelloNLP / MTTTSText dataPhone set & Speech dataInput: Speechhi /h//ai/you /j/u/we /w//i/hi youyou areI am AMLex LMOutput: Speech & TextNLP / MTTTSPhone set & Speech data+HelloRapid Portability: DataFinding “Nice” PromptsFrom very large text databasesFrom very large text databasesFind “nice” sentences:Find “nice” sentences:Containing only high frequency wordsContaining only high frequency words55--15 words15 wordsFind grapheme/phoneme balanced setFind grapheme/phoneme balanced setSelect sentences with best Select sentences with best triphonetriphone/graph/graph500500--1000 sentences1000 sentencesCollect for ASR and TTS acoustic modelingCollect for ASR and TTS acoustic modelingPrompt Selection IssuesNeed good textNeed good textDeDe--htmlifyhtmlify, well, well--written, no misspellingwritten, no misspellingNeed word segmentationNeed word segmentationJapanese, Chinese ThaiJapanese, Chinese ThaiNatural text is often mixed languageNatural text is often mixed languageHindi Newspaper Text has lots of English wordsHindi Newspaper Text has lots of English wordsAutomatic selection has errorsAutomatic selection has errorsNeed Speaker to do further selectionNeed Speaker to do further selectionE.g. lots of telephone numbers, E.g. lots of telephone numbers, formatingformatingcommandscommandsCMU Arctic used similar methodsCMU Arctic used similar methodsRecording PromptsGlobalPhoneMultilingual