Speech Processing 15-492/18-492Speech TranslationCase study: TranstacDetailsTranstac: Two S2S SystemDARPA developed forDARPA developed forCheck points, medical and civil defenseCheck points, medical and civil defenseRequirementsRequirementsTwo wayTwo wayEyesEyes--free (no screen)free (no screen)PortablePortableUsable by real Usable by real usersSusersSTranstac SystemLaptop secured in BackpackOptional speech controlPush-to-Talk ButtonsClose-talking MicrophoneSmall powerful SpeakersTranstac System DetailsTwo way systemTwo way system2 ASR systems: English and Iraqi2 ASR systems: English and Iraqi2 way statistical translation2 way statistical translation2 synthesizers2 synthesizersPushPush--toto--talk systemtalk system(Users don’t like “translate everything mode”)(Users don’t like “translate everything mode”)Echo back ASR resultEcho back ASR resultAnd then translationAnd then translationIraqi LanguageIraqi Arabic is a dialectIraqi Arabic is a dialectMost Iraqi’s write Modern Standard ArabicMost Iraqi’s write Modern Standard ArabicMost Iraqi’s do not write their own dialectMost Iraqi’s do not write their own dialectNo standardized spellingNo standardized spellingTranstacTranstacproject invented oneproject invented oneBut Iraqi’s may not be used to itBut Iraqi’s may not be used to itArabic (MSA and dialects)Arabic (MSA and dialects)Do not write short vowels in wordsDo not write short vowels in wordsData for TrainingCollected human mediated dialogsCollected human mediated dialogsHuman acts as a machineHuman acts as a machinePassed a microphone back an forwardPassed a microphone back an forwardTry to get people not to talk at same timeTry to get people not to talk at same timeLarge number of collections (over 4 years)Large number of collections (over 4 years)650 thousand sentences pairs650 thousand sentences pairsMany different speakersMany different speakersHand transcribed by experts (in Iraqi spelling)Hand transcribed by experts (in Iraqi spelling)Hand translate (Source sentences and Interpreter’s)Hand translate (Source sentences and Interpreter’s)Iraqi ASRAcoustic model from Iraqi dataAcoustic model from Iraqi dataBased on MSA Based on MSA phonesetphonesetNeeds to be small fast modelsNeeds to be small fast modelsDiscriminative TrainingDiscriminative TrainingSpeaker specific adaptationSpeaker specific adaptationLexiconLexiconBased on LDC provided lexiconBased on LDC provided lexiconMultiple pronunciations/typos still a problemMultiple pronunciations/typos still a problemStatistically trained LTS rulesStatistically trained LTS rulesLanguage ModelLanguage ModelTrained on Iraqi input (and translated output)Trained on Iraqi input (and translated output)English ASRAcoustic modelAcoustic modelOriginally using other modelsOriginally using other modelsThen trained from collected dataThen trained from collected data(Mostly military personnel)(Mostly military personnel)LexiconLexiconExisting lexicon but needed to add Military speak: Existing lexicon but needed to add Military speak: MRAP, IEDMRAP, IEDLanguage modelLanguage modelTrained from data providedTrained from data providedTrained from “similar” data found on the webTrained from “similar” data found on the webTraining from hand created “typical” examplesTraining from hand created “typical” examplesTTSStandard English TTSStandard English TTSAppropriate “command” voiceAppropriate “command” voiceUnit selectionUnit selectionAdded lots of military vocabularyAdded lots of military vocabularyIraqi TTSIraqi TTSRecorded from Iraqi radio announcerRecorded from Iraqi radio announcerBased on example sentences in the domainBased on example sentences in the domainLDC lexicon and LTS rules (same as ASR)LDC lexicon and LTS rules (same as ASR)Hand tunedHand tunedS2S Interface IssuesHow do you teach people to use the systemHow do you teach people to use the system““TranstacTranstacsay instructions”say instructions”Not really sufficientNot really sufficientHow can you tell it translated correctlyHow can you tell it translated correctlyGive (speech) feedback.Give (speech) feedback.BacktranslationBacktranslationASR echo backASR echo backS2S Interface IssuesHow do you translate namesHow do you translate namesA correct translation/transliteration is hard to A correct translation/transliteration is hard to understandunderstandMark names in translationsMark names in translations“My name is … Abdullah”“My name is … Abdullah”“He lives on … al“He lives on … al--AqarAqar… street”… street”S2S Evaluation (Transtac)Offline testsOffline testsASRASR-->Text and Text>Text and Text-->Text>TextCompare to translation referencesCompare to translation referencesWER and “BLEU” scoreWER and “BLEU” scoreOnline testsOnline testsConcept transfer (through defined scenarios)Concept transfer (through defined scenarios)Speed (number of concepts per minute)Speed (number of concepts per minute)(English speech masking)(English speech masking)Utility testsUtility testsDoes it really workDoes it really workTranstac ParticipantsDeveloper groupsDeveloper groupsIBMIBMSRISRIBBNBBNCMUCMUUSCUSCEvaluationsEvaluationsTwice a year in Iraqi (somewhere in DC)Twice a year in Iraqi (somewhere in DC)One surprise language (Farsi, One surprise language (Farsi, BahasaBahasaMalay)Malay)Other evaluations with military groupsOther evaluations with military groupsDoes it work??Yes, mostlyYes, mostly27 concepts out of 3027 concepts out of 30--ish turnsish turnsSystems are mostly similarSystems are mostly similarBut some better than othersBut some better than othersOther techniquesOther techniquesBelt/holster based PC with handheld speakerBelt/holster based PC with handheld speakerSmall PC in pouchSmall PC in pouchChest mounted array microphoneChest mounted array microphoneS2S ASR Advanced issuesTight