Data Mining Information Extraction and Search in Spoken Documents Julia Hirschberg CS 4706 01 14 19 1 Today Data mining from text Searching audio data instead of text Information extraction from spoken documents Speech data mining 01 14 19 2 Data Mining Discovery of trends and patterns across very large datasets usually for decision making purposes Fraud detection in banking telephony Stock market Indications of demographic disasters New causes of diseases finding things you don t know you re looking for Information retrieval vs mining for nuggets 01 14 19 3 Dating Mining in Computational Linguistics Finding lexical co occurrence information Finding parallel text corpora on the web for MT Finding new topics in news stories TDT task Exploring citation links Networks of influence Information extraction e g find mutual acquaintances 01 14 19 4 Snowball Agichtein et al 01 Seed set of patterns e g Norman Mailer 59 firstname lastname age the 59year old Mailer the age year old lastname Find more patterns by looking for e g Mailer close to 59 Mailer turned 59 last week Though Mailer is 59 01 14 19 5 But Searching Audio Data is Harder Large amounts of audio data available on the web in company archives in our homes We have tools supporting random access to text but for audio we re limited to serial search How can we develop methods to search audio as easily as text 01 14 19 6 Applications Searching online TV and radio news and archives Library of Congress Searching a v archives movies Searching trial recordings and legislative sessions Searching meetings customer care exchanges focus groups Telephone calls and voicemail 01 14 19 7 Current Approach Train adapt a speech recognizer for the corpus Produce an ASR transcript Segment spoken documents into sentences turns topics Index errorful transcripts for Information Retrieval and link to audio via timestamps Enables audio search by content 01 14 19 8 Some Examples SpeechBot searching internet broadcasts Google Voice Search search audio by voice not yet SCANMail searching voicemail 01 14 19 9 Information Extraction and QA from Speech DARPA GALE project improve information gathering from text speech translations Current Domain newswire and news broadcasts in English Arabic and Mandarin 3 competing teams ASR MT bakeoffs Distillation evaluations QA User studies Requires identification and annotation of information and formatting in speech 01 14 19 10 Sample Distillation Questions List facts about event Find people who are mutual acquaintances of person1 and person2 Identify persons arrested from organization and give their name and role in that organization Produce a biography of person Provide information on organization Find statements made by or attributed to person about topic How did country react to event 01 14 19 11 Nightingale Architecture Automatic Annotation Distillation Speaker modeling Audio diarization Prosodic metadata MT ASR Source Language Prosodic analysis Information assimilation Linguistic structure Names Relations Target Language Punctuation Capitalization Info repository Intelligence delivery Topic modeling 01 14 19 12 Information Annotation Spoken documents Lack many cues found in text documents Format sentences turns paragraphs Include spontaneous speech phenomena which are difficult for ASR and NLP technologies to handle Disfluencies fragments Contain errors Annotation can turn a weakness into a strength 01 14 19 13 From an ASR Transcript aides tonight in boston in depth the truth squad for special series until election day tonight the truth about the budget surplus of the candidates are promising the two international flash points getting worse while the middle east and a new power play by milosevic and a lifelong a family tries to say one child life by having another amazing breakthrough the u s was was told local own boss good evening uh from the university of massachusetts in boston the site of the widely anticipated first of eight between vice president al gore and governor george w bush with the election now just five weeks away this is the beginning of a sprint to the finish and a strong start here tonight is important this is the stage for the two candidates will appear before a national television audience taking questions from jim lehrer of p b s n b c s david gregory is here with governor bush claire shipman is covering the vice president claire you begin tonight please 01 14 19 14 To Speaker Segmentation Diarization Speaker 0 aides tonight in boston in depth the truth squad for special series until election day tonight the truth about the budget surplus of the candidates are promising the two international flash points getting worse while the middle east and a new power play by milosevic and a lifelong a family tries to say one child life by having another amazing breakthrough the u s was was told local own boss good evening uh from the university of massachusetts in boston Speaker 1 the site of the widely anticipated first of eight between vice president al gore and governor george w bush with the election now just five weeks away this is the beginning of a sprint to the finish and a strong start here tonight is important this is the stage for the two candidates will appear before a national television audience taking questions from jim lehrer of p b s n b c s david gregory is here with governor bush claire shipman is covering the vice president claire you begin tonight please 01 14 19 15 Add Speaker Role Labels Anchor aides tonight in boston in depth the truth squad for special series until election day tonight the truth about the budget surplus of the candidates are promising the two international flash points getting worse while the middle east and a new power play by milosevic and a lifelong a family tries to say one child life by having another amazing breakthrough the u s was was told local own boss good evening uh from the university of massachusetts in boston Reporter the site of the widely anticipated first of eight between vice president al gore and governor george w bush with the election now just five weeks away this is the beginning of a sprint to the finish and a strong start here tonight is important this is the stage for the two candidates will appear before a national television audience taking questions from jim lehrer of p b s n b c s david gregory is here with governor bush claire shipman is covering the vice president claire you begin tonight please 01 14 19 16 Perform Sentence