Recognizing Metadata Segmentation and Disfluencies Julia Hirschberg CS 4706 01 14 19 1 DARPA EARS Program Effective Affordable Reusable Speech to Text Goals Produce transcriptions that are more readable and usable by humans and downstream processes Segment speech into text like units Sentences speaker turns topics Add appropriate punctuation Eliminate spontaneous speech phenomena Filled pauses Self repairs Discourse markers Tasks ASR Metadata Extraction MDE 01 14 19 2 Motivation ASR Transcription aides tonight in boston in depth the truth squad for special series until election day tonight the truth about the budget surplus of the candidates are promising the two international flash points getting worse while the middle east and a new power play by milosevic and a lifelong a family tries to say one child life by having another amazing breakthrough the u s was was told local own boss good evening uh from the university of massachusetts in boston the site of the widely anticipated first of eight between vice president al gore and governor george w bush with the election now just five weeks away this is the beginning of a sprint to the finish and a strong start here tonight is important this is the stage for the two candidates will appear before a national television audience taking questions from jim lehrer of p b s n b c s david gregory is here with governor bush claire shipman is covering the vice president claire you begin tonight please 01 14 19 3 Motivation Speaker Segmentation Diarization Speaker 0 aides tonight in boston in depth the truth squad for special series until election day tonight the truth about the budget surplus of the candidates are promising the two international flash points getting worse while the middle east and a new power play by milosevic and a lifelong a family tries to say one child life by having another amazing breakthrough the u s was was told local own boss good evening uh from the university of massachusetts in boston Speaker 1 the site of the widely anticipated first of eight between vice president al gore and governor george w bush with the election now just five weeks away this is the beginning of a sprint to the finish and a strong start here tonight is important this is the stage for the two candidates will appear before a national television audience taking questions from jim lehrer of p b s n b c s david gregory is here with governor bush claire shipman is covering the vice president claire you begin tonight please 01 14 19 4 Motivation Sentence Detection Punctuation Truecasing Speaker Anchor Aides tonight in Boston In depth the truth squad for special series until election day Tonight the truth about the budget surplus of the candidates are promising The two international flash points getting worse While the Middle East And a new power play by Milosevic and a lifelong a family tries to say one child life by having another amazing breakthrough the U S was was told local own boss Good evening uh from the university of Massachusetts in Boston Speaker Reporter The site of the widely anticipated first of eight between Vice President Al Gore and Governor George W Bush With the election now just five weeks away This is the beginning of a sprint to the finish And a strong start here tonight is important This is the stage for the two candidates will appear before a national television audience taking questions from Jim Lehrer of PBS NBC s David Gregory is here with Governor Bush Claire Shipman is covering the vice president Claire you begin tonight please 01 14 19 5 Story Boundary Topic Detection Speaker Anchor Aides tonight in Boston In depth the truth squad for special series until election day Tonight the truth about the budget surplus of the candidates are promising The two international flash points getting worse While the Middle East And a new power play by Milosevic and a lifelong a family tries to say one child life by having another amazing breakthrough the U S was was told local own boss Good evening uh from the university of Massachusetts in Boston Speaker Reporter The site of the widely anticipated first of eight between Vice President Al Gore and Governor George W Bush With the election now just five weeks away This is the beginning of a sprint to the finish And a strong start here tonight is important This is the stage for the two candidates will appear before a national television audience taking questions from Jim Lehrer of PBS NBC s David Gregory is here with Governor Bush Claire Shipman is covering the vice president Claire you begin tonight please 01 14 19 6 Today Segmentation Speakers Sentences Stories Disfluency detection correction Self Repairs 01 14 19 7 Sentence Segmentation Classification task sentence boundary vs no sentence boundary Features Lexical and POS information but ASR is noisy Distance from previous hypothesized boundary Speech information Durations sentence final words are longer Pause F0 f0 modeling pitch reset pitch range Hand annotated training corpus annotated for SLUs Sentence like Units 51pp LDC Manual 01 14 19 9 Punctuation Detection and Truecasing Punctuation Assign each SLU an appropriate final punctuation SLU internal punctuation Capitalization Capitalize words beginning sentences Named Entities how train Features Prosodic and lexical Training Data 01 14 19 10 Topic Story Boundary Detection Rich text based literature Halliday Hasan 1976 lexical cohesion Hearst 1997 TextTiling segments by comparing words before and after each hypothesized topic boundary wrt a word similarity metric Reynar 1999 Beeferman et al 1999 cue phrases Choi 2000 divisive clustering using cosine sim on stems Features used Stem repetition entity repetition word frequency context vectors semantic similarity word distance lexical chains anaphoric chains 01 14 19 11 Spoken Cues to Discourse Topic Structure Pitch range Lehiste 75 Brown et al 83 Silverman 86 Avesani Vayra 88 Ayers 92 Swerts et al 92 Grosz Hirschberg 92 Swerts Ostendorf 95 Hirschberg Nakatani 96 Preceding pause Lehiste 79 Chafe 80 Brown et al 83 Silverman 86 Woodbury 87 Avesani Vayra 88 Grosz Hirschberg 92 Passoneau Litman 93 Hirschberg Nakatani 96 01 14 19 12 Rate Butterworth 75 Lehiste 80 Grosz Hirschberg 92 Hirschberg Nakatani 96 Amplitude Brown et al 83 Grosz Hirschberg 92 Hirschberg Nakatani 96 Contour Brown et al 83 Woodbury 87 Swerts et al 92 01 14 19 13 Finding Sentence and Topic Story Boundaries in ASR Transcripts Shriberg et al 2000 Text based segmentation is