Text Normalization Julia Hirschberg CS 4706 2 14 2011 1 TTS demos ScanSoft Nuance AT T IBM Cepstral SNL Robot Repair An interesting application for TTS 2 14 2011 2 Text Normalization 1 A sworn deposition that Sen John McCain gave in a lawsuit more than 5 years ago appears to contradict one part of a sweeping denial that his campaign issued this week to rebut a New York Times story about his ties to a Washington lobbyist On Wednesday night the Times published a story suggesting that McCain might have done legislative favors for the clients of the lobbyist Vicki Iseman who worked for the firm of Alcalde Fay One example it cited were two letters McCain wrote in late 1999 demanding that the Federal Communications Commission act on a long stalled bid by one of Iseman s clients Florida based Paxson Communications to purchase a Pittsburgh TV station Just hours after the Times s story was posted the McCain campaign issued a point by point response that depicted the letters as routine correspondence handled by his staff and insisted that McCain had never even spoken with anybody from Paxson or Alcalde Fay about the matter No representative of Paxson or Alcalde Fay personally asked Senator McCain to send a letter to the FCC the campaign said in a statement e mailed to reporters But that flat claim seems to be contradicted by an impeccable source McCain himself I was contacted by Mr Lowell Paxson on this issue McCain said in the Sept 25 2002 deposition obtained by NEWSWEEK He wanted their approval very bad for purposes of his business I believe that Mr Paxson had a legitimate complaint While McCain said I don t recall if he ever directly spoke to the firm s lobbyist about the issue an apparent reference to Iseman though she is not named I m sure I spoke to Paxson McCain agreed that his letters on behalf of Paxson a campaign contributor could possibly be an appearance of corruption even though McCain denied doing anything improper McCain s subsequent letters to the FCC coming around the same time that Paxson s firm was flying the senator to campaign events aboard its corporate jet and contributing 20 000 to his campaign first surfaced as an issue during his unsuccessful 2000 presidential bid William Kennard the FCC chair at the time described the sharply worded letters from McCain then chairman of the Senate Commerce Committee as highly unusual 2 14 2011 3 Text Normalization 2 Dr Julia Hirschberg Dept of Computer Science 450 CS Bldg M C 0401 1214 Amsterdam Ave New York NY 10027 julia cs columbia edu Tel 212 939 7114 Fax 212 666 0140 http www cs columbia edu julia 2 14 2011 4 Today Segmentation Tokenization Abbreviations Numbers TTS markup Concept to Speech 2 14 2011 5 Segmentation What is a sentence A sworn deposition that Sen John McCain gave in a lawsuit more than 5 years ago appears to contradict one part of a sweeping denial that his campaign issued this week to rebut a New York Times story about his ties to a Washington lobbyist On Wednesday night the Times published a story suggesting that McCain might have done legislative favors for the clients of the lobbyist Vicki Iseman who worked for the firm of Alcalde Fay One example it cited were two letters McCain wrote in late 1999 demanding that the Federal Communications Commission act on a long stalled bid by one of Iseman s clients Florida based Paxson Communications to purchase a Pittsburgh TV station Just hours after the Times s story was posted the McCain campaign issued a point by point response that depicted the letters as routine correspondence handled by his staff and insisted that McCain had never even spoken with anybody from Paxson or Alcalde Fay about the matter No representative of Paxson or Alcalde Fay personally asked Senator McCain to send a letter to the FCC the campaign said in a statement e mailed to reporters 2 14 2011 6 But that flat claim seems to be contradicted by an impeccable source McCain himself I was contacted by Mr Lowell Paxson on this issue McCain said in the Sept 25 2002 deposition obtained by NEWSWEEK He wanted their approval very bad for purposes of his business I believe that Mr Paxson had a legitimate complaint While McCain said I don t recall if he ever directly spoke to the firm s lobbyist about the issue an apparent reference to Iseman though she is not named I m sure I spoke to Paxson McCain agreed that his letters on behalf of Paxson a campaign contributor could possibly be an appearance of corruption even though McCain denied doing anything improper McCain s subsequent letters to the FCC coming around the same time that Paxson s firm was flying the senator to campaign events aboard its corporate jet and contributing 20 000 to his campaign first surfaced as an issue during his unsuccessful 2000 presidential bid William Kennard the FCC chair at the time described the sharply worded letters from McCain then chairman of the Senate Commerce Committee as highly unusual Rule based Approaches For a potential sentence ending word w followed by a If w is an abbreviation e g Mr or Mrs or Dr or Sen or w does not end the sentence O w w ends the sentence How do we know whether w is an abbreviation What if an abbreviation ends a sentence He works for Cisco Inc 2 14 2011 8 Machine Learning Approaches Labeled data Mechanical Turk What features best predict sentence boundaries Is preceding word a known abbreviation How long is preceding word Is preceding word capitalized Is succeeding word capitalized Create feature vectors for each potential boundary Apply ML algorithm to produce classifier Test on held out data 2 14 2011 9 Hybrid Approaches Combine rules for easy decisions with ML Use rules to label initial corpus and build classifier or Add rules directly to ML results Tokenization What is a word On Wednesday night the Times published a story suggesting that McCain might have done legislative favors for the clients of the lobbyist Vicki Iseman who worked for the firm of Alcalde Fay One example it cited were two letters McCain wrote in late 1999 demanding that the Federal Communications Commission act on a longstalled bid by one of Iseman s clients Florida based Paxson Communications to purchase a Pittsburgh TV station Just hours after the Times s story was posted the McCain campaign issued a point by point response that depicted the letters as routine correspondence handled by his staff and insisted that McCain had never even spoken with anybody from Paxson or Alcalde Fay about the matter No representative of