Text Normalization and Feature Extraction Julia Hirschberg CS 4706 01 13 19 1 ScanSoft Nuance demo AT T demo Cepstral 01 13 19 2 Text Normalization 1 A sworn deposition that Sen John McCain gave in a lawsuit more than 5 years ago appears to contradict one part of a sweeping denial that his campaign issued this week to rebut a New York Times story about his ties to a Washington lobbyist On Wednesday night the Times published a story suggesting that McCain might have done legislative favors for the clients of the lobbyist Vicki Iseman who worked for the firm of Alcalde Fay One example it cited were two letters McCain wrote in late 1999 demanding that the Federal Communications Commission act on a long stalled bid by one of Iseman s clients Florida based Paxson Communications to purchase a Pittsburgh TV station Just hours after the Times s story was posted the McCain campaign issued a point by point response that depicted the letters as routine correspondence handled by his staff and insisted that McCain had never even spoken with anybody from Paxson or Alcalde Fay about the matter No representative of Paxson or Alcalde Fay personally asked Senator McCain to send a letter to the FCC the campaign said in a statement e mailed to reporters But that flat claim seems to be contradicted by an impeccable source McCain himself I was contacted by Mr Lowell Paxson on this issue McCain said in the Sept 25 2002 deposition obtained by NEWSWEEK He wanted their approval very bad for purposes of his business I believe that Mr Paxson had a legitimate complaint While McCain said I don t recall if he ever directly spoke to the firm s lobbyist about the issue an apparent reference to Iseman though she is not named I m sure I spoke to Paxson McCain agreed that his letters on behalf of Paxson a campaign contributor could possibly be an appearance of corruption even though McCain denied doing anything improper McCain s subsequent letters to the FCC coming around the same time that Paxson s firm was flying the senator to campaign events aboard its corporate jet and contributing 20 000 to his campaign first surfaced as an issue during his unsuccessful 2000 presidential bid William Kennard the FCC chair at the time described the sharply worded letters from McCain then chairman of the Senate Commerce Committee as highly unusual 01 13 19 3 Text Normalization 2 Dr Julia Hirschberg Dept of Computer Science 450 CS Bldg M C 0401 1214 Amsterdam Ave New York NY 10027 julia cs columbia edu Tel 212 939 7114 Fax 212 666 0140 http www cs columbia edu julia 01 13 19 4 Today Segmentation Tokenization Abbreviations Numbers Extracting features for downstream processing Pronunciation Intonation assignment TTS markup Concept to Speech 01 13 19 5 Segmentation What is a sentence 01 13 19 A sworn deposition that Sen John McCain gave in a lawsuit more than 5 years ago appears to contradict one part of a sweeping denial that his campaign issued this week to rebut a New York Times story about his ties to a Washington lobbyist On Wednesday night the Times published a story suggesting that McCain might have done legislative favors for the clients of the lobbyist Vicki Iseman who worked for the firm of Alcalde Fay One example it cited were two letters McCain wrote in late 1999 demanding that the Federal Communications Commission act on a long stalled bid by one of Iseman s clients Florida based Paxson Communications to purchase a Pittsburgh TV station Just hours after the Times s story was posted the McCain campaign issued a point by point response that depicted the letters as routine correspondence handled by his staff and insisted that McCain had never even spoken with anybody from Paxson or Alcalde Fay about the matter No representative of Paxson or Alcalde Fay personally asked Senator McCain to send a letter to the FCC the campaign said in a statement e mailed to reporters But that flat claim seems to be contradicted by an impeccable source McCain himself I was contacted by Mr Lowell Paxson on this issue McCain said in the Sept 25 2002 deposition obtained by NEWSWEEK He wanted their approval very bad for purposes of his business I believe that Mr Paxson had a legitimate complaint While McCain said I don t recall if he ever directly spoke to the firm s lobbyist about the issue an apparent reference to Iseman though she is not named I m sure I spoke to Paxson McCain agreed that his letters on behalf of Paxson a campaign contributor could possibly be an appearance of corruption even though McCain denied doing anything improper McCain s subsequent letters to the FCC coming around the same time that Paxson s firm was flying the senator to campaign events aboard its corporate jet and contributing 20 000 to his campaign first surfaced as an issue during his unsuccessful 2000 presidential bid William Kennard the FCC chair at the time described the sharply worded letters from McCain then chairman of the Senate Commerce Committee as highly unusual 6 Rule based approaches If the preceding word is an abbreviation e g Mr or Mrs or Dr or Sen or not sentence boundary How collect all such abbreviations What if an abbreviation ends a sentence He works for Cisco Inc Machine learning approaches Need labeled data usually 01 13 19 7 Create feature vectors for each potential sentence boundary with potential predictors How long is preceding word Is preceding word capitalized Is succeeding word capitalized Discover which feature combinations best predict observed values in training data Test on held out data Hybrid approaches Combine rules for easy decisions with ML Use rules to label initial corpus Add rules to ML results 01 13 19 8 Tokenization What is a word On Wednesday night the Times published a story suggesting that McCain might have done legislative favors for the clients of the lobbyist Vicki Iseman who worked for the firm of Alcalde Fay One example it cited were two letters McCain wrote in late 1999 demanding that the Federal Communications Commission act on a long stalled bid by one of Iseman s clients Florida based Paxson Communications to purchase a Pittsburgh TV station Just hours after the Times s story was posted the McCain campaign issued a point by point response that depicted the letters as routine correspondence handled by his staff and insisted that McCain had never even spoken with anybody from Paxson or Alcalde Fay about the matter No representative of Paxson or Alcalde Fay personally asked Senator McCain to send a letter to the