New version page

MIT 6 863J - Finite state machines & part-of-speech tagging

Documents in this Course
N-grams

N-grams

42 pages

Semantics

Semantics

75 pages

Semantics

Semantics

82 pages

Semantics

Semantics

64 pages

Load more
Upgrade to remove ads

This preview shows page 1-2-3-4-5-6-7-8-9-10-11-81-82-83-84-85-86-87-88-89-90-91-92-162-163-164-165-166-167-168-169-170-171-172 out of 172 pages.

Save
View Full Document
Premium Document
Do you want full access? Go Premium and unlock all 172 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 172 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 172 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 172 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 172 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 172 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 172 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 172 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 172 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 172 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 172 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 172 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 172 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 172 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 172 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 172 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 172 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 172 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 172 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 172 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 172 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 172 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 172 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 172 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 172 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 172 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 172 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 172 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 172 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 172 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 172 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 172 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 172 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 172 pages.
Access to all documents
Download any document
Ad free experience

Upgrade to remove ads
Unformatted text preview:

6.863J Natural Language ProcessingLecture 5: Finite state machines & part-of-speech taggingInstructor: Robert C. BerwickThe Menu Bar• Administrivia:• Schedule alert: Lab1 due next Weds(Feb24)• Lab 2, handed out Feb 24 (look for it on the web as laboratory2.html; due the Weds after this – March 5•Agenda:• Part of speech ‘tagging’ (with sneaky intro to probability theory that we need)• Ch. 6 & 8 in Jurafsky; see ch. 5 on Hidden Markov modelsTwo finite-state approaches to tagging1. Noisy Channel Model (statistical)2. Deterministic baseline tagger composedwith a cascade of fixup transducers• PS: how do we evaluatetaggers? (and such statistical models generally?) • 1, 2, & evaluation = Laboratory 2The real plan…p(X)p(Y | X)p(X, y)*=*p(y | Y)Find x that maximizes this quantityCartoon versionp(X)p(Y | X)p(X, y)*==* *p(y | Y)transducer: scores candidate tag seqson their joint probability with obs words;we should pick best paththe cool directed autosAdj:cortege/0.000001…Noun:Bill/0.002Noun:autos/0.001…Noun:cortege/0.000001Adj:cool/0.003Adj:directed/0.0005Det:the/0.4Det:a/0.6DetStartAdjNounVerbPrepStopNoun0.7Adj 0.3Adj 0.4ε 0.1Noun0.5Det 0.8ε 0.2*What’s the big picture? Why NLP?Computers would be a lot more useful if they could handle our email, do our library research, talk to us …But they are fazed by natural human language.How can we tell computers about language?(Or help them learn it as kids do?)What is NLP for, anyway?• If we could do it perfectly, we could pass the Turing test (more on this below)• Two basic ‘engineering’ tasks – and third scientific one• Text-understanding• Information extraction• ?What about how people ‘process’ language??? [psycholinguistics]Some applications…• Spelling correction, grammar checking …• Better search engines• Information extraction• Language identification (English vs. Polish)• Psychotherapy; Harlequin romances; etc.• And: plagiarism detection - www.turnitin.com• For code: www.cs.berkeley.edu/~aiken/moss.html• New interfaces:• Speech recognition (and text-to-speech)• Dialogue systems (USS Enterprise onboard computer)• Machine translation (the Babel fish)Text understanding is very hardJohn stopped at the donut store on his way home from work. He thought a coffee was good every few hours. But it turned out to be too expensive there.• NLrelies onambiguity! (Why?)• “We haven’t had a sale in 40 years”What’s hard about the story?John stopped at the donut store on his way home from work. He thought a coffee was good every few hours. But it turned out to be too expensive there.To get a donut (spare tire) for his car?What’s hard?John stopped at the donut store on his way home from work. He thought a coffee was good every few hours. But it turned out to be too expensive there.store where donuts shop? or is run by donuts?or looks like a big donut? or made of donut?or has an emptiness at its core?(Think of five other issues…there are lots)What’s hard about this story?John stopped at the donut store on his way home from work. He thought a coffee was good every few hours. But it turned out to be too expensive there.Describes where the store is? Or when he stopped?What’s hard about this story?John stopped at the donut store on his way home from work. He thought a coffee was good every few hours. But it turned out to be too expensive there.Well, actually, he stopped there from hunger and exhaustion, not just from work.What’s hard about this story?John stopped at the donut store on his way home from work. He thought a coffee was good every few hours. But it turned out to be too expensive there.At that moment, or habitually?(Similarly:Mozart composed music.)What’s hard about this story?John stopped at the donut store on his way home from work. He thought a coffee was good every few hours. But it turned out to be too expensive there.That’s how often he thought it?What’s hard about this story?John stopped at the donut store on his way home from work. He thought a coffee was good every few hours. But it turned out to be too expensive there.But actually, a coffee only stays good for about 10 minutes before it gets cold.What’s hard about this story?John stopped at the donut store on his way home from work. He thought acoffee was good every few hours. But it turned out to be too expensive there.Similarly:In America a woman has a baby every 15 minutes. Our job is to find that woman and stop her.What’s hard about this story?John stopped at the donut store on his way home from work. He thought a coffee was good every few hours. But it turnedout to be too expensive there.the particular coffee that was good every few hours? the donut store? the situation?What’s hard about this story?John stopped at the donut store on his way home from work. He thought a coffee was good every few hours. But it turned out to be too expensive there.too expensive for what? what are we supposed to conclude about what John did?how do we connect “it” to “expensive”?Example tagsets• 87 tags - Brown corpus• Three most commonly used:1. Small: 45 Tags - Penn treebank (Mediumsize: 61 tags, British national corpus2. Large: 146 tagsBig question: have we thrown out the right info? Impoverished? How?Current performance• How many tags are correct?• About 97% currently• But baseline is already 90%• Baseline is performance Homer Simpson algorithm:• Tag every word with its most frequent tag• Tag unknown words as nouns• How well do people do?Input: the lead paint is unsafeOutput: the/Det lead/N paint/N is/V unsafe/AdjKnowldege-based (rule-based)vs. Statistically-based systemsA picture: the statistical, noisy channel view x(speech)Wreck a nice beach?Reckon eyes peach?Recognize speech?AcousticModelP(x|y)LanguageModelP(y)y(text)Language models, probability & info• Given a string w,a language model gives us the probability of the string P(w), e.g.,•P(the big dog) > (dog big the) > (dgo gib eth)• Easy for humans; difficult for machines• Let P(w) be called a language modelLanguage models –statisticalview• Application to speech recognition (and parsing, generally)• x= Input (speech)• y= output (text)• We want to find max P(y|x) Problem: we don’t know this!• Solution: We have an estimate of P(y) [the language model] and P(x|y) [the prob. of some sound given text = an acoustic model ]• From Bayes’ law, we have, max P(y|x) = max


View Full Document
Download Finite state machines & part-of-speech tagging
Our administrator received your request to download this document. We will send you the file to your email shortly.
Loading Unlocking...
Login

Join to view Finite state machines & part-of-speech tagging and access 3M+ class-specific study document.

or
We will never post anything without your permission.
Don't have an account?
Sign Up

Join to view Finite state machines & part-of-speech tagging 2 2 and access 3M+ class-specific study document.

or

By creating an account you agree to our Privacy Policy and Terms Of Use

Already a member?