CSCI 5832 Natural Language Processing Jim Martin Lecture 11 01 14 19 1 Today 2 21 Review HMMs EM Example Syntax Context Free Grammars 2 01 14 19 Review Parts of Speech Basic syntactic morphological categories that words belong to Part of Speech tagging Assigning parts of speech to all the words in a sentence 3 01 14 19 Probabilities We want the best set of tags for a sequence of words a sentence W is a sequence of words T is a sequence of tags arg max P T W P W T P T 4 01 14 19 So We start with arg max P T W P W T P T And get n n i 2 i 2 arg max P wi ti P t1 P ti ti 1 5 01 14 19 HMMs This is an HMM n n i 2 i 2 arg max P wi ti P t1 P ti ti 1 The states in the model are the tags and the observations are the words The state to state transitions are driven by the bigram statistics The observed words are based solely on the state that you re currently in 6 01 14 19 State Transitions Noun Verb Det Aux 0 5 7 01 14 19 State Transitions and Observations bark dog bark cat run Noun bite the a Verb Det that Aux can will 0 5 did 8 01 14 19 The State Space Det Det Det Det Noun Noun Noun Noun s s 01 14 19 Aux Aux Aux Aux Verb Verb Verb Verb The dog can run 9 The State Space Det Det Det Det Noun Noun Noun Noun s s 01 14 19 Aux Aux Aux Aux Verb Verb Verb Verb The dog can run 10 The State Space Det Det Det Det Noun Noun Noun Noun s s 01 14 19 Aux Aux Aux Aux Verb Verb Verb Verb The dog can run 11 Viterbi Efficiently return the most likely path Sweep through the columns multiplying the probabilities of one row times the transition probabilities to the next row times the appropriate observation probabilities And store the MAX 12 01 14 19 Forward Efficiently computes the probability of an observed sequence given a model P sequence model Nearly identical to Viterbi replace the MAX with a SUM There is one complication there if you think about the logs that we ve been using 13 01 14 19 EM Forward Backward Efficiently arrive at the right model parameters given a model structure and an observed sequence So for POS tagging Given a tag set And an observed sequence Fill the A B and PI tables with the right numbers Numbers that give a model that Argmax P model data 14 01 14 19 Urn Example A genie has two urns filled with red and blue balls The genie selects an urn and then draws a ball from it and replaces it The genie then selects either the same urn or the other one and then selects another ball The urns are hidden The balls are observed 15 01 14 19 Urn Based on the results of a long series of draws Figure out the distribution of colors of balls in each urn Figure out the genie s preferences in going from one urn to the next 16 01 14 19 Urns and Balls Pi Urn 1 0 9 Urn 2 0 1 Urn 1 Urn 2 A Urn 1 0 6 0 4 Urn 2 B Red Blue 0 3 0 7 Urn 1 Urn 2 0 7 0 3 0 4 0 6 17 01 14 19 Urns and Balls Let s assume the input observables is Blue Blue Red BBR Since both urns contain 6 7 4 red and blue balls any path through Urn 1 Urn 2 this machine 3 could produce this output 18 01 14 19 Urns and Balls Blue Blue Red 111 112 121 122 0 9 0 3 0 6 0 3 0 6 0 7 0 0204 0 9 0 3 0 6 0 3 0 4 0 4 0 0077 0 9 0 3 0 4 0 6 0 3 0 7 0 0136 0 9 0 3 0 4 0 6 0 7 0 4 0 0181 211 212 221 222 0 1 0 6 0 3 0 7 0 6 0 7 0 0052 0 1 0 6 0 3 0 7 0 4 0 4 0 0020 0 1 0 6 0 7 0 6 0 3 0 7 0 0052 0 1 0 6 0 7 0 6 0 7 0 4 0 0070 19 01 14 19 Urns and Balls terbi Says 111 is the most likely state sequence 111 112 121 122 0 9 0 3 0 6 0 3 0 6 0 7 0 0204 0 9 0 3 0 6 0 3 0 4 0 4 0 0077 0 9 0 3 0 4 0 6 0 3 0 7 0 0136 0 9 0 3 0 4 0 6 0 7 0 4 0 0181 211 212 221 222 0 1 0 6 0 3 0 7 0 6 0 7 0 0052 0 1 0 6 0 3 0 7 0 4 0 4 0 0020 0 1 0 6 0 7 0 6 0 3 0 7 0 0052 0 1 0 6 0 7 0 6 0 7 0 4 0 0070 20 01 14 19 Urns and Balls Forward P BBR model 0792 111 112 121 122 0 9 0 3 0 6 0 3 0 6 0 7 0 0204 0 9 0 3 0 6 0 3 0 4 0 4 0 0077 0 9 0 3 0 4 0 6 0 3 0 7 0 0136 0 9 0 3 0 4 0 6 0 7 0 4 0 0181 211 212 221 222 0 1 0 6 0 3 0 7 0 6 0 7 0 0052 0 1 0 6 0 3 0 7 0 4 0 4 0 0020 0 1 0 6 0 7 0 6 0 3 0 7 0 0052 0 1 0 6 0 7 0 6 0 7 0 4 0 0070 21 01 14 19 Urns and Balls EM What if I told you I lied about the numbers in the model Priors A B I just made them up Can I get better numbers just from the input sequence 22 01 14 19 Urns and Balls Yup Just count up and prorate the number of times a given transition is traversed while processing the observations inputs Then use that count to re estimate the transition probability for that transition 23 01 14 19 Urns and Balls But we just saw that don t know the actual path the input took its hidden So prorate the counts from all the possible paths based on the path probabilities the model gives you But you said the numbers were wrong Doesn t matter use the original numbers then replace the old ones with the new ones 24 01 14 19 Urn Example 6 7 4 Urn …
View Full Document
Unlocking...