CSCI 5832 Natural Language Processing Jim Martin Lecture 7 01 13 19 1 Today 2 5 Review LM basics Chain rule Markov Assumptions Why should you care Remaining issues Unknown words Evaluation Smoothing Backoff and Interpolation 2 01 13 19 Language Modeling We want to compute P w1 w2 w3 w4 w5 wn the probability of a sequence Alternatively we want to compute P w5 w1 w2 w3 w4 w5 the probability of a word given some previous words The model that computes P W or P wn w1 w2 wn 1 is called the language model 3 01 13 19 Computing P W How to compute this joint probability P the other day I was walking along and saw a lizard Intuition let s rely on the Chain Rule of Probability 4 01 13 19 The Chain Rule Recall the definition of conditional probabilities P A B P A B Rewriting P B P A B P A B P B More generally P A B C D P A P B A P C A B P D A B C In general P x1 x2 x3 xn P x1 P x2 x1 P x3 x1 x2 P xn x1 xn 1 5 01 13 19 The Chain Rule P the big red dog was P the P big the P red the big P dog the big red P was the big red dog 6 01 13 19 Very Easy Estimate How to estimate P the its water is so transparent that P the its water is so transparent that Count its water is so transparent that the Count its water is so transparent that 7 01 13 19 Very Easy Estimate According to Google those counts are 5 9 Unfortunately 2 of those are to these slides So its really 3 7 8 01 13 19 Unfortunately There are a lot of possible sentences In general we ll never be able to get enough data to compute the statistics for those long prefixes P lizard the other day I was walking along and saw a 9 01 13 19 Markov Assumption Make the simplifying assumption P lizard the other day I was walking along and saw a P lizard a Or maybe P lizard the other day I was walking along and saw a P lizard saw a Or maybe You get the idea 10 01 13 19 Markov Assumption So for each component in the product replace with the approximation assuming a prefix of N n 1 1 P wn w P wn w n 1 n N 1 Bigram version n 1 1 P w n w P w n w n 1 11 01 13 19 Estimating bigram probabilities The Maximum Likelihood Estimate count w i 1 w i P w i w i 1 count w i 1 c w i 1 w i P w i w i 1 c w i 1 12 01 13 19 An example s I am Sam s s Sam I am s s I do not like green eggs and ham s 13 01 13 19 Maximum Likelihood Estimates The maximum likelihood estimate of some parameter of a model M from a training set T Is the estimate that maximizes the likelihood of the training set T given the model M Suppose the word Chinese occurs 400 times in a corpus of a million words Brown corpus What is the probability that a random word from some other text from the same distribution will be Chinese MLE estimate is 400 1000000 004 This may be a bad estimate for some other corpus But it is the estimate that makes it most likely that Chinese will occur 400 times in a million word corpus 14 01 13 19 Berkeley Restaurant Project Sentences can you tell me about any good cantonese restaurants close by mid priced thai food is what i m looking for tell me about chez panisse can you give me a listing of the kinds of food that are available i m looking for a good place to eat breakfast when is caffe venezia open during the day 15 01 13 19 Raw Bigram Counts Out of 9222 sentences Count col row 16 01 13 19 Raw Bigram Probabilities Normalize by unigrams Result 17 01 13 19 Bigram Estimates of Sentence Probabilities P s I want english food s p i s x p want I x p english want x p food english x p s food 000031 18 01 13 19 Kinds of knowledge P english want 0011 P chinese want 0065 P to want 66 P eat to 28 P food to 0 P want spend 0 P i s 25 World knowledge Syntax Discourse 19 01 13 19 The Shannon Visualization Method Generate random sentences Choose a random bigram s w according to its probability Now choose a random bigram w x according to its probability And so on until we choose s Then string the words together s I I want want to to eat eat Chinese Chinese food food s 20 01 13 19 Shakespeare 21 01 13 19 Shakespeare as corpus N 884 647 tokens V 29 066 Shakespeare produced 300 000 bigram types out of V2 844 million possible bigrams so 99 96 of the possible bigrams were never seen have zero entries in the table Quadrigrams worse What s coming out looks like Shakespeare because it is Shakespeare 22 01 13 19 The Wall Street Journal is Not Shakespeare 23 01 13 19 Why Why would anyone want the probability of a sequence of words Typically because of P X S P S P S X P X 24 01 13 19 Unknown words Open versus closed vocabulary tasks If we know all the words in advanced Vocabulary V is fixed Closed vocabulary task Often we don t know this Out Of Vocabulary OOV words Open vocabulary task Instead create an unknown word token UNK Training of UNK probabilities Create a fixed lexicon L of size V At text normalization phase any training word not in L changed to UNK Now we train its probabilities like a normal word At decoding time If text input Use UNK probabilities for any word not in training 25 01 13 19 Evaluation We train parameters of our model on a training set How do we evaluate how well our model works We look at the models performance on some new data This is what happens in the real world we want to know how our model performs on data we haven t seen So a test set A dataset which is different than our training set 26 01 13 19 Evaluating N gram models Best evaluation for an N gram Put model A in a speech recognizer Run recognition get word error rate WER for A Put model B in speech recognition get word error rate for B Compare WER for A and B Extrinsic evaluation 27 01 13 19 Difficulty of extrinsic in vivo evaluation of N gram models Extrinsic evaluation This is really time consuming Can take days to run an experiment So As a …
View Full Document
Unlocking...