CSCI 5832 Natural Language Processing Jim Martin Lecture 6 01 13 19 1 Today 1 31 Probability Basic probability Conditional probability Bayes Rule Language Modeling N grams N gram Intro The Chain Rule Smoothing Add 1 2 01 13 19 Probability Basics Experiment trial Repeatable procedure with well defined possible outcomes Sample Space S the set of all possible outcomes finite or infinite Example QuickTime and a TIFF Uncompressed decompressor are needed to see this picture coin toss experiment possible outcomes S heads tails Example QuickTime and a TIFF Uncompressed decompressor are needed to see this picture die toss experiment possible outcomes S 1 2 3 4 5 6 01 13 19 Slides from Sandiway Fong 3 Probability Basics Definition of sample space depends on what we are asking Sample Space S the set of all possible outcomes Example die toss experiment for whether the number is even or odd possible outcomes even odd not 1 2 3 4 5 6 QuickTime and a TIFF Uncompressed decompressor are needed to see this picture 4 01 13 19 More Definitions Events an event is any subset of outcomes from the sample space Example Die toss experiment Let A represent the event such that the outcome of the die toss experiment is divisible by 3 A 3 6 A is a subset of the sample space S 1 2 3 4 5 6 Example Draw a card from a deck suppose sample space S heart spade club diamond four suits let A represent the event of drawing a heart let B represent the event of drawing a red card A heart B heart diamond QuickTime and a TIFF decompress are Uncompressed needed to see this picture 5 01 13 19 Probability Basics Some definitions Counting suppose operation oi can be performed in ni ways then a sequence of k operations o1o2 ok can be performed in n1 n2 nk ways Example die toss experiment 6 possible outcomes two dice are thrown at the same time number of sample points in sample space 6 6 36 QuickTime and a TIFF Uncompressed decompressor are needed to see this picture 6 01 13 19 Definition of Probability The probability law assigns to an event a number between 0 and 1 called P A Also called the probability of A This encodes our knowledge or belief about the collective likelihood of all the elements of A Probability law must satisfy certain properties 7 01 13 19 Probability Axioms Nonnegativity P A 0 for every event A Additivity If A and B are two disjoint events then the probability of their union satisfies P A U B P A P B Normalization The probability of the entire sample space S is equal to 1 I e P S 1 8 01 13 19 An example An experiment involving a single coin toss There are two possible outcomes H and T Sample space S is H T If coin is fair should assign equal probabilities to 2 outcomes Since they have to sum to 1 P H 0 5 P T 0 5 P H T P H P T 1 0 9 01 13 19 Another example Experiment involving 3 coin tosses Outcome is a 3 long string of H or T S HHH HHT HTH HTT THH THT TTH TTTT Assume each outcome is equiprobable Uniform distribution What is probability of the event that exactly 2 heads occur A HHT HTH THH P A P HHT P HTH P THH 1 8 1 8 1 8 3 8 10 01 13 19 Probability definitions In summary Probability of drawing a spade from 52 well shuffled playing cards 13 1 25 52 4 11 01 13 19 Probabilities of two events If two events A and B are independent then P A and B P A x P B If we flip a fair coin twice What is the probability that they are both heads If draw a card from a deck then put it back draw a card from the deck again What is the probability that both drawn cards are hearts 12 01 13 19 How about non uniform probabilities A biased coin twice as likely to come up tails as heads is tossed twice What is the probability that at least one head occurs Sample space hh ht th tt Sample points probability for the event ht 1 3 x 2 3 2 9 th 2 3 x 1 3 2 9 hh 1 3 x 1 3 1 9 tt 2 3 x 2 3 4 9 Answer 5 9 0 56 sum of weights in red 13 01 13 19 Moving toward language What s the probability of drawing a 2 from a deck of 52 cards with four 2s 4 1 P drawing a two 077 52 13 What s the probability of a random word from a random dictionary page being a verb of ways to get a verb P drawing a verb 01 13 19 all words 14 Probability and part of speech tags What s the probability of a random word from a random dictionary page being a verb P drawing a verb of ways to get a verb all words How to compute each of these All words just count all the words in the dictionary of ways to get a verb number of words which are verbs If a dictionary has 50 000 entries and 10 000 are verbs P V is 10000 50000 1 5 20 15 01 13 19 Conditional Probability A way to reason about the outcome of an experiment based on partial information In a word guessing game the first letter for the word is a t What is the likelihood that the second letter is an h How likely is it that a person has a disease given that a medical test was negative A spot shows up on a radar screen How likely is it that it corresponds to an aircraft 16 01 13 19 More precisely Given an experiment a corresponding sample space S and a probability law Suppose we know that the outcome is within some given event B We want to quantify the likelihood that the outcome also belongs to some other given event A We need a new probability law that gives us the conditional probability of A given B P A B 17 01 13 19 An intuition A is it s snowing now P A in normally arid Colorado is 01 B is it was snowing ten minutes ago P A B means what is the probability of it snowing now if it was snowing 10 minutes ago P A B is probably way higher than P A Perhaps P A B is 10 Intuition The knowledge about B should change update our estimate of the probability of A 18 01 13 19 Conditional probability One of the following 30 items is chosen at random What is P X the probability that it is an X What is P X red the probability that it is an X given that it is red 19 01 13 19 Conditional Probability let A and B be …
View Full Document
Unlocking...