Topic Models Outline Review of directed models Independencies d separation and explaining away Learning for Bayes nets Directed models for text Na ve Bayes models Latent Dirichlet allocation LDA REVIEW OF DIRECTED MODELS AKA BAYES NETS Directed Model Graph Conditional Probability Distributions The Graph Some Pairwise Conditional Independencies Plate notation lets us denote complex graphs S P S s 1 0 t 0 0 u 0 0 v 0 0 Directed Models HMMs S1 S2 S a S S3 S4 P S S s s 0 1 s t 0 9 s u 0 0 a c a 0 9 c 0 1 a 0 5 s t s t 0 5 S 0 5 s X P X S a 0 9 s c 0 1 t a 0 6 t c 0 4 0 6 c 0 4 t 0 9 t a 0 5 0 1 0 8 v u 0 2 a 0 5 a 0 3 c 0 5 c 0 7 Directed Models HMMs S1 a S2 a S3 c S4 a 0 9 c 0 1 a s 0 1 Important point I can compute Pr S2 t aaca So inference does not always follow the arrows 0 5 a 0 6 c 0 4 t 0 9 0 5 1 u 0 8 v 0 2 a 0 5 a 0 3 c 0 5 c 0 7 SOME MORE DETAILS ON DIRECTED MODELS The example police say we re in violation Insufficient use of Monty Hall problem Discussing Bayes nets without discussing burglar alarms The Monty Hall problem highly practical You re in a game show Behind one door is a car Behind the others goats You pick one of three doors say 1 The host Monty Hall opens one door revealing a goat You now can either stick with your guess or change doors A P A 1 0 33 2 0 33 3 0 33 First guess The money A Stick or swap B C D B P B 1 0 33 2 0 33 3 0 33 The revealed goat D P D Stick 0 5 A B C P C A B Swa p 0 5 1 1 2 0 5 1 1 3 0 5 1 2 3 1 0 1 3 2 1 0 Second guess E 1 0 if a b c a b P C c A a B b 0 5 if a b c a b 0 otherwise Slide 10 A few minutes later the goat from behind door C drives away in the car problem A P A 1 0 33 2 0 33 3 0 33 First guess The money A B Stick or swap P E e A C D 1 0 if e a d stick 1 0 if e a c d swap 0 otherwise D Second guess A C C D P E A C D E B P B 1 0 33 2 0 33 3 0 33 The goat A B C P C A B 1 1 2 0 5 1 1 3 0 5 1 2 3 1 0 1 3 2 1 0 1 0 if a b c a b P C c A a B b 0 5 if a b c a b 0 otherwise Slide 12 The Monty Hall problem highly practical We could construct the joint and compute P E B D swap again by the chain rule A P A 1 0 33 2 0 33 3 0 33 First guess A C D P E A C D P D Second guess P C A B P A B Stick or swap P A B C D E P B The money A C D P E A C D E B P B 1 0 33 2 0 33 3 0 33 The goat A B C P C A B 1 1 2 0 5 1 1 3 0 5 1 2 3 1 0 1 3 2 1 0 Slide 13 The Monty Hall problem highly practical The joint table has 3 3 3 2 3 162 rows The conditional probability tables CPTs shown have 3 3 3 3 3 2 3 3 51 rows 162 rows A P A 1 0 33 2 0 33 3 0 33 First guess The money A B Stick or swap B P B 1 0 33 2 0 33 3 0 33 Big questions The goat C D why are the CPTs smaller A B C P C A B how much smaller are the 1 1 2 0 5 CPTs than the1 joint 1 3 0 5 Second guess E 1 2 the 3 1 0 can we compute 1 3 2 1 0 answers to queries like P E B d without building the joint probability tables just using the CPTs A C D P E A C D Slide 14 The Monty Hall problem highly practical Why is the CPTs representation smaller Follow the money B A P A 1 0 33 2 0 33 3 0 33 The money A B Stick or swap P E e A C D C D 1 0 if e a d stick 1 0 if e a c d swap 0 otherwise a b c d e First guess Second guess P E e A a C c D d A C D P E e A a B b C b D d P E A C D E B P B 1 0 33 2 0 33 3 0 33 The goat C P C A B EAis Bconditionally 1 1 2 0 5 independent of B 1 1 3 0 5 given A D C 1 2 3 1 0 1 3 2 1 0 E B A C D I E A C D B Slide 15 Conditional Independence again Definition R and L are conditionally independent given M if for all x y z in T F P R x M y L z P R x M y More generally Let S1 and S2 and S3 be sets of variables Set of variables S1 and set of variables S2 are conditionally independent given S3 if for all assignments of values to the variables in the sets P S1 s assignments S2 s assignments S3 s assignments P S1 s assignments S3 s assignments Slide 16 The Monty Hall problem highly practical First guess What are the conditional indepencies A I A B C Stick or I A C B swap I E A C B D I D E B Second guess The money B C The goat E Slide 17 What Independencies does a Bayes Net Model In order for a Bayesian network to model a probability distribution the following must be true by definition Each variable is conditionally independent of all its non descendants in the graph given the value of all its parents n P X 1 X n P X i parents X i This implies i 1 But what else does it imply Slide 18 What Independencies does a Bayes …
View Full Document
Unlocking...