CMU CS 10708 - lecture (21 pages)

Previewing pages 1, 2, 20, 21 of 21 page document View the full content.
View Full Document

lecture



Previewing pages 1, 2, 20, 21 of actual document.

View the full content.
View Full Document
View Full Document

lecture

96 views


Pages:
21
School:
Carnegie Mellon University
Course:
Cs 10708 - Probabilistic Graphical Models
Probabilistic Graphical Models Documents

Unformatted text preview:

Probabilistic Graphical Models 10 708 Variational Inference Eric Xing Lecture 18 Nov 14 2005 Reading KF Chap 9 Variational Methods z z For a distribution p X associated with a complex graph computing the marginal or conditional probability of arbitrary random variable s is intractable Variational methods z formulating probabilistic inference as an optimization problem e g f f arg max f S F f a tractable probability distribution or solutions to certain probabilistic queries 1 Lower bounds of exponential functions 8 4 2 1 0 1 2 exp x exp x 1 exp x 1 exp x 3 3 x 2 6 x 1 6 Exponential Family z Exponential representation of graphical models p X exp XD A z Includes discrete models Gaussian Poisson exponential and many others E X XD is referred to as the energy of state x p X exp E X A exp E XH xE A xE 2 Example the Boltzmann distribution on atomic lattice p X exp ij Xi X j i 0Xi Z i j i 1 Lower bounding likelihood Representing q XH by exp E XH Lemma Every marginal distribution q XH defines a lower bound of likelihood p xE d xH exp E xH 1 A xE E xH xE E xH where xE denotes observed variables evidence Upgradeable to higher order bound Leisink and Kappen 2000 3 Lower bounding likelihood Representing q XH by exp E XH Lemma Every marginal distribution q XH defines a lower bound of likelihood p xE C E X H x E C E q q XH d xH q xH log q xH Hq where xE denotes observed variables evidence E q E expected energy Hq entropy q Hq Gibbs free energy KL and variational Gibbs free energy z Kullback Leibler Distance KL q p q z ln z z q z p z Boltzmann s Law definition of energy p z 1 C exp E z KL q p q z E z q z ln q z ln C z z Gibbs Free Energy G q minimized when q Z p Z 4 KL and Log Likelihood z Jensen s inequality l x log p x log p x z z p x z q z x p x z q z x log q z x z log q z x z z l x lc x z q Hq L q KL and Lower bound of likelihood KL q p p x z p x z l x log p x log q z log p z x p z x z p x z q z q z log q z p z x z p x z q z q z log q z log q z p z x z z z ln p D L q l x L q KL



View Full Document

Access the best Study Guides, Lecture Notes and Practice Exams

Loading Unlocking...
Login

Join to view lecture and access 3M+ class-specific study document.

or
We will never post anything without your permission.
Don't have an account?
Sign Up

Join to view lecture and access 3M+ class-specific study document.

or

By creating an account you agree to our Privacy Policy and Terms Of Use

Already a member?