# CMU CS 10708 - lecture (21 pages)

Previewing pages 1, 2, 20, 21 of 21 page document
View Full Document

## lecture

Previewing pages 1, 2, 20, 21 of actual document.

View Full Document
View Full Document

## lecture

96 views

Pages:
21
School:
Carnegie Mellon University
Course:
Cs 10708 - Probabilistic Graphical Models
##### Probabilistic Graphical Models Documents

Unformatted text preview:

Probabilistic Graphical Models 10 708 Variational Inference Eric Xing Lecture 18 Nov 14 2005 Reading KF Chap 9 Variational Methods z z For a distribution p X associated with a complex graph computing the marginal or conditional probability of arbitrary random variable s is intractable Variational methods z formulating probabilistic inference as an optimization problem e g f f arg max f S F f a tractable probability distribution or solutions to certain probabilistic queries 1 Lower bounds of exponential functions 8 4 2 1 0 1 2 exp x exp x 1 exp x 1 exp x 3 3 x 2 6 x 1 6 Exponential Family z Exponential representation of graphical models p X exp XD A z Includes discrete models Gaussian Poisson exponential and many others E X XD is referred to as the energy of state x p X exp E X A exp E XH xE A xE 2 Example the Boltzmann distribution on atomic lattice p X exp ij Xi X j i 0Xi Z i j i 1 Lower bounding likelihood Representing q XH by exp E XH Lemma Every marginal distribution q XH defines a lower bound of likelihood p xE d xH exp E xH 1 A xE E xH xE E xH where xE denotes observed variables evidence Upgradeable to higher order bound Leisink and Kappen 2000 3 Lower bounding likelihood Representing q XH by exp E XH Lemma Every marginal distribution q XH defines a lower bound of likelihood p xE C E X H x E C E q q XH d xH q xH log q xH Hq where xE denotes observed variables evidence E q E expected energy Hq entropy q Hq Gibbs free energy KL and variational Gibbs free energy z Kullback Leibler Distance KL q p q z ln z z q z p z Boltzmann s Law definition of energy p z 1 C exp E z KL q p q z E z q z ln q z ln C z z Gibbs Free Energy G q minimized when q Z p Z 4 KL and Log Likelihood z Jensen s inequality l x log p x log p x z z p x z q z x p x z q z x log q z x z log q z x z z l x lc x z q Hq L q KL and Lower bound of likelihood KL q p p x z p x z l x log p x log q z log p z x p z x z p x z q z q z log q z p z x z p x z q z q z log q z log q z p z x z z z ln p D L q l x L q KL

View Full Document

## Access the best Study Guides, Lecture Notes and Practice Exams Unlocking...