View Full Document

7 views

Unformatted text preview:

Journal of Machine Learning Research 6 2005 81 127 Submitted 9 04 Revised 12 04 Published 1 05 Learning Hidden Variable Networks The Information Bottleneck Approach Gal Elidan GALEL CS HUJI AC IL Department of Engineering and Computer Science The Hebrew University Jerusalem 91904 Israel Nir Friedman NIR CS HUJI AC IL Department of Engineering and Computer Science The Hebrew University Jerusalem 91904 Israel Editor David Maxwell Chickering Abstract A central challenge in learning probabilistic graphical models is dealing with domains that involve hidden variables The common approach for learning model parameters in such domains is the expectation maximization EM algorithm This algorithm however can easily get trapped in suboptimal local maxima Learning the model structure is even more challenging The structural EM algorithm can adapt the structure in the presence of hidden variables but usually performs poorly without prior knowledge about the cardinality and location of the hidden variables In this work we present a general approach for learning Bayesian networks with hidden variables that overcomes these problems The approach builds on the information bottleneck framework of Tishby et al 1999 We start by proving formal correspondence between the information bottleneck objective and the standard parametric EM functional We then use this correspondence to construct a learning algorithm that combines an information theoretic smoothing term with a continuation procedure Intuitively the algorithm bypasses local maxima and achieves superior solutions by following a continuous path from a solution of an easy and smooth target function to a solution of the desired likelihood function As we show our algorithmic framework allows learning of the parameters as well as the structure of a network In addition it also allows us to introduce new hidden variables during model selection and learn their cardinality We demonstrate the performance of our procedure on several challenging real life data sets Keywords Bayesian networks hidden variables information bottleneck continuation variational methods 1 Introduction Probabilistic graphical models have been widely used to model real world domains and are particularly appealing due to their natural interpretation Despite extensive research in learning these models from data Pearl 1988 Heckerman 1998 learning with hidden or latent variables has A preliminary version of this paper appeared in the Proceedings of the Nineteenth Conference on Uncertainty in Artificial 2003 UAI 03 c 2005 Gal Elidan and Nir Friedman E LIDAN AND F RIEDMAN remained a central challenge in learning graphical models in general and Bayesian networks in particular Hidden entities play a central role in many real life problems an unknown regulating mechanism can be the key to complex biological systems correlating symptoms might hint at a hidden fundamental problem in a diagnostic system an intentionally masked economic power might be the cause of



Access the best Study Guides, Lecture Notes and Practice Exams

Loading Unlocking...
Login

Join to view Learning Hidden Variable Networks and access 3M+ class-specific study document.

or
We will never post anything without your permission.
Don't have an account?
Sign Up

Join to view Learning Hidden Variable Networks and access 3M+ class-specific study document.

or

By creating an account you agree to our Privacy Policy and Terms Of Use

Already a member?