Parameter and Structure LearningOverviewNoteSlide 4Slide 5Parameter LearningProperties of estimatorSlide 8Slide 9Slide 10Slide 11So why MLE?Let’s tryBack to BNsLearning the CPTsExampleSlide 17Maximum likelihood estimation (MLE) of BN parameters – exampleDecomposabilityTaking derivatives of MLE of BN parameters – General caseStructure LearningScore BasedSlide 23Slide 24Slide 25Slide and other creditsParameter and Structure LearningDhruv Batra,10-708 Recitation10/02/2008Overview•Parameter Learning–Classical view, estimation task–Estimators, properties of estimators–MLE, why MLE?–MLE in BNs, decomposability•Structure Learning–Structure score, decomposable scores –TAN, Chow-Liu–HW2 implementation stepsNote•Plagiarism alert–Some slides taken from others–Credits/references at the endParameter Learning•Classical statistics view / Point Estimation–Parameters unknown but not random–Point estimation = “find the right parameter”–Estimate parameters (or functions of parameters) of the model from data•Estimators–Any statistic–Function of data alone•Say you have a dataset–Need to estimate mean–Is 5, an estimator?–What would you do?Properties of estimator•Since estimator gives rise an estimate that depends on sample points (x1,x2,,,xn) estimate is a function of sample points.•Sample points are random variable therefore estimate is random variable and has probability distribution. •We want that estimator to have several desirable properties like•Consistency•Unbiasedness•Minimum variance•In general it is not possible for an estimator to have all these properties.So why MLE?•MLE has some nice properties–MLEs are often simple and easy to compute.–MLEs have asymptotic optimality properties (consistency and efficiency).–MLEs are invariant under reparameterization.–and more..Let’s tryBack to BNs•MLE in BN–Data–Model DAG G–Parameters CPTs–Learn parameters from datax(1)… x(m)Datastructure parametersCPTs – P(Xi| PaXi)10-708 – Carlos Guestrin 2006-200815Learning the CPTsx(1)… x(m)DataFor each discrete variable XiExample•Learning MLE parameters10-708 – Carlos Guestrin 2006-200817Learning the CPTsx(1)… x(m)DataFor each discrete variable Xi10-708 – Carlos Guestrin 2006-200818Maximum likelihood estimation (MLE) of BN parameters – example •Given structure, log likelihood of data:FluAllergySinusNoseDecomposability•Likelihood Decomposition•Local likelihood functionWhat’s the difference?Global parameter independence!Taking derivatives of MLE of BN parameters – General caseStructure Learning•Constraint Based–Check independences, learn PDAG–HW1•Score Based–Give a score for all possible structures–Maximize scoreScore Based•What’s a good score function?•How about our old friend, log likelihood?•So here’s our score function:Score Based•[Defn]: Decomposable scores•Why do we care about decomposable scores?•Log likelihood based score decomposes!Need regularizationScore Based•Chow-LiuScore Based•Chow-Liu modification for TAN (HW2)Slide and other credits•Zoubin Ghahramani, guest lectures in 10-702•Andrew Moore tutorial–http://www.autonlab.org/tutorials/mle.html•http://cnx.org/content/m11446/latest/•Lecture slides by Carlos
View Full Document
Unlocking...