CS 59000 Statistical Machine learningLecture 7Alan QiAcknowledgement: Sargur Srihari’s slidesOutlineReview of noninformative priors, nonparametric methods, and nonlinear basis functionsRegularized regressionBayesian regressionEquivalent kernel Model ComparisonThe Exponential Family (1)where ´ is the natural parameter andso g(´) can be interpreted as a normalization coefficient.Property of Normalization CoefficientFrom the definition of g(´) we getThusConjugate priorsFor any member of the exponential family, there exists a priorCombining with the likelihood function, we getPrior corresponds to º pseudo-observations with value Â.Noninformative Priors (1)With little or no information available a-priori, we might choose a non-informative prior.• ¸ discrete, K-nomial :• ¸2[a,b] real and bounded: • ¸ real and unbounded: improper!A constant prior may no longer be constant after a change of variable; consider p(¸) constant and ¸=´2:Noninformative Priors (2)Translation invariant priors. ConsiderFor a corresponding prior over ¹, we havefor any A and B. Thus p(¹) = p(¹ { c) and p(¹) must be constant.Noninformative Priors (4)Scale invariant priors. Consider and make the change of variable For a corresponding prior over ¾, we havefor any A and B. Thus p(¾) / 1/¾ and so this prior is improper too. Note that this corresponds to p(ln¾) being constant.Nonparametric Methods (1)Parametric distribution models are restricted to specific forms, which may not always be suitable; for example, consider modelling a multimodal distribution with a single, unimodal model.Nonparametric approaches make few assumptions about the overall shape of the distribution being modelled.Nonparametric Methods (2)Histogram methods partition the data space into distinct bins with widths ¢iand count the number of observations, ni, in each bin.•Often, the same width is used for all bins, ¢i= ¢.•¢ acts as a smoothing parameter.•In a D-dimensional space, using M bins in each dimen-sion will require MDbins!Nonparametric Methods (3)Assume observations drawn from a density p(x) and consider a small region Rcontaining x such thatThe probability that K out of N observations lie inside R is Bin(KjN,P ) and if N is largeIf the volume of R, V, is sufficiently small, p(x) is approximately constant over R andThusNonparametric Methods (5)To avoid discontinuities in p(x), use a smooth kernel, e.g. a GaussianAny kernel such thatwill work.h acts as a smoother.K-Nearest-Neighbours for Classification (1)Given a data set with Nkdata points from class Ckand , we haveand correspondinglySince , Bayes’ theorem givesK-Nearest-Neighbours for Classification (2)K = 1K = 3Linear Regression15Basis FunctionsExamples of Basis Functions (1)Examples of Basis Functions (2)18Maximum Likelihood Estimation (1)Maximum Likelihood Estimation (2)Sequential EstimationRegularized Least SquaresMore RegularizersVisualization of Regularized RegressionBayesian Linear RegressionPosterior Distributions of ParametersPredictive Posterior DistributionExamples of PredictiveDistributionQuestionSuppose we use Gaussian basis functions.What will happen to the predictive distribution if we evaluate it at places far from all training data points?Equivalent KernelGivenPredictive mean is whereEquivalent kernelBasis Function: Equivalent kernel:GaussianPolynomialSigmoidCovariance between two predictionsPredictive mean at nearby points will be highly correlated, whereas for more distant pairs of points the correlation will be smaller.Bayesian Model ComparisonSuppose we want to compare models .Given a training set , we computeModel evidence (also known as marginal likelihood):Bayes factor:Evidence and Parameter PosteriorMarginal likelihood and evidenceParameter posterior distribution and evidenceCrude Evidence ApproximationAssume posterior distribution is centered around its modeEvidence penalizes over-complex modelsGiven M parametersMaximizing evidence leads to a natural trade-off between data fitting & model complexity.Evidence Approximation & Empirical BayesApproximating the predictive distribution by maximizing marginal likelihood. Where hyperparameters maximize the evidence .Known as Empirical Bayes or type II maximum likelihoodModel Evidence and Cross-ValidationRoot-mean-square error Model evidenceFitting polynomial regression
View Full Document
Unlocking...