CS 59000 Statistical Machine LearningLecture 14Yuan (Alan) QiOutline• Review of Gaussian Processes (GPs)• From linear regression to GP • GP for regression• Learning hyperparameters• Automatic Relevance Determination• GP for classificationGaussian ProcessesHow kernels arise naturally in a Bayesian setting?Instead of assigning a prior on parameters w, we assign a prior on function value y.Infinite space in theoryFinite computation in practice (finite number of training set and test set)Linear Regression RevisitedLetWe haveFrom Prior on Parameter to Prior on FunctionThe prior on function value:Stochastic ProcessA stochastic process is specified by giving the joint distribution for any finite set of values in a consistent manner (Loosely speaking, it means that a marginalized joint distribution is the same as the joint distribution that is defined in the subspace.)Gaussian ProcessesThe joint distribution of any variables is a multivariable Gaussian distribution.Without any prior knowledge, we often set mean to be 0. Then the GP is specified by the covariance :Impact of Kernel FunctionCovariance matrix : kernel functionApplication economics & financeGaussian Process for RegressionLikelihood:Prior:Marginal distribution:Samples of Data PointsPredictive Distributionis a Gaussian distribution with mean and variance:Predictive Meanis the nthcomponent ofWe see the same form as kernel ridge regression and kernel PCA.GP RegressionDiscussion: the difference between GP regression and Bayesian regression with Gaussian basis functions?Computational ComplexityGP prediction for a new data point:GP: O(N3) where N is number of data pointsBasis function model: O(M3) where M is the dimension of the feature expansionWhen N is large: computationally expensive.Sparsification: make prediction based on only a few data points (essentially make N small)Learning Hyperparameters Empirical Bayes MethodsAutomatic Relevance DeterminationConsider two-dimensional problems:Maximizing the marginal likelihood will make certain small, reducing its relevance to prediction.Examplet = sin(2 π x1)x2= x1+nx3= eGaussian Processes for ClassificationLikelihood:GP Prior:Covariance function:Sample from GP PriorPredictive DistributionNo analytical solution.Approximate this integration:Laplace’s methodVariational BayesExpectation propagationLaplace’s method for GP Classification (1)Laplace’s method for GP Classification (2)Taylor expansion:Laplace’s method for GP Classification (3)Newton-Raphson update:Laplace’s method for GP Classification (4)Gaussian approximation:Laplace’s method for GP Classification (4)Question: How to get the mean and the variance above?Predictive
View Full Document
Unlocking...