DOC PREVIEW
Purdue CS 59000 - Lecture Notes

This preview shows page 1-2-24-25 out of 25 pages.

Save
View full document
View full document
Premium Document
Do you want full access? Go Premium and unlock all 25 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 25 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 25 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 25 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 25 pages.
Access to all documents
Download any document
Ad free experience

Unformatted text preview:

CS 59000 Statistical machine learning Lecture 16Outline Laplace Approximation for PosteriorPrinciple Component Analysis (PCA)Feature MappingDual VariablesEigen-problem in Feature Space (1)Eigen-problem in Feature Space (2)General Case for Non-zero Mean CaseKernel PCA on Synthetic DataLimitations of Kernel PCALimitations of Kernel PCAGaussian ProcessesLinear Regression RevisitedFrom Prior on Parameter to Prior on FunctionStochastic ProcessGaussian ProcessesImpact of Kernel FunctionGaussian Process for RegressionSamples of GP Prior over FunctionsSamples of Data Points Predictive DistributionPredictive MeanGP RegressionComputational ComplexityCS 59000 Statistical machine learningLecture 16Alan QiOutlineReview of kernel PCAGaussian processesLaplace Approximation for PosteriorGaussian approximation around mode:What value shall we assign to z0?Principle Component Analysis (PCA)Assume We haveis a normalized eigenvector:Feature MappingEigen-problem in feature spaceDual VariablesSuppose (why it cannot be smaller than 0?), we haveEigen-problem in Feature Space (1)Multiplying both sides by , we obtainEigen-problem in Feature Space (2)Normalization condition:Projection coefficient:General Case for Non-zero Mean CaseKernel Matrix:Kernel PCA on Synthetic DataContour plots of projection coefficients in feature spaceLimitations of Kernel PCADiscussion…Limitations of Kernel PCAIf N is big, it is computationally expensive since K is N by N while S is D by D.Not easy for low-rank approximation.Gaussian ProcessesHow kernels arise naturally in a Bayesian setting?Instead of assigning a prior on parameters w, we assign a prior on function value y.Infinite space in theoryFinite computation in practice (finite number of training set and test set)Linear Regression RevisitedLetWe haveFrom Prior on Parameter to Prior on FunctionThe prior on function value:Stochastic ProcessA stochastic process is specified by giving the joint distribution for any finite set of values in a consistent manner (Loosely speaking, it means that a marginalized distribution is the same as the joint distribution that is defined in the subspace.)Gaussian ProcessesThe joint distribution of any variables is a multivariable Gaussian distribution.Without any prior knowledge, we often set mean to be 0. Then the GP is specified by the covariance :Impact of Kernel FunctionCovariance matrix : kernel functionApplication economics & financeGaussian Process for RegressionLikelihood:Prior:Marginal distribution:Samples of GP Prior over FunctionsSamples of Data PointsPredictive Distributionis a Gaussian distribution with mean and variance:Predictive MeanWe see the same form as kernel ridge regression and kernel PCA.GP RegressionDiscussion: the difference between GP regression and Bayesian regression with Gaussian basis functions?Computational ComplexityGP prediction for a new data point:GP: O(N3) where N is number of data pointsBasis function model: O(M3) where M is the dimension of the feature expansionWhen N is large: computationally expensive.Sparsification: make prediction based on only a few basis points that summarize the information in the whole data


View Full Document

Purdue CS 59000 - Lecture Notes

Documents in this Course
Lecture 4

Lecture 4

42 pages

Lecture 6

Lecture 6

38 pages

Load more
Download Lecture Notes
Our administrator received your request to download this document. We will send you the file to your email shortly.
Loading Unlocking...
Login

Join to view Lecture Notes and access 3M+ class-specific study document.

or
We will never post anything without your permission.
Don't have an account?
Sign Up

Join to view Lecture Notes 2 2 and access 3M+ class-specific study document.

or

By creating an account you agree to our Privacy Policy and Terms Of Use

Already a member?