MIT 9 520 - Spectral Regularization (45 pages)

Previewing pages 1, 2, 3, 21, 22, 23, 43, 44, 45 of 45 page document View the full content.
View Full Document

Spectral Regularization



Previewing pages 1, 2, 3, 21, 22, 23, 43, 44, 45 of actual document.

View the full content.
View Full Document
View Full Document

Spectral Regularization

73 views


Pages:
45
School:
Massachusetts Institute of Technology
Course:
9 520 - Statistical Learning Theory and Applications
Statistical Learning Theory and Applications Documents

Unformatted text preview:

Spectral Regularization Lorenzo Rosasco 9 520 Class 08 March 1 2010 L Rosasco Spectral Regularization About this class Goal To discuss how a class of regularization methods originally designed for solving ill posed inverse problems give rise to regularized learning algorithms These algorithms are kernel methods that can be easily implemented and have a common derivation but different computational and theoretical properties L Rosasco Spectral Regularization Plan From ERM to Tikhonov regularization Linear ill posed problems and stability Spectral Regularization and Filtering Example of Algorithms L Rosasco Spectral Regularization Basic Notation training set S x1 y1 xn yn X is the n by d input matrix Y y1 yn is the output vector k denotes the kernel function K the n by n kernel matrix with entries Kij k xi xj and H the RKHS with kernel k RLS estimator solves n 1X min yi f xi 2 kf k2H f H n i 1 L Rosasco Spectral Regularization Representer Theorem We have seen that RKHS allow us to write the RLS estimator in the form fS x n X ci k x xi i 1 with K n I c Y where c c1 cn L Rosasco Spectral Regularization Empirical risk minimization Similarly we can prove that the solution of empirical risk minimization n 1X min yi f xi 2 f H n i 1 can be written as fS x n X ci k x xi i 1 where the coefficients satisfy Kc Y L Rosasco Spectral Regularization The Role of Regularization We observed that adding a penalization term can be interpreted as way to to control smoothness and avoid overfitting min f H n n i 1 i 1 1X 1X yi f xi 2 min yi f xi 2 kf k2H n f H n L Rosasco Spectral Regularization The Role of Regularization Now we can observe that adding a penalty has an effect from a numerical point of view Kc Y K n I c Y it stabilizes a possibly ill conditioned matrix inversion problem This is the point of view of regularization for ill posed inverse problems L Rosasco Spectral Regularization Ill posed Inverse Problems Hadamard introduced the definition of ill posedness Ill posed problems



View Full Document

Access the best Study Guides, Lecture Notes and Practice Exams

Loading Unlocking...
Login

Join to view Spectral Regularization and access 3M+ class-specific study document.

or
We will never post anything without your permission.
Don't have an account?
Sign Up

Join to view Spectral Regularization and access 3M+ class-specific study document.

or

By creating an account you agree to our Privacy Policy and Terms Of Use

Already a member?