# MIT 9 520 - Spectral Regularization (45 pages)

Previewing pages*1, 2, 3, 21, 22, 23, 43, 44, 45*of 45 page document

**View the full content.**# Spectral Regularization

Previewing pages *1, 2, 3, 21, 22, 23, 43, 44, 45*
of
actual document.

**View the full content.**View Full Document

## Spectral Regularization

0 0 73 views

- Pages:
- 45
- School:
- Massachusetts Institute of Technology
- Course:
- 9 520 - Statistical Learning Theory and Applications

**Unformatted text preview: **

Spectral Regularization Lorenzo Rosasco 9 520 Class 08 March 1 2010 L Rosasco Spectral Regularization About this class Goal To discuss how a class of regularization methods originally designed for solving ill posed inverse problems give rise to regularized learning algorithms These algorithms are kernel methods that can be easily implemented and have a common derivation but different computational and theoretical properties L Rosasco Spectral Regularization Plan From ERM to Tikhonov regularization Linear ill posed problems and stability Spectral Regularization and Filtering Example of Algorithms L Rosasco Spectral Regularization Basic Notation training set S x1 y1 xn yn X is the n by d input matrix Y y1 yn is the output vector k denotes the kernel function K the n by n kernel matrix with entries Kij k xi xj and H the RKHS with kernel k RLS estimator solves n 1X min yi f xi 2 kf k2H f H n i 1 L Rosasco Spectral Regularization Representer Theorem We have seen that RKHS allow us to write the RLS estimator in the form fS x n X ci k x xi i 1 with K n I c Y where c c1 cn L Rosasco Spectral Regularization Empirical risk minimization Similarly we can prove that the solution of empirical risk minimization n 1X min yi f xi 2 f H n i 1 can be written as fS x n X ci k x xi i 1 where the coefficients satisfy Kc Y L Rosasco Spectral Regularization The Role of Regularization We observed that adding a penalization term can be interpreted as way to to control smoothness and avoid overfitting min f H n n i 1 i 1 1X 1X yi f xi 2 min yi f xi 2 kf k2H n f H n L Rosasco Spectral Regularization The Role of Regularization Now we can observe that adding a penalty has an effect from a numerical point of view Kc Y K n I c Y it stabilizes a possibly ill conditioned matrix inversion problem This is the point of view of regularization for ill posed inverse problems L Rosasco Spectral Regularization Ill posed Inverse Problems Hadamard introduced the definition of ill posedness Ill posed problems

View Full Document