Support Vector Machines Aarti Singh Machine Learning 10 701 15 781 Oct 18 2010 Support Vector Machines w x b 0 w x b 0 Linearly separable case min w w w b s t w xj b yj 1 j Solve efficiently by quadratic programming QP g g Well studied solution algorithms 2 Support Vectors Linear hyperplane defined by w x b 0 support vectors w x b 0 j w xj b yj 1 Moving other points a little doesn t effect the decision boundary only need to store the support vectors to predict labels of new points g g How many support vectors in linearly separable case m 1 3 What if data is not linearly separable Use features of features of features of features x12 x22 x1x2 exp x1 But run risk of overfitting 4 What if data is still not linearly separable Allow error in classification min w w C j w b j s t w xj b yj 1 j j j 0 j j slack variables 1 if xj misclassifed pay linear penalty if mistake Soft margin approach C tradeoff parameter chosen by cross validation Still QP 5 Soft margin SVM Soften the constraints w xj b yj 1 j j j 0 j Penalty for misclassifying C j How do we recover hard margin SVM Set C 6 Support Vectors Soften the constraints w xj b yj 1 j j j 0 j Penalty for misclassifying C j How do we recover hard margin SVM Set C 7 Slack variables Hinge loss Complexity penalization min w w C j w b j s t w xj b yj 1 j j j 0 j Hinge loss 0 1 loss 1 0 1 8 Constrained Optimization Primal problem b ve Moving the constraint to objective function Lagrangian Dual problem 0 constraint is ineffective 0 constraint is effective 9 Dual SVM linearly separable case Primal problem w weights on features Dual problem weights on training pts 10 Dual SVM linearly separable case Dual problem If we can solve for s dual problem then we have a solution for w b primal problem 11 Dual SVM Interpretation Sparsity j 0 j 0 j 0 j 0 Only few js can be non zero where constraint is tight w xj b yj 1 j 0 j 0 Support vectors training points j whose js are non zero 12 Dual SVM linearly separable case Dual problem is also QP Solution gives js Use support vectors to compute b 13 Dual SVM non separable case Primal problem Dual problem Lagrange Multipliers 14 Dual SVM non separable case comes from Intuition Earlier If constraint violated i Now If constraint violated i C Dual problem is also QP Solution gives js 15 So why solve the dual SVM There are some quadratic programming algorithms that can solve the dual faster than the primal specially in high dimensions m n But more importantly the kernel trick 16 What if data is not linearly separable Use features of features of features of features x x12 x22 x1x2 exp x1 Feature space becomes really large very quickly 17 Higher Order Polynomials m input features d degree of polynomial grows fast d 6 m 100 about 1 6 billion terms 18 Dual formulation only depends on dot products not on w x High dimensional feature space but never need it explicitly as long as we can compute the dot product fast using some Kernel K 19 Dot Product of Polynomials d 1 d 2 d 20 Finally The Kernel Trick Never represent features explicitly Compute dot products in closed form Constant time high dimensional dotproducts for many classes of features Very interesting theory Reproducing Kernel Hilbert Spaces Not covered in detail in 10701 15781 more in 10702 21 Common Kernels Polynomials of degree d Polynomials of degree up to d Gaussian Radial kernels polynomials of all orders recall series expansion of exp Sigmoid 22 Overfitting Huge feature space with kernels what about overfitting Maximizing margin leads to sparse set of support vectors Some interesting theory says that SVMs search for simple hypothesis with large margin Often robust to overfitting 23 What about classification time For a new input x if we need to represent x we are in trouble Recall classifier sign w x b Using kernels we are cool 24 SVMs with Kernels Choose a set of features and kernel function Solve dual problem to obtain support vectors i At classification time compute Classify as 25 SVMs vs Kernel Regression SVMs Kernel Regression or Differences SVMs Learn weights i and bandwidth Often sparse solution KR Fixed weights learn bandwidth Solution may not be sparse Much simpler to implement 26 SVMs vs Logistic Regression Loss function High dimensional features with kernels Solution sparse Semantics of output SVMs Logistic Regression Hinge loss Log loss Yes Yes Often yes Almost always no Margin Real probabilities 27 Kernels in Logistic Regression Define weights in terms of features Derive simple gradient descent rule on i 28 SVMs vs Logistic Regression Loss function High dimensional features with kernels Solution sparse Semantics of output SVMs Logistic Regression Hinge loss Log loss Yes Yes Often yes Almost always no Margin Real probabilities 29 What you need to know Dual SVM formulation How it s derived The kernel trick Common kernels Differences between SVMs and kernel regression Differences between SVMs and logistic regression Kernelized logistic regression 30 Announcements Midterm When Wednesday 10 20 Where In Class What You your pencil your textbook your notes course slides your calculator your good mood What NOT No computers iphones or anything else that has an internet connection Material Everything from the beginning of the semester until and including SVMs and the Kernel trick 31 Midterm Review What is ML loss functions Bayes optimal rules classification regression Parametric approaches Learning distributions MLE MAP Classification Na ve Bayes Logistic Regression Regression Linear Non parametric approaches Density estimation Histogram Kernel density estimation Classification kNN Decision Trees Regression Kernel regression Model Selection Overfitting Bias variance tradeoff estimating the generalization error Boosting SVM 32
View Full Document
Unlocking...