Support Vector MachinesAarti SinghMachine Learning 10-701/15-781Oct 18, 2010Support Vector Machines2w.x + b > 0w.x + b < 0ggmin w.ww,bs.t. (w.xj+b) yj≥ 1 j Solve efficiently by quadratic programming (QP)– Well-studied solution algorithmsLinearly separable caseSupport Vectors3w.x + b > 0w.x + b < 0ggLinear hyperplane defined by “support vectors”j: (w.xj+b) yj= 1Moving other points a little doesn’t effect the decision boundary only need to store the support vectors to predict labels of new pointsHow many support vectors in linearly separable case?≤ m+1What if data is not linearly separable?4Use features of features of features of features….But run risk of overfitting!x12, x22, x1x2, …., exp(x1)What if data is still not linearly separable?5min w.w + C Σξjw,bs.t. (w.xj+b) yj≥ 1-ξjjξj≥ 0 jjAllow “error” in classificationξj - “slack” variables = (>1 if xjmisclassifed)pay linear penalty if mistakeC - tradeoff parameter (chosen by cross-validation)Still QP Soft margin approachSoft-margin SVM6(w.xj+b) yj≥ 1-ξjjξj≥ 0 jSoften the constraints:Penalty for misclassifying:C ξjHow do we recover hardmargin SVM?Set C = ∞Support Vectors7(w.xj+b) yj≥ 1-ξjjξj≥ 0 jSoften the constraints:Penalty for misclassifying:C ξjHow do we recover hardmargin SVM?Set C = ∞Slack variables – Hinge loss8min w.w + C Σξjw,bs.t. (w.xj+b) yj≥ 1-ξjjξj≥ 0 jjComplexity penalizationHinge loss0-1 loss0-11Constrained Optimization9Moving the constraint to objective functionLagrangian:Dual problem: = 0 constraint is ineffective > 0 constraint is effectiveb +vePrimal problem:Dual SVM – linearly separable case• Primal problem:• Dual problem: 10w – weights on features – weights on training ptsDual SVM – linearly separable case• Dual problem: 11If we can solve for s (dual problem), then we have a solution for w,b(primal problem)Dual SVM Interpretation: Sparsity12Only few js can be non-zero : where constraint is tight(w.xj+ b)yj= 1Support vectors –training points j whose js are non-zeroj> 0j> 0j> 0j= 0j= 0j= 0Dual SVM – linearly separable caseDual problem is also QPSolution gives js13Use support vectors to compute bDual SVM – non-separable case14• Primal problem:• Dual problem: Lagrange MultipliersDual SVM – non-separable case15Dual problem is also QPSolution gives jscomes fromIntuition:Earlier - If constraint violated, i→∞Now - If constraint violated, i≤ CSo why solve the dual SVM?• There are some quadratic programming algorithms that can solve the dual faster than the primal, (specially in high dimensions m>>n)• But, more importantly, the “kernel trick”!!!16What if data is not linearly separable?17Use features of features of features of features….Feature space becomes really large very quickly!Φ(x) = (x12, x22, x1x2, …., exp(x1))Higher Order Polynomials18m – input features d – degree of polynomialgrows fast!d = 6, m = 100about 1.6 billion termsDual formulation only depends on dot-products, not on w!19Φ(x) – High-dimensional feature space, but never need it explicitly as long as we can compute the dot product fast using some Kernel KDot Product of Polynomials20d=1d=2dFinally: The Kernel Trick!21• Never represent features explicitly– Compute dot products in closed form• Constant-time high-dimensional dot-products for many classes of features• Very interesting theory – Reproducing Kernel Hilbert Spaces– Not covered in detail in 10701/15781, more in 10702Common Kernels22• Polynomials of degree d• Polynomials of degree up to d• Gaussian/Radial kernels (polynomials of all orders – recall series expansion of exp)• SigmoidOverfitting23• Huge feature space with kernels, what about overfitting???– Maximizing margin leads to sparse set of support vectors – Some interesting theory says that SVMs search for simple hypothesis with large margin– Often robust to overfittingWhat about classification time?24• For a new input x, if we need to represent (x), we are in trouble!• Recall classifier: sign(w.(x)+b)• Using kernels we are cool!SVMs with Kernels25• Choose a set of features and kernel function• Solve dual problem to obtain support vectors i• At classification time, compute:Classify as26SVMsKernel RegressionorDifferences:• SVMs:– Learn weights i(and bandwidth)– Often sparse solution• KR:– Fixed “weights”, learn bandwidth– Solution may not be sparse– Much simpler to implementSVMs vs. Kernel RegressionSVMs vs. Logistic Regression27SVMs LogisticRegressionLoss function Hinge loss Log-lossHigh dimensional features with kernelsYes! Yes!Solution sparse Often yes! Almost always no!Semantics of output“Margin” Real probabilitiesKernels in Logistic Regression28• Define weights in terms of features:• Derive simple gradient descent rule on iSVMs vs. Logistic Regression29SVMs LogisticRegressionLoss function Hinge loss Log-lossHigh dimensional features with kernelsYes! Yes!Solution sparse Often yes! Almost always no!Semantics of output“Margin” Real probabilitiesWhat you need to know…30• Dual SVM formulation– How it’s derived• The kernel trick• Common kernels• Differences between SVMs and kernel regression• Differences between SVMs and logistic regression• Kernelized logistic regressionAnnouncements - Midterm• When: Wednesday, 10/20• Where: In Class• What: You, your pencil, your textbook, your notes, course slides, your calculator, your good mood :)• What NOT: No computers, iphones, or anything else that has an internet connection.• Material: Everything from the beginning of the semester, until, and including SVMs and the Kernel trick31Midterm Review• What is ML? loss functions• Bayes optimal rules (classification, regression)• Parametric approaches– Learning distributions: MLE, MAP– Classification: Naïve Bayes, Logistic Regression– Regression: Linear• Non-parametric approaches– Density estimation: Histogram, Kernel density estimation– Classification: kNN, Decision Trees– Regression: Kernel regression• Model Selection, Overfitting, Bias-variance tradeoff, estimating the generalization error• Boosting,
View Full Document