DOC PREVIEW
CMU CS 10701 - Lecture12

This preview shows page 1-2-15-16-31-32 out of 32 pages.

Save
View full document
View full document
Premium Document
Do you want full access? Go Premium and unlock all 32 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 32 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 32 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 32 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 32 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 32 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 32 pages.
Access to all documents
Download any document
Ad free experience

Unformatted text preview:

Support Vector MachinesAarti SinghMachine Learning 10-701/15-781Oct 18, 2010Support Vector Machines2w.x + b > 0w.x + b < 0ggmin w.ww,bs.t. (w.xj+b) yj≥ 1 j Solve efficiently by quadratic programming (QP)– Well-studied solution algorithmsLinearly separable caseSupport Vectors3w.x + b > 0w.x + b < 0ggLinear hyperplane defined by “support vectors”j: (w.xj+b) yj= 1Moving other points a little doesn’t effect the decision boundary only need to store the support vectors to predict labels of new pointsHow many support vectors in linearly separable case?≤ m+1What if data is not linearly separable?4Use features of features of features of features….But run risk of overfitting!x12, x22, x1x2, …., exp(x1)What if data is still not linearly separable?5min w.w + C Σξjw,bs.t. (w.xj+b) yj≥ 1-ξjjξj≥ 0 jjAllow “error” in classificationξj - “slack” variables = (>1 if xjmisclassifed)pay linear penalty if mistakeC - tradeoff parameter (chosen by cross-validation)Still QP Soft margin approachSoft-margin SVM6(w.xj+b) yj≥ 1-ξjjξj≥ 0 jSoften the constraints:Penalty for misclassifying:C ξjHow do we recover hardmargin SVM?Set C = ∞Support Vectors7(w.xj+b) yj≥ 1-ξjjξj≥ 0 jSoften the constraints:Penalty for misclassifying:C ξjHow do we recover hardmargin SVM?Set C = ∞Slack variables – Hinge loss8min w.w + C Σξjw,bs.t. (w.xj+b) yj≥ 1-ξjjξj≥ 0 jjComplexity penalizationHinge loss0-1 loss0-11Constrained Optimization9Moving the constraint to objective functionLagrangian:Dual problem: = 0 constraint is ineffective > 0 constraint is effectiveb +vePrimal problem:Dual SVM – linearly separable case• Primal problem:• Dual problem: 10w – weights on features – weights on training ptsDual SVM – linearly separable case• Dual problem: 11If we can solve for s (dual problem), then we have a solution for w,b(primal problem)Dual SVM Interpretation: Sparsity12Only few js can be non-zero : where constraint is tight(w.xj+ b)yj= 1Support vectors –training points j whose js are non-zeroj> 0j> 0j> 0j= 0j= 0j= 0Dual SVM – linearly separable caseDual problem is also QPSolution gives js13Use support vectors to compute bDual SVM – non-separable case14• Primal problem:• Dual problem: Lagrange MultipliersDual SVM – non-separable case15Dual problem is also QPSolution gives jscomes fromIntuition:Earlier - If constraint violated, i→∞Now - If constraint violated, i≤ CSo why solve the dual SVM?• There are some quadratic programming algorithms that can solve the dual faster than the primal, (specially in high dimensions m>>n)• But, more importantly, the “kernel trick”!!!16What if data is not linearly separable?17Use features of features of features of features….Feature space becomes really large very quickly!Φ(x) = (x12, x22, x1x2, …., exp(x1))Higher Order Polynomials18m – input features d – degree of polynomialgrows fast!d = 6, m = 100about 1.6 billion termsDual formulation only depends on dot-products, not on w!19Φ(x) – High-dimensional feature space, but never need it explicitly as long as we can compute the dot product fast using some Kernel KDot Product of Polynomials20d=1d=2dFinally: The Kernel Trick!21• Never represent features explicitly– Compute dot products in closed form• Constant-time high-dimensional dot-products for many classes of features• Very interesting theory – Reproducing Kernel Hilbert Spaces– Not covered in detail in 10701/15781, more in 10702Common Kernels22• Polynomials of degree d• Polynomials of degree up to d• Gaussian/Radial kernels (polynomials of all orders – recall series expansion of exp)• SigmoidOverfitting23• Huge feature space with kernels, what about overfitting???– Maximizing margin leads to sparse set of support vectors – Some interesting theory says that SVMs search for simple hypothesis with large margin– Often robust to overfittingWhat about classification time?24• For a new input x, if we need to represent (x), we are in trouble!• Recall classifier: sign(w.(x)+b)• Using kernels we are cool!SVMs with Kernels25• Choose a set of features and kernel function• Solve dual problem to obtain support vectors i• At classification time, compute:Classify as26SVMsKernel RegressionorDifferences:• SVMs:– Learn weights i(and bandwidth)– Often sparse solution• KR:– Fixed “weights”, learn bandwidth– Solution may not be sparse– Much simpler to implementSVMs vs. Kernel RegressionSVMs vs. Logistic Regression27SVMs LogisticRegressionLoss function Hinge loss Log-lossHigh dimensional features with kernelsYes! Yes!Solution sparse Often yes! Almost always no!Semantics of output“Margin” Real probabilitiesKernels in Logistic Regression28• Define weights in terms of features:• Derive simple gradient descent rule on iSVMs vs. Logistic Regression29SVMs LogisticRegressionLoss function Hinge loss Log-lossHigh dimensional features with kernelsYes! Yes!Solution sparse Often yes! Almost always no!Semantics of output“Margin” Real probabilitiesWhat you need to know…30• Dual SVM formulation– How it’s derived• The kernel trick• Common kernels• Differences between SVMs and kernel regression• Differences between SVMs and logistic regression• Kernelized logistic regressionAnnouncements - Midterm• When: Wednesday, 10/20• Where: In Class• What: You, your pencil, your textbook, your notes, course slides, your calculator, your good mood :)• What NOT: No computers, iphones, or anything else that has an internet connection.• Material: Everything from the beginning of the semester, until, and including SVMs and the Kernel trick31Midterm Review• What is ML? loss functions• Bayes optimal rules (classification, regression)• Parametric approaches– Learning distributions: MLE, MAP– Classification: Naïve Bayes, Logistic Regression– Regression: Linear• Non-parametric approaches– Density estimation: Histogram, Kernel density estimation– Classification: kNN, Decision Trees– Regression: Kernel regression• Model Selection, Overfitting, Bias-variance tradeoff, estimating the generalization error• Boosting,


View Full Document

CMU CS 10701 - Lecture12

Documents in this Course
lecture

lecture

12 pages

lecture

lecture

17 pages

HMMs

HMMs

40 pages

lecture

lecture

15 pages

lecture

lecture

20 pages

Notes

Notes

10 pages

Notes

Notes

15 pages

Lecture

Lecture

22 pages

Lecture

Lecture

13 pages

Lecture

Lecture

24 pages

Lecture9

Lecture9

38 pages

lecture

lecture

26 pages

lecture

lecture

13 pages

Lecture

Lecture

5 pages

lecture

lecture

18 pages

lecture

lecture

22 pages

Boosting

Boosting

11 pages

lecture

lecture

16 pages

lecture

lecture

20 pages

Lecture

Lecture

20 pages

Lecture

Lecture

39 pages

Lecture

Lecture

14 pages

Lecture

Lecture

18 pages

Lecture

Lecture

13 pages

Exam

Exam

10 pages

Lecture

Lecture

27 pages

Lecture

Lecture

15 pages

Lecture

Lecture

24 pages

Lecture

Lecture

16 pages

Lecture

Lecture

23 pages

Lecture6

Lecture6

28 pages

Notes

Notes

34 pages

lecture

lecture

15 pages

Midterm

Midterm

11 pages

lecture

lecture

11 pages

lecture

lecture

23 pages

Boosting

Boosting

35 pages

Lecture

Lecture

49 pages

Lecture

Lecture

22 pages

Lecture

Lecture

16 pages

Lecture

Lecture

18 pages

Lecture

Lecture

35 pages

lecture

lecture

22 pages

lecture

lecture

24 pages

Midterm

Midterm

17 pages

exam

exam

15 pages

lecture

lecture

19 pages

Lecture

Lecture

32 pages

boosting

boosting

11 pages

pca-mdps

pca-mdps

56 pages

bns

bns

45 pages

mdps

mdps

42 pages

svms

svms

10 pages

Notes

Notes

12 pages

lecture

lecture

42 pages

lecture

lecture

29 pages

lecture

lecture

15 pages

Lecture

Lecture

12 pages

Lecture

Lecture

24 pages

Lecture

Lecture

22 pages

Midterm

Midterm

5 pages

mdps-rl

mdps-rl

26 pages

Load more
Download Lecture12
Our administrator received your request to download this document. We will send you the file to your email shortly.
Loading Unlocking...
Login

Join to view Lecture12 and access 3M+ class-specific study document.

or
We will never post anything without your permission.
Don't have an account?
Sign Up

Join to view Lecture12 2 2 and access 3M+ class-specific study document.

or

By creating an account you agree to our Privacy Policy and Terms Of Use

Already a member?