DOC PREVIEW
CMU CS 10701 - PAC-learning, VC Dimension and Margin-based Bounds

This preview shows page 1-2-3-4-27-28-29-30-56-57-58-59 out of 59 pages.

Save
View full document
View full document
Premium Document
Do you want full access? Go Premium and unlock all 59 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 59 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 59 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 59 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 59 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 59 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 59 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 59 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 59 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 59 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 59 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 59 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 59 pages.
Access to all documents
Download any document
Ad free experience

Unformatted text preview:

PAC-learning, VC Dimension and Margin-based BoundsAnnouncements 1Announcements 2What now…How likely is learner to pick a bad hypothesisUnion boundHow likely is learner to pick a bad hypothesisReview: Generalization error in finite hypothesis spaces [Haussler ’88]Using a PAC boundReview: Generalization error in finite hypothesis spaces [Haussler ’88]Limitations of Haussler ‘88 boundSimpler question: What’s the expected error of a hypothesis?But we are comparing many hypothesis: Union boundGeneralization bound for |H| hypothesisPAC bound and Bias-Variance tradeoffWhat about the size of the hypothesis space?Boolean formulas with n binary featuresNumber of decision trees of depth kPAC bound for decision trees of depth kNumber of decision trees with k leavesPAC bound for decision trees with k leaves – Bias-Variance revisitedWhat did we learn from decision trees?What about continuous hypothesis spaces?How many points can a linear boundary classify exactly? (1-D)How many points can a linear boundary classify exactly? (2-D)How many points can a linear boundary classify exactly? (d-D)Shattering a set of pointsVC dimensionPAC bound using VC dimensionExamples of VC dimensionAnother VC dim. examplePAC bound for SVMsVC dimension and SVMs: Problems!!!Margin-based VC dimensionApplying margin VC to SVMs?Structural risk minimization theoremReality check – Bounds are looseWhat you need to knowBig PictureWhat you have learned thus farReview material in terms of…Text ClassificationFunction fittingMonitoring a complex systemTypes of learning problemsThe learning problemComparing learning algorithmsNaïve Bayes versus Logistic regressionNaïve Bayes versus Logistic regression – Classification as density estimationLogistic regression versus BoostingLinear classifiers – Logistic regression versus SVMsWhat’s the difference between SVMs and Logistic Regression? (Revisited again)SVMs and instance-based learningInstance-based learning versus Decision treesLogistic regression versus Neural netsLinear regression versus Kernel regressionKernel-weighted linear regressionSVM regressionBIG PICTURE (a few points of comparison)©2006 Carlos Guestrin1More details:General: http://www.learning-with-kernels.org/Example of more complex bounds:http://www.research.ibm.com/people/t/tzhang/papers/jmlr02_cover.ps.gzPAC-learning, VC Dimension and Margin-based BoundsMachine Learning – 10701/15781Carlos GuestrinCarnegie Mellon UniversityMarch 6th, 2006©2006 Carlos Guestrin2Announcements 1 Midterm on Wednesday open book, texts, notes,… no laptops bring a calculator©2006 Carlos Guestrin3Announcements 2 Final project details are out!!! http://www.cs.cmu.edu/~guestrin/Class/10701/projects.html Great opportunity to apply ideas from class and learn more Example project:  Take a dataset Define learning task Apply learning algorithms Design your own extension Evaluate your ideas many of suggestions on the webpage, but you can also do your own Boring stuff: Individually or groups of two students It’s worth 20% of your final grade You need to submit a one page proposal on Wed. 3/22 (just after the break) A 5-page initial write-up (milestone) is due on 4/12 (20% of project grade) An 8-page final write-up due 5/8 (60% of the grade) A poster session for all students will be held on Friday 5/5 2-5pm in NSH atrium (20% of the grade) You can use late days on write-ups, each student in team will be charged a late day per day.  MOST IMPORTANT:©2006 Carlos Guestrin4What now… We have explored many ways of learning from data But… How good is our classifier, really? How much data do I need to make it “good enough”?©2006 Carlos Guestrin5How likely is learner to pick a bad hypothesis Prob. h with errortrue(h) ≥ ε gets m data points right There are k hypothesis consistent with data How likely is learner to pick a bad one?©2006 Carlos Guestrin6Union bound P(A or B or C or D or …)©2006 Carlos Guestrin7How likely is learner to pick a bad hypothesis Prob. h with errortrue(h) ≥ ε gets m data points right There are k hypothesis consistent with data How likely is learner to pick a bad one?©2006 Carlos Guestrin8Review: Generalization error in finite hypothesis spaces [Haussler ’88] Theorem: Hypothesis space H finite, dataset Dwith m i.i.d. samples, 0 < ε < 1 : for any learned hypothesis h that is consistent on the training data:©2006 Carlos Guestrin9Using a PAC bound Typically, 2 use cases: 1: Pick ε and δ, give you m 2: Pick m and δ, give you ε©2006 Carlos Guestrin10Review: Generalization error in finite hypothesis spaces [Haussler ’88] Theorem: Hypothesis space H finite, dataset Dwith m i.i.d. samples, 0 < ε < 1 : for any learned hypothesis h that is consistent on the training data:Even if h makes zero errors in training data, may make errors in test©2006 Carlos Guestrin11Limitations of Haussler ‘88 bound Consistent classifier Size of hypothesis space©2006 Carlos Guestrin12Simpler question: What’s the expected error of a hypothesis? The error of a hypothesis is like estimating the parameter of a coin! Chernoff bound: for m i.d.d. coin flips, x1,…,xm, where xi∈ {0,1}. For 0<ε<1:©2006 Carlos Guestrin13But we are comparing many hypothesis: Union boundFor each hypothesis hi:What if I am comparing two hypothesis, h1 and h2?©2006 Carlos Guestrin14Generalization bound for |H| hypothesis Theorem: Hypothesis space H finite, dataset Dwith m i.i.d. samples, 0 < ε < 1 : for any learned hypothesis h:©2006 Carlos Guestrin15PAC bound and Bias-Variance tradeoff  Important: PAC bound holds for all h, but doesn’t guarantee that algorithm finds best h!!!or, after moving some terms around,with probability at least 1-δ:©2006 Carlos Guestrin16What about the size of the hypothesis space? How large is the hypothesis space?©2006 Carlos Guestrin17Boolean formulas with n binary features©2006 Carlos Guestrin18Number of decision trees of depth kRecursive solution Given n attributesHk= Number of decision trees of depth kH0=2Hk+1= (#choices of root attribute) *(# possible left subtrees) *(# possible right subtrees)= n * Hk* HkWrite Lk= log2HkL0= 1Lk+1= log2n + 2LkSo Lk= (2k-1)(1+log2n) +1©2006 Carlos Guestrin19PAC bound for decision trees of depth k Bad!!! Number of points is exponential in depth! But, for m data points, decision tree can’t get too


View Full Document

CMU CS 10701 - PAC-learning, VC Dimension and Margin-based Bounds

Documents in this Course
lecture

lecture

12 pages

lecture

lecture

17 pages

HMMs

HMMs

40 pages

lecture

lecture

15 pages

lecture

lecture

20 pages

Notes

Notes

10 pages

Notes

Notes

15 pages

Lecture

Lecture

22 pages

Lecture

Lecture

13 pages

Lecture

Lecture

24 pages

Lecture9

Lecture9

38 pages

lecture

lecture

26 pages

lecture

lecture

13 pages

Lecture

Lecture

5 pages

lecture

lecture

18 pages

lecture

lecture

22 pages

Boosting

Boosting

11 pages

lecture

lecture

16 pages

lecture

lecture

20 pages

Lecture

Lecture

20 pages

Lecture

Lecture

39 pages

Lecture

Lecture

14 pages

Lecture

Lecture

18 pages

Lecture

Lecture

13 pages

Exam

Exam

10 pages

Lecture

Lecture

27 pages

Lecture

Lecture

15 pages

Lecture

Lecture

24 pages

Lecture

Lecture

16 pages

Lecture

Lecture

23 pages

Lecture6

Lecture6

28 pages

Notes

Notes

34 pages

lecture

lecture

15 pages

Midterm

Midterm

11 pages

lecture

lecture

11 pages

lecture

lecture

23 pages

Boosting

Boosting

35 pages

Lecture

Lecture

49 pages

Lecture

Lecture

22 pages

Lecture

Lecture

16 pages

Lecture

Lecture

18 pages

Lecture

Lecture

35 pages

lecture

lecture

22 pages

lecture

lecture

24 pages

Midterm

Midterm

17 pages

exam

exam

15 pages

Lecture12

Lecture12

32 pages

lecture

lecture

19 pages

Lecture

Lecture

32 pages

boosting

boosting

11 pages

pca-mdps

pca-mdps

56 pages

bns

bns

45 pages

mdps

mdps

42 pages

svms

svms

10 pages

Notes

Notes

12 pages

lecture

lecture

42 pages

lecture

lecture

29 pages

lecture

lecture

15 pages

Lecture

Lecture

12 pages

Lecture

Lecture

24 pages

Lecture

Lecture

22 pages

Midterm

Midterm

5 pages

mdps-rl

mdps-rl

26 pages

Load more
Download PAC-learning, VC Dimension and Margin-based Bounds
Our administrator received your request to download this document. We will send you the file to your email shortly.
Loading Unlocking...
Login

Join to view PAC-learning, VC Dimension and Margin-based Bounds and access 3M+ class-specific study document.

or
We will never post anything without your permission.
Don't have an account?
Sign Up

Join to view PAC-learning, VC Dimension and Margin-based Bounds 2 2 and access 3M+ class-specific study document.

or

By creating an account you agree to our Privacy Policy and Terms Of Use

Already a member?