DOC PREVIEW
CMU CS 10701 - Machine Learning, Decision Trees, Overfitting

This preview shows page 1-2-3-18-19-36-37-38 out of 38 pages.

Save
View full document
View full document
Premium Document
Do you want full access? Go Premium and unlock all 38 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 38 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 38 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 38 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 38 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 38 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 38 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 38 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 38 pages.
Access to all documents
Download any document
Ad free experience

Unformatted text preview:

Machine Learning,Decision Trees, OverfittingMachine Learning 10-701Tom M. MitchellMachine Learning DepartmentCarnegie Mellon UniversitySeptember 12, 2006Reading: Mitchell, Chapter 3Bishop Section 1.6Machine Learning 10-701/15-781Instructors• Tom Mitchell•Eric XingTA’s• Fan Guo• Yifen Huang• Indra RustandiCourse assistant• Sharon Cavlovichwww.cs.cmu.edu/~epxing/Class/10701/See webpage for • Office hours• Grading policy• Final exam date• Late homework policy• Syllabus details•...Machine Learning:Study of algorithms that• improve their performanceP• at some taskT• with experienceEwell-defined learning task: <P,T,E>Learning to Predict Emergency C-Sections9714 patient records, each with 215 features[Sims et al., 2000]Learning to detect objects in imagesExample training images for each orientation(Prof. H. Schneiderman)Learning to classify text documentsCompany home pagevsPersonal home pagevsUniversity home pagevs…Reading a noun (vs verb)[Rustandi et al., 2005]Growth of Machine Learning• Machine learning is preferred approach to– Speech recognition, Natural language processing– Computer vision– Medical outcomes analysis– Robot control–…• This ML niche is growing– Improved machine learning algorithms – Increased data capture, networking– Software too complex to write by hand– New sensors / IO devices– Demand for self-customization to user, environmentAll software apps.ML apps.Function Approximation and Decision tree learningFunction approximationSetting:• Set of possible instances X• Unknown target function f: XÆY• Set of function hypotheses H={ h | h: XÆY }Given:• Training examples {<xi,yi>} of unknown target function fDetermine:• Hypothesis h ∈ H that best approximates fEach internal node: test one attribute XiEach branch from a node: selects one value for XiEach leaf node: predict Y (or P(Y|X ∈ leaf))How would you representAB ∨ CD(¬E)?node = Root[ID3, C4.5, …]EntropyEntropy H(X) of a random variable XH(X) is the expected number of bits needed to encode a randomly drawn value of X (under most efficient code) Why? Information theory:• Most efficient code assigns -log2P(X=i) bits to encode the message X=i• So, expected number of bits to code one random X is: # of possible values for XEntropyEntropy H(X) of a random variable XSpecific conditional entropy H(X|Y=v) of X given Y=v :Conditional entropy H(X|Y) of X given Y :Mututal information (aka information gain) of X and Y :Sample EntropySubset of Sfor which A=vGain(S,A) = mutual information between A and target class variable over sample SWhich Tree Should We Output?• ID3 performs heuristic search through space of decision trees• It stops at smallest acceptable tree. Why?Occam’s razor: prefer the simplest hypothesis that fits the dataSplit data into training and validation setCreate tree that classifies training set correctlyWhat you should know:• Well posed function approximation problems:– Instance space, X– Sample of labeled training data { <xi, yi>}– Hypothesis space, H = { f: XÆY }• Learning is a search/optimization problem over H– Various objective functions• minimize training error (0-1 loss) • among hypotheses that minimize training error, select shortest • Decision tree learning– Greedy top-down learning of decision trees (ID3, C4.5, ...)– Overfitting and tree/rule post-pruning– Extensions…Questions to think about (1)• Why use Information Gain to select attributes in decision trees? What other criteria seem reasonable, and what are the tradeoffs in making this choice?Questions to think about (2)• ID3 and C4.5 are heuristic algorithms that search through the space of decision trees. Why not just do an exhaustive search?Questions to think about (3)• Consider target function f: <x1,x2> Æ y, where x1 and x2 are real-valued, y is boolean. What is the set of decision surfaces describable with decision trees that use each attribute at most


View Full Document

CMU CS 10701 - Machine Learning, Decision Trees, Overfitting

Documents in this Course
lecture

lecture

12 pages

lecture

lecture

17 pages

HMMs

HMMs

40 pages

lecture

lecture

15 pages

lecture

lecture

20 pages

Notes

Notes

10 pages

Notes

Notes

15 pages

Lecture

Lecture

22 pages

Lecture

Lecture

13 pages

Lecture

Lecture

24 pages

Lecture9

Lecture9

38 pages

lecture

lecture

26 pages

lecture

lecture

13 pages

Lecture

Lecture

5 pages

lecture

lecture

18 pages

lecture

lecture

22 pages

Boosting

Boosting

11 pages

lecture

lecture

16 pages

lecture

lecture

20 pages

Lecture

Lecture

20 pages

Lecture

Lecture

39 pages

Lecture

Lecture

14 pages

Lecture

Lecture

18 pages

Lecture

Lecture

13 pages

Exam

Exam

10 pages

Lecture

Lecture

27 pages

Lecture

Lecture

15 pages

Lecture

Lecture

24 pages

Lecture

Lecture

16 pages

Lecture

Lecture

23 pages

Lecture6

Lecture6

28 pages

Notes

Notes

34 pages

lecture

lecture

15 pages

Midterm

Midterm

11 pages

lecture

lecture

11 pages

lecture

lecture

23 pages

Boosting

Boosting

35 pages

Lecture

Lecture

49 pages

Lecture

Lecture

22 pages

Lecture

Lecture

16 pages

Lecture

Lecture

18 pages

Lecture

Lecture

35 pages

lecture

lecture

22 pages

lecture

lecture

24 pages

Midterm

Midterm

17 pages

exam

exam

15 pages

Lecture12

Lecture12

32 pages

lecture

lecture

19 pages

Lecture

Lecture

32 pages

boosting

boosting

11 pages

pca-mdps

pca-mdps

56 pages

bns

bns

45 pages

mdps

mdps

42 pages

svms

svms

10 pages

Notes

Notes

12 pages

lecture

lecture

42 pages

lecture

lecture

29 pages

lecture

lecture

15 pages

Lecture

Lecture

12 pages

Lecture

Lecture

24 pages

Lecture

Lecture

22 pages

Midterm

Midterm

5 pages

mdps-rl

mdps-rl

26 pages

Load more
Download Machine Learning, Decision Trees, Overfitting
Our administrator received your request to download this document. We will send you the file to your email shortly.
Loading Unlocking...
Login

Join to view Machine Learning, Decision Trees, Overfitting and access 3M+ class-specific study document.

or
We will never post anything without your permission.
Don't have an account?
Sign Up

Join to view Machine Learning, Decision Trees, Overfitting 2 2 and access 3M+ class-specific study document.

or

By creating an account you agree to our Privacy Policy and Terms Of Use

Already a member?