DOC PREVIEW
CMU CS 10701 - DTrees And Over fitting-1-11-2011_final

This preview shows page 1-2-19-20 out of 20 pages.

Save
View full document
View full document
Premium Document
Do you want full access? Go Premium and unlock all 20 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 20 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 20 pages.
Access to all documents
Download any document
Ad free experience
View full document
Premium Document
Do you want full access? Go Premium and unlock all 20 pages.
Access to all documents
Download any document
Ad free experience
Premium Document
Do you want full access? Go Premium and unlock all 20 pages.
Access to all documents
Download any document
Ad free experience

Unformatted text preview:

1 Machine Learning 10-701 Tom M. Mitchell Machine Learning Department Carnegie Mellon University January 11, 2011 Today: • What is machine learning? • Decision tree learning • Course logistics Readings: • “The Discipline of ML” • Mitchell, Chapter 3 • Bishop, Chapter 14.4 Machine Learning: Study of algorithms that • improve their performance P • at some task T • with experience E well-defined learning task: <P,T,E>2 Learning to Predict Emergency C-Sections 9714 patient records, each with 215 features [Sims et al., 2000] Learning to detect objects in images Example training images for each orientation (Prof. H. Schneiderman)3 Learning to classify text documents Company home page vs Personal home page vs University home page vs … Reading a noun (vs verb) [Rustandi et al., 2005]4 Machine Learning - Practice Object recognition Mining Databases Speech Recognition Control learning • Supervised learning • Bayesian networks • Hidden Markov models • Unsupervised clustering • Reinforcement learning • .... Text analysis Machine Learning - Theory PAC Learning Theory # examples (m) representational complexity (H) error rate (ε) failure probability (δ) Other theories for • Reinforcement skill learning • Semi-supervised learning • Active student querying • … … also relating: • # of mistakes during learning • learner’s query strategy • convergence rate • asymptotic performance • bias, variance (supervised concept learning)5 Animal learning (Cognitive science, Psychology, Neuroscience) Machine learning Statistics Computer science Adaptive Control Theory Evolution Economics and Organizational Behavior Machine Learning in Computer Science • Machine learning already the preferred approach to – Speech recognition, Natural language processing – Computer vision – Medical outcomes analysis – Robot control – … • This ML niche is growing (why?) All software apps. ML apps.6 • Machine learning already the preferred approach to – Speech recognition, Natural language processing – Computer vision – Medical outcomes analysis – Robot control – … • This ML niche is growing – Improved machine learning algorithms – Increased data capture, networking, new sensors – Software too complex to write by hand – Demand for self-customization to user, environment All software apps. ML apps. Machine Learning in Computer Science Function Approximation and Decision tree learning7 Function approximation Problem Setting: • Set of possible instances X!• Unknown target function f : XY!• Set of function hypotheses H={ h | h : XY }!Input: • Training examples {<x(i),y(i)>} of unknown target function f Output: • Hypothesis h ∈ H that best approximates target function f!superscript: ith training example!Each internal node: test one attribute Xi Each branch from a node: selects one value for Xi Each leaf node: predict Y (or P(Y|X ∈ leaf)) A Decision tree for F: <Outlook, Humidity, Wind, Temp>  PlayTennis?8 Problem Setting: • Set of possible instances X – each instance x in X is a feature vector – e.g., <Humidity=low, Wind=weak, Outlook=rain, Temp=hot> • Unknown target function f : XY!– Y is discrete valued • Set of function hypotheses H={ h | h : XY }!– each hypothesis h is a decision tree – trees sorts x to leaf, which assigns y Decision Tree Learning Decision Tree Learning Problem Setting: • Set of possible instances X – each instance x in X is a feature vector x = < x1, x2 … xn> • Unknown target function f : XY!– Y is discrete valued • Set of function hypotheses H={ h | h : XY }!– each hypothesis h is a decision tree Input: • Training examples {<x(i),y(i)>} of unknown target function f!Output: • Hypothesis h ∈ H that best approximates target function f!9 Decision Trees Suppose X = <X1,… Xn> where Xi are boolean variables How would you represent Y = X2 X5 ? Y = X2 ∨ X5 How would you represent X2 X5 ∨ X3X4(¬X1) !10 node = Root [ID3, C4.5, Quinlan] Entropy Entropy H(X) of a random variable X H(X) is the expected number of bits needed to encode a randomly drawn value of X (under most efficient code) Why? Information theory: • Most efficient code assigns -log2P(X=i) bits to encode the message X=i • So, expected number of bits to code one random X is: # of possible values for X!11 Sample Entropy Entropy Entropy H(X) of a random variable X Specific conditional entropy H(X|Y=v) of X given Y=v : Conditional entropy H(X|Y) of X given Y : Mututal information (aka Information Gain) of X and Y :12 Information Gain is the mutual information between input attribute A and target variable Y Information Gain is the expected reduction in entropy of target variable Y for data sample S, due to sorting on variable A1314 Decision Tree Learning Applet • http://www.cs.ualberta.ca/%7Eaixplore/learning/DecisionTrees/Applet/DecisionTreeApplet.html Which Tree Should We Output? • ID3 performs heuristic search through space of decision trees • It stops at smallest acceptable tree. Why? Occam’s razor: prefer the simplest hypothesis that fits the data15 Why Prefer Short Hypotheses? (Occam’s Razor) Arguments in favor: Arguments opposed: Why Prefer Short Hypotheses? (Occam’s Razor) Argument in favor: • Fewer short hypotheses than long ones  a short hypothesis that fits the data is less likely to be a statistical coincidence  highly probable that a sufficiently complex hypothesis will fit the data Argument opposed: • Also fewer hypotheses with prime number of nodes and attributes beginning with “Z” • What’s so special about “short” hypotheses?161718 Split data into training and validation set!Create tree that classifies training set correctly!1920 What you should know: • Well posed function approximation problems: – Instance space, X – Sample of labeled training data { <x(i), y(i)>} – Hypothesis space, H = { f: XY } • Learning is a search/optimization problem over H – Various objective functions • minimize training error (0-1 loss) • among hypotheses that minimize training error, select smallest (?) • Decision tree learning – Greedy top-down learning of decision trees (ID3,


View Full Document

CMU CS 10701 - DTrees And Over fitting-1-11-2011_final

Documents in this Course
lecture

lecture

12 pages

lecture

lecture

17 pages

HMMs

HMMs

40 pages

lecture

lecture

15 pages

lecture

lecture

20 pages

Notes

Notes

10 pages

Notes

Notes

15 pages

Lecture

Lecture

22 pages

Lecture

Lecture

13 pages

Lecture

Lecture

24 pages

Lecture9

Lecture9

38 pages

lecture

lecture

26 pages

lecture

lecture

13 pages

Lecture

Lecture

5 pages

lecture

lecture

18 pages

lecture

lecture

22 pages

Boosting

Boosting

11 pages

lecture

lecture

16 pages

lecture

lecture

20 pages

Lecture

Lecture

20 pages

Lecture

Lecture

39 pages

Lecture

Lecture

14 pages

Lecture

Lecture

18 pages

Lecture

Lecture

13 pages

Exam

Exam

10 pages

Lecture

Lecture

27 pages

Lecture

Lecture

15 pages

Lecture

Lecture

24 pages

Lecture

Lecture

16 pages

Lecture

Lecture

23 pages

Lecture6

Lecture6

28 pages

Notes

Notes

34 pages

lecture

lecture

15 pages

Midterm

Midterm

11 pages

lecture

lecture

11 pages

lecture

lecture

23 pages

Boosting

Boosting

35 pages

Lecture

Lecture

49 pages

Lecture

Lecture

22 pages

Lecture

Lecture

16 pages

Lecture

Lecture

18 pages

Lecture

Lecture

35 pages

lecture

lecture

22 pages

lecture

lecture

24 pages

Midterm

Midterm

17 pages

exam

exam

15 pages

Lecture12

Lecture12

32 pages

lecture

lecture

19 pages

Lecture

Lecture

32 pages

boosting

boosting

11 pages

pca-mdps

pca-mdps

56 pages

bns

bns

45 pages

mdps

mdps

42 pages

svms

svms

10 pages

Notes

Notes

12 pages

lecture

lecture

42 pages

lecture

lecture

29 pages

lecture

lecture

15 pages

Lecture

Lecture

12 pages

Lecture

Lecture

24 pages

Lecture

Lecture

22 pages

Midterm

Midterm

5 pages

mdps-rl

mdps-rl

26 pages

Load more
Download DTrees And Over fitting-1-11-2011_final
Our administrator received your request to download this document. We will send you the file to your email shortly.
Loading Unlocking...
Login

Join to view DTrees And Over fitting-1-11-2011_final and access 3M+ class-specific study document.

or
We will never post anything without your permission.
Don't have an account?
Sign Up

Join to view DTrees And Over fitting-1-11-2011_final 2 2 and access 3M+ class-specific study document.

or

By creating an account you agree to our Privacy Policy and Terms Of Use

Already a member?