INTRODUCTIONTOMachineLearningETHEM ALPAYDIN© The MIT Press, [email protected]://www.cmpe.boun.edu.tr/~ethem/i2mlLecture Slides forCHAPTER15:CombiningMultipleLearnersLecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.0)3Rationale No Free Lunch thm: There is no algorithm that is always the most accurate Generate a group of base-learners which when combined has higher accuracy Different learners use different Algorithms Hyperparameters Representations (Modalities) Training sets SubproblemsLecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.0)4Voting Linear combination ClassificationLecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.0)5 Bayesian perspective: If djare iid Bias does not change, variance decreases by L Average over randomnessLecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.0)6Error-Correcting Output Codes K classes; L problems (Dietterich and Bakiri, 1995) Code matrix W codes classes in terms of learners One per classL=K PairwiseL=K(K-1)/2Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.0)7 Full code L=2(K-1)-1 With reasonable L, find W such that the Hamming distance btw rows and columns are maximized. Voting scheme Subproblems may be more difficult than one-per-KLecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.0)8Bagging Use bootstrapping to generate L training sets and train one base-learner with each (Breiman, 1996) Use voting (Average or median with regression) Unstable algorithms profit from baggingLecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.0)9AdaBoostGenerate a sequence of base-learners each focusing on previous one’s errors(Freund and Schapire, 1996)Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.0)10Mixture of ExpertsVoting where weights are input-dependent (gating)(Jacobs et al., 1991)Experts or gating can be nonlinearLecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.0)11Stacking Combiner f () is another learner (Wolpert, 1992)Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.0)12CascadingUse djonly if preceding ones are not confidentCascade learners in order of
View Full Document