Instance based Learning Machine Learning 10701 15781 Carlos Guestrin Carnegie Mellon University February 19th 2007 2005 2007 Carlos Guestrin 1 Why not just use Linear Regression 2005 2007 Carlos Guestrin 2 Using data to predict new data 2005 2007 Carlos Guestrin 3 Nearest neighbor 2005 2007 Carlos Guestrin 4 Univariate 1 Nearest Neighbor Given datapoints x1 y1 x2 y2 xN yN where we assume yi f xi for some unknown function f Given query point xq your job is to predict q Nearest Neighbor 1 Find the closest xi in our set of datapoints y f x i nn argmin xi xq i Predict y yi nn H theere t da clo his tap se is oin st t Here s a dataset with one input one output and four datapoints is s t i th es e s t er clo oin H he ap t at d H th ere da e c th ta los is po e is int st 2 y Here this is the closest datapoint x 2005 2007 Carlos Guestrin 5 1 Nearest Neighbor is an example of Instance based learning A function approximator that has been around since about 1910 x1 x2 x3 To make a prediction search database for similar datapoints and fit with the local points xn y1 y2 y3 yn Four things make a memory based learner A distance metric How many nearby neighbors to look at A weighting function optional How to fit with the local points 2005 2007 Carlos Guestrin 6 1 Nearest Neighbor Four things make a memory based learner 1 A distance metric Euclidian and many more 2 How many nearby neighbors to look at One 3 A weighting function optional Unused 4 How to fit with the local points Just predict the same output as the nearest neighbor 2005 2007 Carlos Guestrin 7 Multivariate 1 NN examples Regression Classification 2005 2007 Carlos Guestrin 8 Multivariate distance metrics Suppose the input vectors x1 x2 xn are two dimensional x1 x11 x12 x2 x21 x22 xN xN1 xN2 One can draw the nearest neighbor regions in input space Dist xi xj xi1 xj1 2 xi2 xj2 2 Dist xi xj xi1 xj1 2 3xi2 3xj2 2 The relative scalings in the distance metric affect region shapes 2005 2007 Carlos Guestrin 9 Euclidean distance metric Or equivalently where D x x x x 2 2 i i i i D x x x x T x x Other Metrics Mahalanobis Rank based Correlation based 2005 2007 Carlos Guestrin 10 Notable distance metrics and their level sets Scaled Euclidian L2 L1 norm absolute Mahalanobis here on the previous slide is not necessarily diagonal but is symmetric L max norm 2005 2007 Carlos Guestrin 11 Consistency of 1 NN Consider an estimator fn trained on n examples e g 1 NN neural nets regression Estimator is consistent if true error goes to zero as amount of data increases e g for no noise data consistent if Regression is not consistent Representation bias 1 NN is consistent under some mild fineprint What about variance 2005 2007 Carlos Guestrin 12 1 NN overfits 2005 2007 Carlos Guestrin 13 k Nearest Neighbor Four things make a memory based learner 1 A distance metric Euclidian and many more 2 How many nearby neighbors to look at k 1 A weighting function optional Unused 2 How to fit with the local points Just predict the average output among the k nearest neighbors 2005 2007 Carlos Guestrin 14 k Nearest Neighbor here k 9 K nearest neighbor for function fitting smoothes away noise but there are clear deficiencies What can we do about all the discontinuities that k NN gives us 2005 2007 Carlos Guestrin 15 Weighted k NNs Neighbors are not all the same 2005 2007 Carlos Guestrin 16 Kernel regression Four things make a memory based learner 1 A distance metric Euclidian and many more 2 How many nearby neighbors to look at All of them 3 A weighting function optional wi exp D xi query 2 Kw2 Nearby points to the query are weighted strongly far points weakly The KW parameter is the Kernel Width Very important 4 How to fit with the local points Predict the weighted average of the outputs predict wiyi wi 2005 2007 Carlos Guestrin 17 Weighting functions wi exp D xi query 2 Kw2 Typically optimize Kw using gradient descent Our examples use Gaussian 2005 2007 Carlos Guestrin 18 Kernel regression predictions KW 10 KW 20 KW 80 Increasing the kernel width Kw means further away points get an opportunity to influence you As Kw the prediction tends to the global average 2005 2007 Carlos Guestrin 19 Kernel regression on our test cases KW 1 32 of x axis width KW 1 32 of x axis width KW 1 16 axis width Choosing a good Kw is important Not just for Kernel Regression but for all the locally weighted learners we re about to see 2005 2007 Carlos Guestrin 20 Kernel regression can look bad KW Best KW Best KW Best Time to try something more powerful 2005 2007 Carlos Guestrin 21 Locally weighted regression Kernel regression Take a very very conservative function approximator called AVERAGING Locally weight it Locally weighted regression Take a conservative function approximator called LINEAR REGRESSION Locally weight it 2005 2007 Carlos Guestrin 22 Locally weighted regression Four things make a memory based learner A distance metric Any How many nearby neighbors to look at All of them A weighting function optional Kernels wi exp D xi query 2 Kw2 How to fit with the local points General weighted regression 2 argmin wk yk T xk N 2 k 1 2005 2007 Carlos Guestrin 23 How LWR works Query Linear regression Locally weighted regression Same parameters for all queries Solve weighted linear regression for each query X X X Y T 1 WX WX WXT WY T T 2005 2007 Carlos Guestrin w1 0 W 0 0 1 0 0 0 w2 0 0 0 O 0 0 0 wn 24 Another view of LWR Image from Cohn D A Ghahramani Z and Jordan M I 1996 Learning with Statistical Models JAIR Volume 4 pages 25 129 145 2005 2007 Carlos Active Guestrin LWR on our test cases KW 1 16 of x axis width KW 1 32 of x axis width 2005 2007 Carlos Guestrin KW 1 8 of x axis width 26 Locally weighted polynomial regression Kernel Regression Kernel width KW at optimal level LW Linear Regression Kernel width KW at optimal level LW Quadratic Regression Kernel width KW at optimal level KW 1 100 x axis KW 1 40 x axis KW 1 15 x axis Local quadratic regression is easy just add quadratic terms to the WXTWX matrix As the regression degree increases the kernel width can increase without introducing bias 2005 2007 Carlos Guestrin 27 Curse of dimensionality for instance based learning Must store and retreve all data Most real work done during testing For every test sample must search through all dataset very slow We ll see fast methods for dealing with large datasets Instance based learning often poor with noisy or irrelevant features 2005 2007 Carlos …
View Full Document
Unlocking...