Predicting Real valued outputs an introduction to Regression Note to other teachers and users of these slides Andrew would be delighted if you found this source material useful in giving your own lectures Feel free to use these slides verbatim or to modify them to fit your own needs PowerPoint originals are available If you make use of a significant portion of these slides in your own lecture please include this message or the following link to the source repository of Andrew s tutorials http www cs cmu edu awm tutorials Comments and corrections gratefully received Andrew W Moore Professor School of Computer Science Carnegie Mellon University www cs cmu edu awm awm cs cmu edu 412 268 7599 l ateria red m s e d r l Net is reo This he Neura Favorite t e h t from d ms re an orith lectu ssion Alg e Regr re lectu Copyright 2001 2003 Andrew W Moore SingleParameter Linear Regression Copyright 2001 2003 Andrew W Moore 2 1 Linear Regression DATASET inputs w 1 outputs x1 1 y1 1 x2 3 y2 2 2 x3 2 y3 2 x4 1 5 y4 1 9 x5 4 y5 3 1 Linear regression assumes that the expected value of the output given an input E y x is linear Simplest case Out x wx for some unknown w Given the data we can estimate w Copyright 2001 2003 Andrew W Moore 3 1 parameter linear regression Assume that the data is formed by yi wxi noisei where the noise signals are independent the noise has a normal distribution with mean 0 and unknown variance 2 p y w x has a normal distribution with mean wx variance 2 Copyright 2001 2003 Andrew W Moore 4 2 Bayesian Linear Regression p y w x Normal mean wx var 2 We have a set of datapoints x1 y1 x2 y2 xn yn which are EVIDENCE about w We want to infer w from the data p w x1 x2 x3 xn y1 y2 yn You can use BAYES rule to work out a posterior distribution for w given the data Or you could do Maximum Likelihood Estimation Copyright 2001 2003 Andrew W Moore 5 Maximum likelihood estimation of w Asks the question For which value of w is this data most likely to have happened For what w is p y1 y2 yn x1 x2 x3 xn w maximized For what w is n p y w x maximized i i i 1 Copyright 2001 2003 Andrew W Moore 6 3 For what w is n p y i w x i maximized i 1 For what w isn 1 exp 2 i 1 yi wxi 2 maximized For what w is n i 1 For what w is 2 1 y wx i i maximized 2 2 n y i 1 i wx i minimized Copyright 2001 2003 Andrew W Moore 7 Linear Regression The maximum likelihood w is the one that minimizes sumof squares of residuals E w w yi wxi 2 i yi 2 xi yi w 2 i x w 2 2 i We want to minimize a quadratic function of w Copyright 2001 2003 Andrew W Moore 8 4 Linear Regression Easy to show the sum of squares is minimized when xy w x i i 2 i The maximum likelihood model is Out x wx We can use it for prediction Copyright 2001 2003 Andrew W Moore 9 Linear Regression Easy to show the sum of squares is minimized when w x y x i i 2 i The maximum likelihood model is Out x wx We can use it for prediction Copyright 2001 2003 Andrew W Moore p w w Note In Bayesian stats you d have ended up with a prob dist of w And predictions would have given a prob dist of expected output Often useful to know your confidence Max likelihood can give some kinds of confidence too 10 5 Multivariate Linear Regression Copyright 2001 2003 Andrew W Moore 11 Multivariate Regression What if the inputs are vectors 3 6 4 x2 2 d input example 5 8 10 x1 Dataset has form x1 x2 x3 xR Copyright 2001 2003 Andrew W Moore y1 y2 y3 yR 12 6 Multivariate Regression Write matrix X and Y thus x1 x11 x x 2 21 x M x R xR1 x12 x22 xR 2 x1m y1 y x2 m y 2 M M xRm yR there are R datapoints Each input has m components The linear regression model assumes a vector w such that Out x wTx w1x 1 w2x 2 wmx D The max likelihood w is w XTX 1 XTY Copyright 2001 2003 Andrew W Moore 13 Multivariate Regression Write matrix X and Y thus x1 x11 x x 2 21 x M x R xR1 x12 x22 xR 2 x1m y1 y x2 m y 2 M M xRm yR IMPORTANT EXERCISE there are R datapoints Each input hasPROVE m components IT The linear regression model assumes a vector w such that Out x wTx w1x 1 w2x 2 wmx D The max likelihood w is w XTX 1 XTY Copyright 2001 2003 Andrew W Moore 14 7 Multivariate Regression con t The max likelihood w is w XTX 1 XTY R XTX is an m x m matrix i j th elt is x x ki kj k 1 R XTY is an m element vector i th elt x k 1 y ki k Copyright 2001 2003 Andrew W Moore 15 Constant Term in Linear Regression Copyright 2001 2003 Andrew W Moore 16 8 What about a constant term We may expect linear data that does not go through the origin Statisticians and Neural Net Folks all agree on a simple obvious hack Can you guess Copyright 2001 2003 Andrew W Moore 17 The constant term The trick is to create a fake input X0 that always takes the value 1 X1 X2 Y X0 X1 X2 Y 2 3 5 4 4 5 16 17 20 1 1 1 2 3 5 4 4 5 16 17 20 Before After Y w1X1 w2X2 Y w0X0 w1X1 w2X2 w0 w1X1 w2X2 has to be a poor model Copyright 2001 2003 Andrew W Moore In this example You should be able to see the MLE w0 w1 and w2 by inspection has a fine constant term 18 9 H et icity t s a ced s o r e Linear Regression with varying noise Copyright 2001 2003 Andrew W Moore 19 Regression with varying noise Suppose you know the variance of the noise that was added to each datapoint xi 1 2 2 3 yi 1 1 3 2 Assume y 3 i2 4 1 1 4 4 1 4 2 y 2 1 2 1 y 1 1 2 2 y 0 x 0 x 1 x 2 yi N wxi i2 Copyright 2001 2003 Andrew W Moore x 3 LE eM h t fw at s W h at e o m esti 20 10 MLE estimation with varying noise argmax log p y y y 1 2 R …
View Full Document
Unlocking...