Bayes Classifier Linear Regression 10701 15781 Recitation January 29 2008 Parts of the slides are from previous years recitation and lecture notes and from Prof Andrew Moore s data mining tutorials Classification and Regression Classification Goal Learn the underlying function f X features Y class or category e g words spam or not spam Regression f X features Y continuous values e g GPA salary Supervised Classification How to find an unknown function f X Y features class or equivalently P Y X f x arg max k P Y k X x Classifier 1 Find P X Y P Y and use Bayes rule generative Find P Y X directly discriminative 2 Classification Learn P Y X 1 Bayes rule P Y X P X Y P Y P X P X Y P Y Learn P X Y P Y Generative classifier 2 Learn P Y X directly Discriminative to be covered later in class e g logistic regression Generative Classifier Bayes Classifier Learn P X Y P Y e g email classification problem 3 classes for Y spam not spam maybe 10 000 binary features for X Cash Rolex How many parameters do we have P Y P X Y Generative learning Na ve Bayes Introduce conditional independence P X1 X2 Y P X1 Y P X2 Y P Y X P X Y P Y P X for X Xi Xn P X1 Y P Xn Y P Y P X prodi P Xi Y P Y P X Learn P X1 Y P Xn Y P Y instead of learning P X1 Xn Y directly Na ve Bayes 3 classes for Y spam not spam maybe 10 000 binary features for X Cash Rolex Now how many parameters P Y P X Y fewer parameters simpler less likely to overfit Full Bayes vs Na ve Bayes P Y 1 X1 X2 0 1 XOR X1 X2 Y 1 0 1 0 1 1 1 1 0 0 0 0 Full Bayes P Y 1 P X1 X2 0 1 Y 1 Na ve Bayes P Y 1 P X1 X2 0 1 Y 1 Regression Prediction of continuous variables e g I want to predict salaries from GPA I can regress that Learn the mapping f X Y f x i hi x i Model is linear in the parameters some noise linear regression Assume Gaussian noise Learn MLE 1 parameter linear regression Normal linear regression Y X N 0 2 or equivalently Y N X 2 MLE MLE 2 Multivariate linear regression What if the inputs are vectors Write matrix X and Y x1 x11 x x x 2 21 x n xn1 x12 x22 xn 2 x1k y1 y x2 k y 2 xnk yn n data points k features for each data MLE X T X 1 X TY Constant term We may expect linear data that does not go through the origin Trick The constant term Regression another example Assume the following model to fit the data The model has one unknown parameter to be learned from data Y N log X 1 A maximum likelihood estimation of
View Full Document
Unlocking...