Logistic Regression Machine Learning 10701 15781 Carlos Guestrin Carnegie Mellon University September 24th 2007 1 Carlos Guestrin 2005 2007 Generative v Discriminative classifiers Intuition Want to Learn h X a Y X features Y target classes Bayes optimal classifier P Y X Generative classifier e g Na ve Bayes Assume some functional form for P X Y P Y Estimate parameters of P X Y P Y directly from training data Use Bayes rule to calculate P Y X x This is a generative model Indirect computation of P Y X through Bayes rule But can generate a sample of the data P X y P y P X y Discriminative classifiers e g Logistic Regression Assume some functional form for P Y X Estimate parameters of P Y X directly from training data This is the discriminative model Directly learn P Y X But cannot obtain a sample of the data because P X is not available 2 Carlos Guestrin 2005 2007 1 Logistic function or Sigmoid Logistic Regression Learn P Y X directly Assume a particular functional form Sigmoid applied to a linear function of the data Z Features can be discrete or continuous 3 Carlos Guestrin 2005 2007 Understanding the sigmoid w0 2 w1 1 w0 0 w1 1 w0 0 w1 0 5 1 1 1 0 9 0 9 0 9 0 8 0 8 0 8 0 7 0 7 0 7 0 6 0 6 0 6 0 5 0 5 0 5 0 4 0 4 0 4 0 3 0 3 0 3 0 2 0 2 0 2 0 1 0 1 0 6 4 2 0 2 4 6 0 6 0 1 4 2 0 2 4 6 0 6 4 2 0 2 4 6 4 Carlos Guestrin 2005 2007 2 Logistic Regression a Linear classifier 1 0 9 0 8 0 7 0 6 0 5 0 4 0 3 0 2 0 1 0 6 4 2 0 2 4 6 5 Carlos Guestrin 2005 2007 Very convenient implies implies linear classification rule implies 6 Carlos Guestrin 2005 2007 3 What if we have continuous Xi Eg character recognition Xi is ith pixel Gaussian Na ve Bayes GNB Sometimes assume variance is independent of Y i e i or independent of Xi i e k or both i e 7 Carlos Guestrin 2005 2007 Example GNB for classifying mental states Mitchell et al 1 mm resolution 2 images per sec 15 000 voxels image non invasive safe 10 sec measures Blood Oxygen Level Dependent BOLD response Typical impulse response 8 Carlos Guestrin 2005 2007 4 Learned Bayes Models Means for P BrainActivity WordCategory Pairwise classification accuracy 85 People words Mitchell et al Animal words 9 Carlos Guestrin 2005 2007 Logistic regression v Na ve Bayes Consider learning f X Y where Could use a Gaussian Na ve Bayes classifier X is a vector of real valued features X1 Xn Y is boolean assume all Xi are conditionally independent given Y model P Xi Y yk as Gaussian N ik i model P Y as Bernoulli 1 What does that imply about the form of P Y X Cool 10 Carlos Guestrin 2005 2007 5 Derive form for P Y X for continuous Xi 11 Carlos Guestrin 2005 2007 Ratio of class conditional probabilities 12 Carlos Guestrin 2005 2007 6 Derive form for P Y X for continuous Xi 13 Carlos Guestrin 2005 2007 Gaussian Na ve Bayes v Logistic Regression Set of Gaussian Na ve Bayes parameters feature variance independent of class label Representation equivalence Set of Logistic Regression parameters But only in a special case GNB with class independent variances But what s the difference LR makes no assumptions about P X Y in learning Loss function Optimize different functions Obtain different solutions 14 Carlos Guestrin 2005 2007 7 Logistic regression for more than 2 classes Logistic regression in more general case where Y 2 Y1 YR learn R 1 sets of weights 15 Carlos Guestrin 2005 2007 Logistic regression more generally Logistic regression in more general case where Y 2 Y1 YR learn R 1 sets of weights for k R for k R normalization so no weights for this class Features can be discrete or continuous 16 Carlos Guestrin 2005 2007 8 Loss functions Likelihood v Conditional Likelihood Generative Na ve Bayes Loss function Data likelihood Discriminative models cannot compute P xj w But discriminative logistic regression loss function Conditional Data Likelihood Doesn t waste effort learning P X focuses on P Y X all that matters for classification 17 Carlos Guestrin 2005 2007 Expressing Conditional Log Likelihood 18 Carlos Guestrin 2005 2007 9 Maximizing Conditional Log Likelihood Good news l w is concave function of w no locally optimal solutions Bad news no closed form solution to maximize l w Good news concave functions easy to optimize 19 Carlos Guestrin 2005 2007 Optimizing concave function Gradient ascent Conditional likelihood for Logistic Regression is concave Find optimum with gradient ascent Gradient Learning rate 0 Update rule Gradient ascent is simplest of optimization approaches e g Conjugate gradient ascent much better see reading 20 Carlos Guestrin 2005 2007 10 Maximize Conditional Log Likelihood Gradient ascent 21 Carlos Guestrin 2005 2007 Gradient Descent for LR Gradient ascent algorithm iterate until change For i 1 n repeat 22 Carlos Guestrin 2005 2007 11 That s all M C LE How about MAP One common approach is to define priors on w Corresponds to Regularization Normal distribution zero mean identity covariance Pushes parameters towards zero Helps avoid very large weights and overfitting More on this later in the semester MAP estimate 23 Carlos Guestrin 2005 2007 M C AP as Regularization Penalizes high weights also applicable in linear regression 24 Carlos Guestrin 2005 2007 12 Gradient of M C AP 25 Carlos Guestrin 2005 2007 MLE vs MAP Maximum conditional likelihood estimate Maximum conditional a posteriori estimate 26 Carlos Guestrin 2005 2007 13 Na ve Bayes vs Logistic Regression Consider Y boolean Xi continuous X X1 Xn Number of parameters NB 4n 1 LR n 1 Estimation method NB parameter estimates are uncoupled LR parameter estimates are coupled 27 Carlos Guestrin 2005 2007 G Na ve Bayes vs Logistic Regression 1 Ng Jordan 2002 Generative and Discriminative classifiers Asymptotic comparison training examples infinity when model correct GNB LR produce identical classifiers when model incorrect LR is less biased does not assume conditional independence therefore LR expected to outperform GNB 28 Carlos Guestrin 2005 2007 14 G Na ve Bayes vs Logistic Regression 2 Ng Jordan 2002 Generative and Discriminative classifiers Non asymptotic analysis convergence rate of parameter estimates n of attributes in X Size of training data to get close to infinite data solution GNB needs O log n samples LR needs O n samples GNB converges more quickly to its perhaps less helpful asymptotic estimates 29 Carlos Guestrin 2005 2007 Na ve bayes Logistic Regression Some experiments from UCI data sets 30 Carlos Guestrin 2005 2007 15 What you should know about Logistic Regression LR Gaussian Na ve Bayes
View Full Document
Unlocking...