Review•Multivariate Gaussian‣N(X | μ, Σ) = (1/√|2pi Σ|) exp{–0.5 (x–μ)T Σ–1 (x–μ)}1Multivariate Gaussians!! !"#" !$!#%&##%&""%&!!%&$$%&'μ3 = [-.5; 0]μ1 = [0; 3]μ2 = [2; 1]Σi = UDiUTU = √2 / 2√2 / 2√2 / 2–√2 / 2D2 = diag([.02 .26])D1 = diag([.2 .08])D3 = diag([.1 .1])note UTU = I2Review•Naïve Bayes, Gaussian NB, Fisher model:‣all lead to linear discriminants‣w0 + ∑j wj Xj ≥ 0 or w0 + wTX ≥ 0‣formulas for w0, w depend on which model3Fisher linear discriminant!2 2 6!2024figure from book4Fisher w/ bad Σfigure from book!2 2 6!20245Features•Can generalize to use features of X:‣w0 + ∑j wj ϕj(X) ≥ 0‣ϕj(X) are features•Why might we want to do so?6Use of ϕj(X)x1x2!1 0 1!1017Use of ϕj(X)φ1φ20 0.5 100.518Lots of discriminants•One of most important types of classifier•Consequently, many ways to train LDs‣based on different assumptions about data•We saw 3 so far‣ •Another one: coming up soon9Class probability•We showed:‣log P(Y=1 | X) – log P(Y=0 | X) = •This implies‣log P(Y = 1) = 10Sigmoid: σ(z) = 1/(1+exp(–z))!! !" # " !##$"#$!#$%#$&'()11NB = MLE (or MAP)•P(Y=1 | X) = σ(z)‣z = w0 + ∑ wj Xj•NB is one algorithm for finding w•NB = maximum likelihood in this model‣arg max12Conditional likelihood•Another: maximum conditional likelihood‣given data (X1, Y1), …, (XN, YN)‣arg max•Same model, different training criterion•Cond. MLE for logistic linear discriminant: logistic regression13Discussion•maxw P(X, Y | w) vs. maxw P(Y | X, w)•We’ve seen cond. MLE before:•Why choose one?‣MLE:‣cond. MLE:14Generative vs discriminative•Same trick works for any graphical model‣if we know we’re always going to be asking same query (Y given X1…XM), optimize for it‣max•Can improve performance, but also more risk of overfitting15Logistic regression•given data (X1, Y1), …, (XN, YN)•arg maxw ∏i P(Yi | Xi, w)16Neg. log likelihood!! !" !# $ # " !$%#&"'!()*+!,-./0)1/2/*+17Example!! !"#" !$##%!#%&#%'#%("18Weight spacew0w1–5 10!! !" # " !##$%&!! !" # " !##$%&!! !" # " !##$%&!! !" # " !##$%&!! !" # " !##$%&!! !" # " !##$%&!! !" # " !##$%&!! !" # " !##$%&!! !" # " !##$%&!! !" # " !##$%&!! !" # " !##$%&!! !" # " !##$%&!! !"#" !##$%&!! !"#" !##$%&!! !"#" !##$%&!! !"#" !##$%&–6419(X, Y) = (1.2, –1)!!"!#"!$!%!&"&%"!#"#!'"'#!()*+,-20(X, Y) = (1.2, –1)!"!#!$ !% " % $ & '!&!$!%"%21(X, Y) = (–1, 1)!"!#!$ !% " % $ & '!&!$!%"%22(X, Y) = (2, 1)!"!#!$ !% " % $ & '!&!$!%"%23–log(P(Y1..3 | X1..3, W))!"!#!$ !% " % $ & '!&!$!%"%24Generalization: multiple classes•One weight vector per class: Y ∈ {1,2,…,C}‣P(Y=k) = ‣Zk = •In 2-class case:25Multiclass example!6 !4 !2 0 2 4 6!6!4!20246figure from book26Conditional MAP logistic regression•P(Y | X, W) = ‣Z = •As in linear regression, can put prior on W‣common priors: L2 (ridge), L1 (sparsity)•maxw P(W=w | X, Y)27Software•Logistic regression software is easily available: most stats packages provide it‣e.g., glm function in R‣or, http://www.cs.cmu.edu/~ggordon/IRLS-example/•Most common algorithm: Newton’s method on log-likelihood (or L2-penalized version)‣called “iteratively reweighted least squares”‣for L1, slightly harder (less software available)28Bayesian regression•In linear and logistic regression, we’ve looked at ‣conditional MLE: maxw P(Y | X, w)‣conditional MAP: maxw P(W=w | X, Y)•But of course, a true Bayesian would turn up nose at both‣why?29Sample from posterior!"!#!$ " $ #"!%!&!'"'&30Predictive distribution!!"!#" "#"!"""$!"$%"$&"$'#31Overfitting•True Bayesian inference never leads to overfitting‣may still lead to bad results for other reasons!‣e.g., not enough data, bad model class, …•Overfitting is an indicator that the MLE or MAP approximation is a bad
View Full Document
Unlocking...