Review•Multivariate Gaussian!N(X | ", #) = (1/!|2pi #|) exp{–0.5 (x–")T #–1 (x–")}1Multivariate Gaussians!! !"#" !$!#%&##%&""%&!!%&$$%&'"3 = [-.5; 0]"1 = [0; 3]"2 = [2; 1]#i = UDiUTU = !2 / 2!2 / 2!2 / 2–!2 / 2D2 = diag([.02 .26])D1 = diag([.2 .08])D3 = diag([.1 .1])note UTU = I2Review•Naïve Bayes, Gaussian NB, Fisher model:!all lead to linear discriminants!w0 + "j wj Xj # 0 or w0 + wTX # 0!formulas for w0, w depend on which model3Fisher linear discriminant!2 2 6!2024figure from book4Fisher w/ bad #figure from book!2 2 6!20245Features•Can generalize to use features of X:!w0 + "j wj $j(X) # 0!$j(X) are features•Why might we want to do so?6Use of $j(X)x1x2!1 0 1!1017Use of $j(X)φ1φ20 0.5 100.518Lots of discriminants•One of most important types of classifier•Consequently, many ways to train LDs!based on different assumptions about data•We saw 3 so far! •Another one: coming up soon9Application: USPS digits10Class means for 0 and 1Class means5 10 15 20 255101520255 10 15 20 2551015202511A linear discriminantA linear separator5 10 15 20 2551015202512Application: fMRI classification“dog”brain: Patrick J. Lynch, C. Carl Jaffe (CC att. 2.5)fMRI: Mitchell et alAnother example of information integration7co-occurs(dog,cat) = 1co-occurs(dog,walk) = 1co-occurs(dog,physics) = 0co-occurs(dog,cupcake) = 0:-):-));->fMRIfMRIfMRI(Word + Picture) stimulusDogText CorpusA fMRI brain imaging experiment.Augmented with side information about the stimulus. Brain activityMitchell et al. (2008) Predicting Human Brain Activity Associated with the Meaning of Nouns. Science.13Class probability•We showed:!log P(Y=1 | X) – log P(Y=0 | X) = •This implies!log P(Y = 1) = 14Sigmoid: %(z) = 1/(1+exp(–z))!! !" # " !##$"#$!#$%#$&'()15NB = MLE (or MAP)•P(Y=1 | X) = %(z)!z = w0 + " wj Xj•NB is one algorithm for finding w•NB = maximum likelihood in this model!arg max16Conditional likelihood•Another: maximum conditional likelihood!given data (X1, Y1), …, (XN, YN)!arg max•Same model, different training criterion•Cond. MLE for logistic linear discriminant: logistic regression17Discussion•maxw P(X, Y | w) vs. maxw P(Y | X, w)•We’ve seen cond. MLE before:•Why choose one?!MLE:!cond. MLE:18Generative vs discriminative•Same trick works for any graphical model!if we know we’re always going to be asking same query (Y given X1…XM), optimize for it!max•Can improve performance, but also more risk of overfitting19Logistic regression•given data (X1, Y1), …, (XN, YN)•arg maxw $i P(Yi | Xi, w)20Neg. log likelihood!! !" !# $ # " !$%#&"'!()*+!,-./0)1/2/*+21Example!! !"#" !$##%!#%&#%'#%("22Weight spacew0w1–5 10!! !" # " !##$%&!! !" # " !##$%&!! !" # " !##$%&!! !" # " !##$%&!! !" # " !##$%&!! !" # " !##$%&!! !" # " !##$%&!! !" # " !##$%&!! !" # " !##$%&!! !" # " !##$%&!! !" # " !##$%&!! !" # " !##$%&!! !"#" !##$%&!! !"#" !##$%&!! !"#" !##$%&!! !"#" !##$%&–6423(X, Y) = (1.2, –1)!!"!#"!$!%!&"&%"!#"#!'"'#!()*+,-24(X, Y) = (1.2, –1)!"!#!$ !% " % $ & '!&!$!%"%25(X, Y) = (–1, 1)!"!#!$ !% " % $ & '!&!$!%"%26(X, Y) = (2, 1)!"!#!$ !% " % $ & '!&!$!%"%27–log(P(Y1..3 | X1..3, W))!"!#!$ !% " % $ &
View Full Document
Unlocking...