Neural Networks Required reading Neural nets Mitchell chapter 4 Optional reading Bias Variance error decomposition Bishop 9 1 9 2 Machine Learning 10 701 Tom M Mitchell Center for Automated Learning and Discovery Carnegie Mellon University October 4 2005 Today Finish up MLE vs MAP for logisitic regression Generative Discriminative classifiers Artificial neural networks MLE vs MAP Maximum conditional likelihood estimate Maximum a posteriori estimate Gaussian P W N 0 ln P W c i wi2 MLE vs MAP Maximum conditional likelihood estimate Maximum a posteriori estimate Gaussian P W N 1 n ln P W c i wi i 2 Generative vs Discriminative Classifiers Training classifiers involves estimating f X Y or P Y X Generative classifiers Assume some functional form for P X Y P X Estimate parameters of P X Y P X from training data Use Bayes rule to calculate P Y X xi Discriminative classifiers Assume some functional form for P Y X Estimate parameters of P Y X from training data Na ve Bayes vs Logistic Regression Consider Y boolean Xi continuous X X1 Xn Number of parameters NB 4n 1 LR n 1 Estimation method NB parameter estimates are uncoupled LR parameter estimates are coupled Na ve Bayes vs Logistic Regression Ng Jordan 2002 Generative and Discriminative classifiers Asymptotic comparison training examples infinity when model correct GNB LR produce identical classifiers when model incorrect LR is less biased does not assume cond indep therefore expected to outperform GNB Na ve Bayes vs Logistic Regression Generative and Discriminative classifiers Non asymptotic analysis see Ng Jordan 2002 convergence rate of parameter estimates slightly oversimplified see paper for bounds GNB order log n where n of attributes in X LR order n GNB converges more quickly to its perhaps less helpful asymptotic estimates Some experiments from UCI data sets What you should know Logistic regression Functional form follows from Na ve Bayes assumptions But training procedure picks parameters without the conditional independence assumption MLE training pick W to maximize P Y X W MAP training pick W to maximize P W X Y regularization Gradient ascent descent General approach when closed form solutions unavailable Generative vs Discriminative classifiers Artificial Neural Networks Artificial Neural Networks to learn f X Y f might be non linear function X vector of continuous and or discrete vars Y vector of continuous and or discrete vars Represent f by network of threshold units Each unit is a logistic function MLE train weights of all units to minimize sum of squared errors at network outputs ALVINN Pomerleau 1993 MLE Training for Neural Networks Consider regression problem f X Y for scalar Y y f x noise N 0 deterministic Learned neural network MAP Training for Neural Networks Consider regression problem f X Y for scalar Y y f x noise N 0 deterministic Gaussian P W N 0 ln P W c i wi2 MLE Training for Neural Networks Consider regression problem f X Y for Y y1 yN yi fi x i noise N 0 drawn independently for each output yi deterministic td target output od observed unit output Semantic Memory Model Based on ANN s McClelland Rogers Nature 2003 No hierarchy given Train with assertions e g Can Canary Fly Humans act as though they have a hierarchical memory organization 1 Victims of Semantic Dementia progressively lose knowledge of objects But they lose specific details first general properties later suggesting hierarchical memory Thing NonLiving 2 Children appear to learn general categories and properties first following the same hierarchy top down Living Animal Plant Fish Bird Canary Question What learning mechanism could produce this emergent hierarchy some debate remains on this Memory deterioration follows semantic hierarchy McClelland Rogers Nature 2003 ANN Also Models Progressive Deterioration McClelland Rogers Nature 2003 average effect of noise in inputs to hidden layers Original MLE error fn 2 Bias Variance Decomposition of Error Bias Variance decomposition of error Reading Bishop chapter 9 1 9 2 Consider simple regression problem f X Y y f x noise N 0 deterministic What are sources of prediction error learned Sources of error What if we have perfect learner infinite data Our learned h x satisfies h x f x Still have remaining unavoidable error 2 Sources of error What if we have only n training examples What is our expected error Taken over random training sets of size n drawn from distribution D p x y Sources of error
View Full Document
Unlocking...