Linear Regression and Artificial Neural Networks Nan Li 2011 09 27 What is Linear Regression Assume that Y target is a linear function of X features E g Matrix form y x T Rent 1 2 k y x x x 0 1 2 k There are other forms but let s consider this case for now Living area What is a good LR model The model that best predicts the target Y given features X i e y i x i y i x iT y i to be close to zero Our goal is to seek that minimize the following cost function Least Mean Square LSM n 1 J x iT y i 2 2 i 1 How to nd the optimal Gradient descent j t 1 j J j t t 1 n T t 2 j x i yi j 2 i 1 t n j t y i x iT t x ij i 1 Converge to global optimum Batch Guraanteed convergence Online Fast How to nd the optimal Directly minimize J Take derivative and set to zero Derivation Elements tr T X T X tr T X T X trABC trCAB trBCA A trABAT C CAB C T ABT X T X X T X T X T X X T X T T try X tr y X T T y X T X y trABC trCAB trBCA A trAB BT More Derivation 1 J tr T X T X T X T y y T X y T y 2 1 tr T X T X 2 try T X try T y 2 1 X T X X T X 2X T y 2 X T X X T y 0 tr T X T X X T X X T X try T X X T y The normal equations EquiIvalence of LMS and MLE Assume yi T x i where follows a Gaussian N 0 Then Equivalence of LMS and MLE cont By independence assumption n n L p y i x i i 1 i 1 y i T x i 2 1 exp 2 2 2 n T 2 y x 1 n i i exp i 1 2 2 2 The log likelihood is l log L n log 1 1 1 n 2 i 1 y i T x i 2 2 2 Recall that Maximizing l is equivalent to minimizing J Ridge Regression vs Lasso X X Ridge Regression Lasso s with constant J level sets of J s with constant l2 norm 2 s with constant l1 norm 1 Lasso l1 penalty results in sparse solu2ons vector with more zero coordinates Good for high dimensional problems don t have to store all coordinates Bayesian Interpretation Ridge regression Gaussian Prior Prior belief that is Gaussian with zero mean biases soluAon to small Lasso regression Laplace Prior Prior belief that is Laplace with zero mean biases soluAon to small Something More LR with non linear basis functions LR does not mean we can only deal with linear relationships Linear means linear to Features can be non linear where the j x are xed basis functions and we de ne 0 x 1 Weighting points Locally weighted LR Higher weights for training examples closer to the query point Robust regression Higher weights for the training examples that t well Arti cial Neural Networks A set of connected perceptrons that make prediction based on unit inputs Let s start with a single perceptron Perceptron Learning Find w such that it minimizes sum of squared training errors How Gradient descent What s the decision boundary Non linear Why Multi layer Neural Networks Inputs Age 6 34 Gender Stage Independent variables 2 4 1 3 2 0 6 5 2 7 Output 4 8 2 Weights Hidden Layer Weights Probability of beingAlive Dependent variable Prediction Highly exible Non linear decision boundary How to learn Backpropagation An iterative method For each example Perform forward propagation Start from output layer Compute gradient of node with parents Update weight based on the gradient Convergence May stuck in local optimum
View Full Document
Unlocking...