10701 RecitationTopicsQuestions?0-1 Error vs Squared ErrorSquared ErrorExample: Intrinsic Error and Modeling ErrorModeling Error = Bias2+VarianceModeling Error = Bias2+VariancePractical Solution to the Bias-Variance ProblemCross-ValidationCross-Validation ExampleNeural NetworksNeural Networks: Sigmoid FunctionNeural Networks: Sigmoid FunctionNeural Networks: Sigmoid FunctionNeural Networks: BackpropagationNeural Networks: Simple Example10701 RecitationDerek HoiemFeb 3, 2005Topics• Questions– Last homework– Next homework– Class material• Where does error come from?– Measuring error– Intrinsic error– Bias and Variance components•Overfitting– Cross-validation• Backpropagation ExampleQuestions?0-1 Error vs Squared ErrorLossConfidenceSquared Error∑=−=NnnnNtwxye121));((Squared ErrorInfinite Data∫∫−= dtdxxtptwxye ),());((2∫∫−+−= dxxpxtExtEdxxpxtEwxye )())|()|(()())|();((222Tricky StuffIntrinsic ErrorModeling ErrorExample: Intrinsic Error and Modeling ErrorModeling Error = Bias2+Variance∫∫−+−= dxxpxtExtEdxxpxtEwxye )())|()|(()())|();((222Modeling error, huh? Sounds suspicious….Spectacled Friend #1There’s nothing I can do….Spectacled Friend #2What is in your circle of concern?Stephen CoveyModeling Error = Bias2+Variance∫+− dxxpxtEwxy )())|();((2∫−= dxxpxtExtEe )())|()|((22Tricky Stuff∫∫−+−= dxxpwxyEwxyEdxxpxtEwxyEeEDDDmD)())];(();([)()]|());(([)(22Bias2VariancePractical Solution to the Bias-Variance Problem1. Sufficiently complex classifier for problem (low bias)• How can we tell if its too complex or not complex enough?2. Cross-validation or validation set (reduce variance)Cross-Validation• How many folds?–K-fold– Leave-One-OutCross-Validation ExampleNeural Networks• Defined by:– Num inputs– Num hidden layers– Hidden nodes per layer– Num outputs– Activation functionsxh11h12h21h22y1 1 1Neural Networks: Sigmoid Function))(exp(11);(0∑+−+=iiixwwwxσxh11h121w0h11w0h12w1h12w1h11xh11h12h21h22y1 1 1Neural Networks: Sigmoid Function))(exp(11);(0∑+−+=iiixwwwxσw0Æ shift kw Æ slopeNeural Networks: Sigmoid Function))(exp(11);(0∑+−+=iiixwwwxσNeural Networks: Backpropagationxh11h12h21h22y1 1 1Neural Networks: Simple ExampleTraining Input:x=1y=1x=0.5y = 01 11-1h
View Full Document