PowerPoint PresentationIntroductionThe AlgorithmImplementation and ResultsResults (II)Nonlinear Conjugate Gradient Method for Supervised Training of MLPAlexandra RateringECE/CS/ME 539December 14, 2001IntroductionIntroductionBack-Propagation AlgorithmCan oscillate and be caught in local minimaSlow convergence rate (zigzag path to the minimum)Many parameters have to be adjusted be the user –learning rate, momentum constant …Nonlinear Conjugate Gradient MethodSecond order optimization approachFaster convergenceFewer parameters to adjustThe AlgorithmThe AlgorithmDirection vector = conjugate gradient vector–Linear combination of past direction vectors and the current negative gradient vector –Reduces oscillatory behavior in the minimum search –Reinforces weight adjustment in accordance with previous successful path directionsLearning rate–Optimal rate determined for every iteration via line search–Robustness of line search is critical for performance of CG-Algorithm )()(min)( nnEnavpwImplementation and ResultsImplementation and ResultsIn Matlab code with interface similar to bpResults for approximation problem of homework #40 50 100 150051015202530training error (epoch size = 64)epocherror0 20 40 60 80 100 120 140 1600102030405060708090training error EavepocherrorBP CGTraining error 0.0021 6.0807 e-4Testing error 4.9477 e-42.4293 e-4Results (II)Results (II)Results for pattern classification problem –Two equally sized 2D Gaussian distributions (30 samples)–Final training result for both CG and BP: Crate = 88.3% after 500 iterations0 100 200 300 400 500 600678910111213141516training error (epoch size = 60)epocherror0 100 200 300 400 500 600678910111213141516training error
View Full Document
Unlocking...