Neural Network Assignment Due April 10 2003 1 You are given a simple Perceptron with 3 inputs A B and C that uses a step function with threshold of 0 5 Find weights between 0 and 1 such that the output is a A and B or C b A and B or C 2 You are given the following Perceptron to train on the xor function Fill out the table provided in order to determine what the new weights will be after two training iterations Use the weight modification rules provided in class for a sigmoidal excitation function with the following change For the hidden and output neurons there is a bias that is applied to the sigmoid functions This can be interpreted as implied neuron that always fires and whose weight is set equal to that bias This allows one to learn biases for the neurons as well as learning the weights Use 0 5 a b h1 h2 o Initial Timestep 1 New 2 weights Case a 0 b 0 a 0 b 1 a 1 b 0 a 1 b 1 weights a 0 b 0 a 0 b 1 a 1 b 0 a 1 b 1 erro errh1 0 0279 errh2 wbias h1 0 68 wbias h1 0 00312 wa h1 0 38 wa h1 0 wb h1 0 83 wb h1 0 wbias h2 0 5 wbias h2 wa h2 0 71 wa h2 0 wb h2 0 43 wb h2 0 wbias o 0 30 wbias o wh1 o 0 19 wh1 o wh2 o 0 19 wh2 o wbias h1 wa h1 wb h1 wbias h2 wa h2 wb h2 wbias o wh1 o wh2 o 0 37989 Neural Network Assignment 3 Attached is a derivation for the backpropagation rules for Perceptrons that use a sigmoidal excitation function Derive an appropriate error function and weight update rule for Perceptrons that use a tanh excitation function Make the rules as simple as possible by choosing appropriate substitutions such as r r tanh x oi h for tanh xi wi i h Hint sec h 2 x 1 tanh 2 x x Page 2 of 4 Neural Network Assignment Derivation of Backpropagation for Sigmoid For each time step we present all training data to our multi layered Perceptron The change in the weight from neuron i to neuron j where i s output is being fed into j is given by P x c wi j x c T wi j Where is a parameter that controls how quickly the weights change and P x c is the performance of the network for the training data with input x and correct output c This performance is defined as r r 2 P x c F w x c If we define the output error i e the error at the last level of our Perceptron to be r r out F w x c then clearly we can rewrite the performance as 2 P x c out We can now calculate the derivative of our performance with respect to a given weight wi j as r r P x c F w x 2 2 wi j wi j wi j We will now adopt the convention that neuron i will be identified as an input neuron by the addition of the symbol i as a hidden neuron by the addition of the symbol h we could use h1 and h2 if we had two hidden layers and as an output neuron by the addition of the symbol o Therefore a weight connecting input neuron i to hidden neuron j would be represented by wi i j h Additionally all inputs to a hidden neuron i r r will be represented as xi and the weights associated with those inputs by wi The output r r of this neuron is therefore f xi wi i h or oi h We will also use the convention that for r r r an output neuron j the inputs can be represented as f x w so that the output from that r r r r neuron is f f x win whid or o j o This notation allows us to see that there are two r r special cases to consider when taking the derivative of F w x with respect to wi j r The first case is if wi j whid Using the chain rule we find r r r r r F w x r r r f f x win whid f xi wi i h wi h j o where f x f x For the sigmoid function f x f x 1 f x x r The second case is if wi j win After a double application of the chain rule we find r r r r r F w x r r r f f x win whid whid f x w j h xi wi i j h This leads to performance derivatives that can be written as Page 3 of 4 Neural Network Assignment P x c wi h j o P x c wi h j o r r r r r r 2 f xi wi i h f f x win whid out or r r r r r r r r r r 2 f xi wi i h f f x win whid 1 f f x win whid out r r for the hidden neurons If we use the alternative identifications oi h f xi wi i h and r r r r o j o f f x win whid this becomes P x c 2oi h o j o 1 o j o out wi h j o Likewise for the input neurons we find P x c r r r r r r 2 f f x win whid whid f x w j h xi out or wi i j h P x c wi i j h 2 xi o j h 1 o j o w j h k o o j h 1 o j h out If we define the error for the hidden neurons to then be j h w j h k o o j h 1 o j h out and the inputs xi to be considered as the outputs from the input neurons oi i then we can finally combine both equations into P x c 2oi o j 1 o j j wi j Page 4 of 4