Neural Networks - 1 !Robert Stengel! Robotics and Intelligent Systems, MAE 345, Princeton University, 2013"• Natural and artificial neurons"• Natural and computational neural networks"– Linear network"– Perceptron"– Sigmoid network"• Applications of neural networks"• Supervised training"– Left pseudoinverse"– Steepest descent"– Back-propagation"– Exact algebraic fit"Copyright 2013 by Robert Stengel. All rights reserved. For educational use only.!http://www.princeton.edu/~stengel/MAE345.html!Learning Objectives!Applications of Computational Neural Networks"• Classification of data sets"• Nonlinear function approximation"• Efficient data storage and retrieval"• System identification"• Nonlinear and adaptive control systems"Neurons"• Afferent (unipolar) neurons send signals from organs and the periphery to the central nervous system"• Efferent (multipolar) neurons issue commands from the CNS to effector (e.g., muscle) cells"• Interneurons (multipolar) send signals between neurons in the central nervous system"• Signals are ionic, i.e., chemical (neurotransmitter atoms and molecules) and electrical (potential)"• Biological cells with significant electrochemical activity"• ~10-100 billion neurons in the brain"• Inputs from thousands of other neurons"• Output is scalar, but may have thousands of branches"Activation Input to Soma Causes Change in Output Potential"• Stimulus from"– Other neurons"– Muscle cells"– Pacemakers (c.g. cardiac sino-atrial node)"• When membrane potential of neuronal cell exceeds a threshold"– Cell is polarized"– Action potential pulse is transmitted from the cell"– Activity measured by amplitude and firing frequency of pulses"• Cell depolarizes and potential returns to rest"Neural Action Potential"• Maximum Firing Rate: 500/sec"• Refractory Period: Minimum time increment between action potential firing ~ 1-2 msec"Some Recorded Action Potential Pulse Trains "Impulse, Pulse-Train, and Step Response of a LTI 2nd-Order Neural Model "Neglecting absolute refractory period"In the limit, neglecting absolute refractory period"Multipolar Neuron"Mathematical Model of Neuron Components"w11"w12"w13"w21"w22"w23"Synapse effects represented by weights (gains or multipliers)"Neuron firing frequency is modeled by linear gain or nonlinear element"The Neuron Function"• Vector input, u, to a single neuron"– Sensory input or output from upstream neurons"– Linear operation produces scalar, r"– Add bias, b, for zero adjustment"• Scalar output, u, of a single neuron (or node)"– Scalar linear or nonlinear operation, s(r)"r = wTu + bu = s r( )Layout of a Neural Network"Layered, parallel structure for computation"Input-Output Characteristics of a Neural Network Layer"• Single layer"– Number of inputs = n!• dim(u) = (n x 1)"– Number of nodes = m!• dim(r) = dim(b) = dim(s) = (m x 1)"r = Wu + bu = s r( )W =w1Tw2TwnT⎡⎣⎢⎢⎢⎢⎢⎤⎦⎥⎥⎥⎥⎥Two-Layer Network"• Two layers"– Number of nodes in each layer need not be the same"– Node functions may be different, e.g.,"• Sigmoid hidden layer"• Linear output layer"y = u2= s2r2( )= s2W2u1+ b2( )= s2W2s1r1( )+ b2⎡⎣⎤⎦= s2W2s1W1u0+ b1( )+ b2⎡⎣⎤⎦= s2W2s1W1x + b1( )+ b2⎡⎣⎤⎦Is a Neural Network Serial or Parallel?"3rd-degree power series"4 coefficients"Express as a neural network?"y = a0+ a1x + a2x2+ a3x3= a0'+ a1' r + a2' r2+ a3' r3= a0'+ a1' c1x + b1( )+ a2' c1x + b2( )2+ a3' c1x + b3( )3= w0+ w1s1u( )+ w2s2u( )+ w3s3u( )Is a Neural Network Serial or Parallel?"Power series is serial, but it can be expressed as a parallel neural network (with dissimilar nodes)"MATLAB Neural Network Toolbox "• Implementation of many neural network architectures"• Common calling sequences"• Pre- and post-processing"• Command-line and GUI"MATLAB Training and Evaluation of Backpropagation Neural Networks "! Backpropagation (Ch. 5)"! Preprocessing to normalize data (5-62)"! Architecture (5-8)"! Simulation (5-14)"! Training algorithms (5-15, 5-52) "Linear Neural Network"• Outputs provide linear scaling of inputs"• Equivalent to matrix transformation of a vector, y = Wx + b"• Therefore, linear network is easy to train (left pseudoinverse)"• MATLAB symbology"Idealizations of Nonlinear Neuron Input-Output Characteristic"Sigmoid with two inputs, one output"Logistic sigmoid function"Step function (Perceptron )"u = s(r) =1, r > 00, r ≤ 0⎧⎨⎪⎩⎪u = s(r) =11 + e− ru = s(r) =11 + e− w1r1+ w2r2+b( )Perceptron Neural Network"Each node is a step function"Weighted sum of features is fed to each node"Each node produces a linear classification of the input space"Perceptron Neural Network"Weights adjust slopes"Biases adjust zero crossing points"Single-Layer, Single-Node Perceptron Discriminants"Two inputs, single step function"Discriminant"Three inputs, single step function"Discriminant" x =x1x2⎡ ⎣ ⎢ ⎤ ⎦ ⎥  x =x1x2x3⎡ ⎣ ⎢ ⎢ ⎢ ⎤ ⎦ ⎥ ⎥ ⎥  w1x1+ w2x2+ b = 0or x2=−1w2w1x1+ b( ) w1x1+ w2x2+ w3x3+ b = 0or x3=−1w3w1x1+ w2x2+ b( )u = s(wTx + b) =1, (wTx + b) > 00, (wTx + b) ≤ 0⎧⎨⎪⎩⎪Single-Layer, Multi-Node Perceptron Discriminants" x =x1x2x3⎡ ⎣ ⎢ ⎢ ⎢ ⎤ ⎦ ⎥ ⎥ ⎥  x =x1x2⎡ ⎣ ⎢ ⎤ ⎦ ⎥ • Multiple inputs, nodes, and outputs"– More inputs lead to more dimensions in discriminants"– More outputs lead to more discriminants"u = s(Wx + b)Multi-Layer Perceptrons Can Classify With Boundaries or Clusters "Classification capability of multi-layer perceptrons"Classifications of classifications"Open or closed regions"Sigmoid Activation Functions"! Alternative sigmoid functions"! Logistic function: 0 to 1"! Hyperbolic tangent: –1 to 1"! Augmented ratio of squares: 0 to 1"! Smooth nonlinear functions"u = s(r) =11 + e− ru = s(r) = tanh r =1 − e−2r1 + e−2ru = s(r) =r21 + r2Sigmoid Neural Network"Single Sigmoid Layer is Sufficient …"! Sigmoid network with single hidden layer can approximate any continuous function"! Therefore, additional sigmoid layers are unnecessary"! Typical sigmoid network contains"! Single sigmoid hidden layer (nonlinear fit)"! Single linear output layer (scaling)"Typical Sigmoid