5-15.1 IntroductionThe fascination of artificial neural networks started in the middle of the previous century. First artificial neurons were proposed by McCulloch and Pitts [MP43] and they showed the power of the threshold logic. Later Hebb [H49] introduced his learning rules. A decade later, Rosenblatt [R58] introduced the perceptron concept. In the early 1960s, Widrow and Holf [WH60] developed intelligent systems such as ADALINE and MADALINE. Nilsson [N65] in his book, Learning Machines, summarized many developments of that time. The publication of the Mynsky and Paper [MP69] book, with some discouraging results, stopped for sometime the fascination with artificial neural networks, and achievements in the mathematical founda-tion of the backpropagation algorithm by Werbos [W74] went unnoticed. The current rapid growth in the area of neural networks started with the work of Hopfield’s [H82] recurrent network, Kohonen’s [K90] unsupervised training algorithms, and a description of the backpropagation algorithm by Rumelhart et al. [RHW86]. Neural networks are now used to solve many engineering, medical, and business problems [WK00,WB01,B07,CCBC07,KTP07,KT07,MFP07,FP08,JM08,W09]. Descriptions of neural network tech-nology can be found in many textbooks [W89,Z92,H99,W96].5.2 The NeuronA biological neuron is a complicated structure, which receives trains of pulses on hundreds of excitatory and inhibitory inputs. Those incoming pulses are summed with different weights (averaged) during the time period [WPJ96]. If the summed value is higher than a threshold, then the neuron itself is generat-ing a pulse, which is sent to neighboring neurons. Because incoming pulses are summed with time, the neuron generates a pulse train with a higher frequency for higher positive excitation. In other words, if the value of the summed weighted inputs is higher, the neuron generates pulses more frequently. At the same time, each neuron is characterized by the nonexcitability for a certain time after the firing pulse. This so-called refractory period can be more accurately described as a phenomenon, where after excita-tion, the threshold value increases to a very high value and then decreases gradually with a certain time constant. The refractory period sets soft upper limits on the frequency of the output pulse train. In the biological neuron, information is sent in the form of frequency-modulated pulse trains.AQ15Understanding of Neural Networks5.1 Introduction ......................................................................................5-15.2 The Neuron ........................................................................................5-15.3 Should We Use Neurons with Bipolar or Unipolar Activation Functions? ......................................................................5-55.4 Feedforward Neural Networks .......................................................5-5References ....................................................................................................5-10Bogdan M. WilamowskiAuburn UniversityK10149_C005.indd 1 8/31/2010 4:32:01 AM5-2 Intelligent SystemsThe description of neuron action leads to a very complex neuron model, which is not practical. McCulloch and Pitts [MP43] show that even with a very simple neuron model, it is possible to build logic and memory circuits. Examples of McCulloch-Pitts’ neurons realizing OR, AND, NOT, and MEMORY operations are shown in Figure 5.1.Furthermore, these simple neurons with thresholds are usually more powerful than typical logic gates used in computers (Figure 5.1). Note that the structure of OR and AND gates can be identical. With the same structure, other logic functions can be realized, as shown in Figure 5.2.The McCulloch-Pitts neuron model (Figure 5.3a) assumes that incoming and outgoing signals may have only binary values 0 and 1. If incoming signals summed through positive or negative weights have a value equal or larger than threshold, then the neuron output is set to 1. Otherwise, it is set to 0. outnet Tnet T=≥<10ifif (5.1)whereT is the thresholdnet value is the weighted sum of all incoming signals (Figure 5.3)MemoryA + B + CABCABCABC+1+1+1+1+1+1+1+1–2Write 0Write 1–1T = –0.5T = 0.5T = 0.5T = 2.5AAFigure 5.1 Examples of logical operations using McCulloch–Pitts neurons.ABCA + B + CABCABC+1+1+1+1+1+1T = 0.5T = 2.5AB + BC + CAABC+1+1+1T = 1.5Figure 5.2 The same neuron structure and the same weights, but a threshold change results in different logical functions.x1x2x3x4xn(a)T = tT = 0wi xinet =ni =1∑x1x2x3x4xn+1(b)wixi+ wn+1net =ni =1∑wn+1= –tFigure 5.3 Threshold implementation with an additional weight and constant input with +1 value: (a) neuron with threshold T and (b) modified neuron with threshold T = 0 and additional weight wn+1= −t.K10149_C005.indd 2 8/31/2010 4:32:04 AMUnderstanding of Neural Networks 5-3The perceptron model has a similar structure (Figure 5.3b). Its input signals, the weights, and the thresholds could have any positive or negative values. Usually, instead of using variable threshold, one additional constant input with a negative or positive weight can be added to each neuron, as Figure 5.3 shows. Single-layer perceptrons are successfully used to solve many pattern classification problems. Most known perceptron architectures are ADALINE and MADALINE [WH60] shown in Figure 5.4.Perceptrons using hard threshold activation functions for unipolar neurons are given by o f netnetnetnetuni= =+=≥<( )sgn( ) 121 00 0ifif (5.2)and for bipolar neurons o f net netnetnetbip= = =≥− <( ) sgn( )1 01 0ifif (5.3)For these types of neurons, most of the known training algorithms are able to adjust weights only in single-layer networks. Multilayer neural networks (as shown in Figure 5.8) usually use soft activation functions, either unipolar o f net netuni= =+ −( )( )exp11 λ (5.4)or bipolar o f net net net bip= =( )=+ −( )−( ) tanh .exp0 5211λλ (5.5)These soft activation functions allow for the gradient-based training of multilayer networks. Soft activa-tion functions make neural network transparent for training [WT93]. In other words, changes in weight values always produce changes on the network outputs. This would not be possible when hard activation +1ADALINEwi xi+ wn+1net =ni = 1∑+1+1MADALINEo = netDuringtrainingHidden layerFigure 5.4 ADALINE and MADALINE perceptron