1Various Neural NetworksNetworksNeural Networks A mathematical model to solve engineering problems Group of connected neurons to realize compositions of non linear functions Tasks Classification Discrimination Estimation  2 types of networks Feed forward Neural Networks Recurrent Neural Networks Feed Forward Neural Networks The information is propagated from the inputs to the outputs Computations of2nd hiddenOutput layerfunctions from n input variables by compositions of algebraic functions Time has no role (NO cycle between outputs and inputs)x1x2 xn…..1st hidden layerlayerRecurrent Neural Networks Can have arbitrary topologies Can model systems with internal states (dynamic ones) Delays are associated to a specific weight100specific weight Training is more difficult Performance may be problematic Stable Outputs may be more difficult to evaluate Unexpected behavior (oscillation, chaos, …)x1 x201010Properties of Neural Networks Supervised networks are universal approximators networks) Theorem : Any limited function can be approximated by a neural network with a finite number of hidden neurons to an arbitrary precisionSupervised learning The desired response of the neural network in function of particular inputs is well known. A “Professor” may provide examples and teach the neural network how to fulfill a certain task2Unsupervised learning Idea : group typical input data in function of resemblance criteria un-known a priori Data clusteringNdffNo need of a professor The network finds itself the correlations between the data Examples of such networks : Kohonen feature mapsClassification (Discrimination) Class objects in defined categories Rough decision OREstimation of the probability for a certainEstimation of the probability for a certain object to belong to a specific classExample : Data mining  Applications : Economy, speech and patterns recognition, sociology, etc. ExampleExamples of handwritten postal codes drawn from a database available from the US Postal serviceWhat needed to create NN ? Determination of relevant inputs Collection of data for the learning and testing phases of the neural networkFi di th ti b f hidd dFinding the optimum number of hidden nodes Learning the parameters Evaluate the performances of the network If performances are not satisfactory then review all the precedent pointsPopular neural architectures Perceptron Multi-Layer Perceptron (MLP)Radial Basis FunctionNetwork (RBFN)Radial Basis Function Network (RBFN) Time Delay Neural Network (TDNN)  Other architecturesPerceptron Rosenblatt (1962) Linear separation Inputs :Vector of real valuesO++++++++++++++++++++++++++++1=yOutputs :1 or -1022110=++ xcxcc++++++++0=y0c1c2c∑1x2x122110xcxccv++=)(0 vstepy=3 The perceptron algorithm converges if examples are linearly separableMulti-Layer Perceptron One or more hidden layersOutput layer1st hidden layer2nd hiddenlayerInput dataStructureTypes ofDecision RegionsExclusive-ORProblemClasses withMeshed regionsMost GeneralRegion ShapesSingle-LayerHalf PlaneBounded ByHyperplaneAABBBADifferent non linearly separable problemsTwo-LayerThree-LayerHyperplaneConvex OpenOrClosed RegionsAbitrary(ComplexityLimited by No.of Nodes)ABAABBAABBBABA A radial basis function (RBF) is a real-valued function whose value depends only on the distance from some other point c, called a center, φ(x) = f(||x-c||)f f()Radial Basis Functions  Any function φ that satisfies the property φ(x) = f(||x-c||) is a radial function.  The distance is usually the Euclidean distance()∑=−=−Niii cxcx122|||| The popular output of radial basis functions is the Gaussian function:())exp(2⎟⎟⎞⎜⎜⎛−−=−ΦjjcxacxRadial Basis Functions ())exp(⎟⎟⎠⎜⎜⎝Φjjacxσa=1, c1=0.75, c2=3.25Radial Basis Functions Network (RBFN) Features One hidden layer The activation of a hidden unit is determined by a radial basis functionRadial unitsOutputsInputs4 Generally, the hidden unit function is the Gaussian function The output Layer is linear:()∑Φ=KcxWxs)(()∑=−Φ=jjjcxWxs1)(())exp(2⎟⎟⎟⎠⎞⎜⎜⎜⎝⎛−−=−ΦjjjcxwcxjσRBFN Learning The training is performed by deciding on How many hidden nodes there should be The centers and the sharpness of the Gaussians2t2 steps In the 1st stage, the input data set is used to determine the parameters of the RBF In the 2nd stage, RBFs are kept fixed while the second layer weights are learned ( Simple BP algorithm like for MLPs)Time Delay Neural Network (TDNN) Introduced by Waibel in 1989 Properties Local, shift invariant feature extractionN ti f ti fi ld bi i l l i f tiNotion of receptive fields combining local information into more abstract patterns at a higher level Weight sharing concept (All neurons in a feature share the same weights) All neurons detect the same feature but in different position Principal Applications Speech recognition Image analysis TDNNs (cont’d) Objects recognition in an image Each hidden unit receive inputs only from a small HiddenLayer 2region of the input space : receptive field Shared weights for all receptive fields => translation invariance in the response of the networkInputsHiddenLayer 1 Advantages Reduced number of weights Require fewer examples in the training setqpg Faster learning Invariance under time or space translation Faster execution of the net (in comparison of full connected MLP)Summary Neural networks are utilized as statistical tools Adjust non linear functions to fulfill a task Need of multiple and representative examples but fewer than in other methods Neural networks enable to model complex static phenomena (Feed-Forward)as well as dynamic ones (RecurentNN)Forward) as well as dynamic ones (RecurentNN) NN are good classifiers BUT Good representations of data have to be formulated Training vectors must be statistically representative of the entire input space Unsupervised techniques can help The use of NN needs a good comprehension of the