LECTURE NOTES Professor Anita Wasilewska Neural Networks Classification IntroductionNeural Networks ClassifierData NormalizationSlide Number 5Example of Max-Min NormalizationDecimal Scaling NormalizationNeural NetworkNeural Network Learning Neural Network LearningA Multilayer Feed-Forward (MLFF) Neural NetworkMLFF Neural NetworkMLFF Neural NetworkMLFF Neural NetworkMLFF Network InputMLFF Network TopologyMLFF Network TopologyClassification by Backpropagation Classification by BackpropagationSteps in Backpropagation AlgorithmSteps in Backpropagation AlgorithmA Neuron; a Hidden, or Output Unit jStep Three: propagate the inputs forwardStep Three: propagate the inputs forwardBack propagation Formulas Step 4: Back propagate the error Step 4: Back propagate the error Step 5: Update weights and biases Weights update Step 5: Update weights and biase Learning Rate Step 5: Update weights and biases Bias update Weights and Biases UpdatesTerminating ConditionsBack Propagation FormulasExample of Back Propagation Example of Back PropagationNet Input and Output CalculationCalculation of Error at Each NodeCalculation of weights and Bias UpdatingSome Facts to be RememberedAdvanced Features of Neural Network (to be covered by students presentations)Training with SubsetsTraining with subsetsModular Neural Network Evolving Network Architectures Constructive vs Destructive AlgorithmFaster ConvergenceLECTURE NOTES Professor Anita WasilewskaNEURAL NETWORKSNeural Networks Classification Introduction– INPUT: classification data, i.e. it contains an classification (class) attribute.– WE also say that the class label is known for all data.– DATA is divided, as in any classification problem, into TRAINING and TEST data sets.Neural Networks Classifier– ALL DATA must be normalized, i.e.all values of attributes in the dataset has to be changed to contain values inthe interval [0,1], or [-1,1].TWO BASIC normalization techniques:– Max- Min normalization and – Decimal Scaling normalization.Data Normalization• Max-Min NormalizationPerforms a linear transformation on the original data.• Given an attribute A, we denote by minA, maxA the minimum and maximumvalues of the values of the attribute A.• Max-Min Normalization maps a value vof A to v’ in the range• [new_minA, new_maxA]as follows.AnewAnewAnewAAAvv min_)min_max_(minmaxmin' +−−−=Data NormalizationMax- Min normalization formula is as follows: Example: we want to normalize data to range of the interval [-1,1].We put: new_max A= 1, new_minA = -1.In general, to normalize within interval [a,b], we put:new_max A= b, new_minA = a.Example of Max-Min NormalizationAnewAnewAnewAAAvv min_)min_max_(minmaxmin' +−−−=Max- Min normalization formulaExample: We want to normalize data to range of the interval [0,1].We put: new_max A= 1, new_minA =0.Say, max A was 100 and min A was 20 ( That means maximum and minimum values for the attribute A).Now, if v = 40 ( If for this particular pattern , attribute value is 40 ), v’ will be calculated as , v’ = (40-20) x (1-0) / (100-20) + 0 => v’ = 20 x 1/80=> v’ = 0.4Decimal Scaling NormalizationNormalization by decimal scaling normalizes by moving thedecimal point of values of attribute A.A value v of A is normalized to v’ by computingjvv10' =where j is the smallest integer such that max|v’|<1.Example :A – values range from -986 to 917. Max |v| = 986.v = -986 normalize to v’ = -986/1000 = -0.986Neural Network• Neural Network is a set of connected INPUT/OUTPUT UNITS, where each connection has a WEIGHT associated with it.• Neural Network learning is also called CONNECTIONIST learning due to the connections between units.• Neural Network is always fully connected.• It is a case of SUPERVISED, INDUCTIVE or CLASSIFICATION learning.Neural Network Learning • Neural Network learns by adjusting the weights so as to be able to correctly classify the training data and hence, after testing phase, to classify unknown data.• Neural Network needs long time for training.• Neural Network has a high tolerance to noisy and incomplete data.Neural Network Learning• Learning is being performed by a back propagation algorithm.• The inputs are fed simultaneously into the input layer.• The weighted outputs of these units are, in turn, are fed simultaneously into a “neuron like” units, known as a hidden layer.• The hidden layer’s weighted outputs can be input to another hidden layer, and so on.• The number of hidden layers is arbitrary, but in practice, usually one or two are used.• The weighted outputs of the last hidden layer are inputs to units making up the output layer.kOkjwOutput nodesInput nodesHidden nodesOutput vector;classesInput vector;Record: xiwij- weightsNetwork is fully connectedjOA Multilayer Feed-Forward (MLFF) Neural NetworkMLFF Neural Network• The units in the hidden layers and output layer are sometimes referred to as neurones, due to their symbolic biological basis, or as output units.• A multilayer neural network shown on the previous slide has two layers of output units.• Therefore, we say that it is a two-layer neural network.MLFF Neural Network• A network containing two hidden layers is called a three-layer neural network, and so on.• The network is feed-forward - it means that none of the weights cycles back to an input unit or to an output unit of a previous layer.kOkjwOutput nodesInput nodesHidden nodesOutput vector;3 classes hereInput vector;Record: xi2 attributes here wij- weightsNetwork is fully connectedjOMLFF Neural NetworkMLFF Network Input• INPUT: records without class attribute with normalized attributes values. We call it an input vector. • INPUT VECTOR:X = { x1, x2, …. xn}where n is the number of (non class) attributes.Observe that {,} do not denote a SET symbol here!NN network people like use that symbol for a vector;Normal vector symbol is [ x1, … xn]MLFF Network Topology• INPUT LAYER – there are as many nodes as non-class attributes i.e. as the length of the input vector.• HIDDEN LAYER – the number of nodes in the hidden layer and the number of hidden layers depends on implementation.jOj=1, 2 ..#hidden nodesMLFF Network Topology• OUTPUT LAYER – corresponds to the class attribute.• There are as many nodes as classes (values of the class attribute).kOk= 1, 2,.. #classes• Network is fully connected, i.e. each unit provides input to each unit in the next forward