ECE/CS/ME 539: Artificial Neural NetworksProfessor Yu Hen HuFinal Project ReportSteve Ludwig12-19-03A Comparison of a Learning Decision Tree and a 2-Layer Back-Propagation Neural Network on classifying a car purchase using a 2-Layer Back-Propagation Neural Network constructed in JavaAbstractAutomobiles have quickly become the one of the greatest inventions of our time, with most every household in the United States owning a vehicle – and many owning two, three, or more. With avast assortment of different automobiles now in circulation, many of them being used cars several years old, the question becomes how does one classify a good car purchase from a bad one? Based on several criteria with several settings each, a large amount of data detailing the attributes necessary for good and bad car purchases have been fed into two different programs to be able to predict in the future whether or not a certain car’s qualities will add up to a passing or failing grade. This project focuses on the comparison of these two programs, a Learning Decision Tree and a 2-Layer Back-Propagation Neural Network, and also on the design and implementation of the Neural Network in the Java programming language.IntroductionMy project is a combination of applications and software development. The main purpose of the project is to compare and contrast a 2-Layer Back-Propagation Neural Network with a Learning Decision Tree, and along the way I wrote my own neural network in Java for the comparison. The learning decision tree is a Java program I developed for CS 540 (Introduction to Artificial Intelligence)and its function and purpose are remarkably similar to that of a positive/negative pattern classifying neural network. A learning decision tree is trained using supervised learning data and then tested usinga subset of the train set. It uses a max gain/min remainder algorithm on the attributes of the train set, creating a tree of the attributes with the leaf nodes being a positive or negative classification.The 2-layer back-propagation neural network is written in Java as well. It allows for user defined parameters to be set such as the learning rate, number of epochs to run, number of nodes in the hidden layer, whether or not to see output after each epoch (to see data progression or save time), and whether or not to permute the train data set between each epoch. The program then outputs the train set sum of squares error and the test set classification accuracy after each epoch (if that option was chosen) and after it’s finished it will show the total sum of squares error, final test set classification andalso the classification accuracy on the train set.MotivationAs I concurrently took CS 540 with ECE 539, I was given exposure to two different pattern classifying programs, a learning decision tree built for CS 540 and a 2-layer back-propagation neural network built for ECE 539. Naturally given the similar natures, purposes and functions behind each ofthe programs, a curiosity arose as to which would perform better given identical data. The decision to use attributes of a car and decide if it would be fit to purchase is an arbitrary one, used mainly for ease of securing and using the data. I decided to write the neural network in Java to give myself a little more challenge in this project and also have some consistency on the testing of the programs.The ProgramsA learning decision tree reads in a supervised learning train set, similar to a neural network, to create its decision tree. It can use both text and numeric-based attributes and values. It requires a special attribute file, which lists each input attribute and all the possible values each attribute can take. For the decision to buy a car or not, the attribute file is as follows:buying price: low, med, high, vhighmaintenance price: low, med, high, vhighnumber of doors: 2, 3, 4, 5morenumber of passengers: 2, 4, morethe size of the trunk: small, med, bigestimated safety of the car: low, med, highcar acceptability: unacc, accEach input attribute is listed in the left column followed by a colon and a comma-separated list of all the possible ranges of values that the attribute can take. The last attribute is the output, ‘car acceptability’ with its answers either being ‘unacceptable’ or ‘acceptable.’ The program reads in the train data along with the attributes file and performs a max gain/least remainder algorithm to decide which attribute most immediately classifies the car as acceptable or unacceptable. It then finds the next largest gain, and so on until a tree is formed with each node being an attribute, and each leaf being either a positive (‘acceptable’) or negative (‘unacceptable’) classification. Max gain algorithm: I(%P, %N) = -(%P log2%P) - (%N log2%N)= -(p/(p+n)) log2 (p/(p+n)) - (n/(p+n)) log2 (n/(p+n))where p is the number of positive examples of a given value of an attributeand n is the number of negative examples of the same value of an attributeWhichever attribute has the smallest remainder for the sum over all its attributes has the max gain and is selected as the root node for the tree. For our data, ‘estimated safety of the car’ has the highest gain. This is mainly because whenever ‘estimated safety of the car’ is at value ‘low’ the car is automatically considered unacceptable.Next, in order to classify the test data the program simply looks at the attribute values and traverses the tree starting with the root node. As the data sample traverses the tree by its attribute values it will eventually reach a leaf node which will then classify it as acceptable or unacceptable.The 2-layer back-propagation neural network works as performed in class. It uses sigmoid activation at both the hidden node and output node layers and a -1 bias at each node. Only the training set is used to update the weights. The weight update formulas are as follows:Weights from input-hidden: Δwj,i = α * aj * (Ti - Oi) * Oi * (1 - Oi)Weights from hidden-output: Δwk,j = α * Ik * aj * (1 - aj) * Σwj,i*(Ti-Oi)*Oi*(1-Oi)where α is the learning rate, Ti is the teacher answer, Oi is the output, Ik is the input, aj is the hidden node activation function and Oi * (1 - Oi) is the derivate of the sigmoid activation.The default parameters used for the neural network are α = 0.l, epochs = 500, hidden nodes = 3.DataThe data used
View Full Document
Unlocking...