U n i v e r s i t y o f W i s c o n s i n - M a d i s o n Viability of Machine Learning Algorithms as Network Intrusion Detection Systems Greig Hazell Abstract: Misuse-Based Intrusion Detection Systems (IDS) based on Machine Learning Algorithms (MLA) are intended to strengthen the security of an Information System by preventing unknown attacks. Most misuse-based systems however suffer from a high error rate of false positives. Consequently, most of the research which evaluated an IDS focused solely on the classification accuracy of the system. For corporations whose day-to-day activities rely on fast network connections however, the throughput of the network IDS (N-IDS) as well as the resource utilization also play a critical role in choosing one security system over another. Therefore I focus on assessing the viability of MLAs as N-IDS by evaluating across the classification accuracy, throughput as well resource utilization. Fall 2010 08 Fall2 Introduction The increase in sophisticated and/or coordinated computer attacks against the communication & information systems of private co-corporations as well as government entities have increased the need for “intelligent” Intrusion Detection Systems (IDS). Traditionally, an intrusion detection system detects potential attacks by matching a sequence of operations against a database of well-known attack signatures. Such signature-based systems offer a high accuracy in detecting well-known attacks, but are susceptible to zero-day attacks as well as attacks with minor variations to a known attack signatures (Garcia-Teodoro, et al. 2008). To reduce the risk of unfamiliar attacks, research in IDS has shifted focus from signature-based systems to misuse-based systems, which attempt to classify system behavior as either normal or “abnormal”. Misuse based systems are first trained on normal system behavior, and then flag any behavior which deviates from this normal profile as a potential intrusion (Garcia-Teodoro, et al. 2008). While most of the research has focused on the classification accuracy and detection rate of one misuse-based system versus another, most of the evaluation have omitted the resource utilization as well throughput impact of misuse-based systems. This is especially important for network intrusion detection systems (N-IDS), which prevent malicious attacks by monitoring network traffic (Horng, et al. 2010). The remainder of the paper describes the experiments, which were conducted, on several Machine Learning Algorithms, the results of the experiments and finally my conclusions. Experiments and Setup All simulations were done on Ubuntu™ 10.10 Desktop Operating System on an Intel™ Pentium 4 2.8 GHz uni-processor system with 512 MB Main Memory. The classification algorithms used were from the LNKnet1 software suite, a public domain classification package. The data set used was the 10% KDD ’99 Data Cup Set. The training data set consisted of approximately 500K records with 22 attack categories plus 1 normal behavior category. The testing data set consisted of about 310K records and included an additional 18 attack types. As some of the features, including the attack categories, were in a format not acceptable by the classification algorithms, some preprocessing of the data was required as described below. Each classification algorithm was tested a total of 10 times on the test data. During the first 5 runs, the time required to complete the test set classification was recorded. The latter simulations were used to monitor system resources via the Linux utilities TOP and vmstat. 1 http://www.ll.mit.edu/mission/communications/ist/lnknet/index.html3 Pre-processing & Normalization of Data Most of the pre-processing of the data involved formatting the data into an input format acceptable by the classification algorithms. As most of the values were left as is, this section focuses on the pre-processing carried out on the attack categories and the test files generated. Initially, the normal behavior value was mapped to 0 and each attack type was mapped to a unique integer value starting from 1. Since this approach disallows unknown attack types from being classified during the testing phase, the processed testing data included only known attack categories. However, as preliminary testing results were not very promising, this approach was quickly abandoned. In the second method, attack categories were then processed in the manner described in (Elkan 2000) placing each attack type into one of four broader attack categories (0: Normal, 1: Probe, 2: Dos, 3:U2R, 4:R2L). Two test files were then generated from the original test data. The first test file included all records from the original data set while the second file included only the new attack types. The algorithms were then trained on the raw processed data as well as normalized data. Two normalization methods were used; Principal Component Analysis and 0 mean unit variance normalization. In each case, the results of the raw data produced more desirable results. Machine Learning Algorithms Used The performance of the following five machine-learning algorithms were determined based on the simulation experiment described above: Multi-Layer Perceptron (MLP), Radial Basis Function Networks (RBF), Naïve Bayesian Classifier (NBF), K-Nearest Neighbor (KNN) and Gaussian Classifier (GAUSS). Initially, the optimum settings for each classifier were determined by evaluating each algorithm a series of times altering various parameter values for each run. The configuration, which produced the highest overall classification rate, was then selected and subjected to the experiment in its entirety. Table 1 below summaries the configurations tested and finally selected for each classification algorithm when applicable. Table 1 Classifier Tested Selected MLP Layout/Layers: 41-15-5; 41-25-5; 41-50-5 41-25-5; Learning Rate = 0.1 RBF Cluster Centers: 4, 16, 32 16 KNN k: 1, 3, 5, 16 34 Results & Discussion Table 2 below summarizes the observations of each algorithms performance. The False Positive (FP) column shows the ratio of innocuous behavior misclassified as harmful. The False Negative (FN) column shows the ratio of harmful behavior erroneously classified as innocuous. The CPU column shows the low and high percentage values of the CPU utilized by the algorithms during the testing phase. Table 2 Classifier Test Set Classification Accuracy
View Full Document
Unlocking...