Mining Logs Files for Data-Driven System ManagementWei PengSchool of Computer ScienceFlorida International UniversityMiami, FL [email protected] LiSchool of Computer ScienceFlorida International UniversityMiami, FL [email protected] MaMachine Learning for SystemsIBM T.J. Watson ResearchCenterHawthorne, NY [email protected] advancement in science and technology, computing systemsare becoming increasingly more complex with an increasing vari-ety of heterogeneous software and hardware components. They arethus becoming increasingly more difficult to monitor, manage andmaintain. Traditional approaches to system management have beenlargely based on domain experts through a knowledge acquisitionprocess that translates domain knowledge into operating rules andpolicies. This has been well known and experienced as a cumber-some, labor intensive, and error prone process. In addition, thisprocess is difficult to keep up with the rapidly changing environ-ments. There is thus a pressing need for automatic and efficientapproaches to monitor and manage complex computing systems.A popular approach to system management is based on analyzingsystem log files. However, some new aspects of the log files havebeen less emphasized in existing methods from data mining andmachine learning community. The various formats and relativelyshort text messages of log files, and temporal characteristics in datarepresentation pose new challenges. In this paper, we will describeour research efforts on mining system log files for automatic man-agement. In particular, we apply text mining techniques to catego-rize messages in log files into common situations, improve catego-rization accuracy by considering the temporal characteristics of logmessages, and utilize visualization tools to evaluate and validatethe interesting temporal patterns for system management.KeywordsSystem Log, Categorization, Temporal Information, Visualization,Naive Bayes, Hidden Markov Model1. INTRODUCTIONWhen problems occur, traditional approaches for trouble shootingrely on the knowledge and experience of domain experts to figureout ways to discover the rules or look for the problem solutionslaboriously. It has been estimated that, in medium and large com-panies, anywhere from 30% to70% of their information technologyresources are used in dealing with problems [22]. It is unrealisticand inefficient to depend on domain experts to manually deal withcomplex problems in ever-changing computing systems.Modern computing systems are instrumented to generate hugeamounts of system log data. The data in the log files describe thestatus of each component and record system operational changes,such as the starting and stopping of services, detection of networkapplications, software configuration modifications, and softwareexecution errors. Analyzing log files, as an attractive approach forautomatic system management and monitoring, has been enjoyinga growing amount of attention. However, several new aspects ofthe system log data have been less emphasized in existing analysismethods from data mining and machine learning community andpose several challenges calling for more research. The aspects in-clude disparate formats and relatively short text messages in datareporting, asynchronism in data collection, and temporal character-istics in data representation.First, the heterogeneous nature of the system makes the data morecomplex and complicated [10]. As we know, a typical comput-ing system contains different devices (e.g., routers, processors, andadapters) with different software components (e.g., operating sys-tems, middleware, and user applications), possibly from differentproviders (e.g., Cisco, IBM, and Microsoft). These various com-ponents have multiple ways to report events, conditions, errors andalerts. The heterogeneity and inconsistency of log formats makeit difficult to automate problem determination. For example, thereare many different ways for the components to report the start upprocess. Some might log “the component has started”, while othersmight say that “the component has changed the state from startingto running”. This makes it difficult to perform automated analysisof the historical event data across multiple components when prob-lems occur as one need to know all the messages that reflect thesame status, for all the components involved in the solution [26].To enable automated analysis of the historical event data acrossmultiple components, we need to categorize the text messages withdisparate formats into common situations. Second, text messagesin the log files are relatively short with a large vocabulary size [25].Hence, care must be taken when applying traditional document pro-cessing techniques. Third, each text message usually contains atimestamp. The temporal characteristics provide additional contextinformation of the messages and can be used to facilitate data anal-ysis.In this paper, we describe our research efforts to address the abovechallenges in mining system logs. In particular, we propose to minesystem log files for computing system management by acquiringthe needed knowledge automatically from a large amount of histor-ical log data, possibly from different types of information sourcessuch as system errors, resource performance metrics, and troubleticket text records. Specifically, we will apply text mining tech-niques to automatically categorize the text messages into a set ofcommon categories, incorporate temporal information to improvecategorization performance, and utilize visualization tools to eval-uate and validate the interesting temporal patterns for system man-agement. A preliminary version of this paper has been presentedas a short paper [20] at The 2nd IEEE International Conference onAutonomic Computing (ICAC-05)It should be note that our framework is complementary to the cur-rent knowledge-based approaches, which are based on elicitationof knowledge from domain experts. Automated log data analysiscan be performed without much domain knowledge and its resultsprovide guidance for network managers to perform their jobs moreeffectively. Moreover, the available domain knowledge can be usedto validate, improve, and refine data analysis.The rest of the paper is organized as follows: Section 2 applies textmining techniques to categorize text messages into a set of com-mon categories, Section 3 proposes two approaches of incorpo-rating temporal information to improve the