Characterizing the ‘Security Vulnerability Likelihood’ of Software FunctionsDan DaCosta, Christopher Dahn, Spiros Mancoridis, Vassilis PrevelakisDepartment of Computer ScienceDrexel University, Philadelphia, PA, USADaniel.William.DaCosta, Christopher.Stephen.Dahn,Spiros.Mancoridis, Vassilis.Prevelakis@drexel.eduAbstractSoftware maintainers and auditors would benefit from atool to help them focus their attention on functions that arelikely to be the source of security vulnerabilities. However,the existence of such a tool is predicated on the ability tocharacterize a function’s ‘security vulnerability likelihood.’Our hypothesis is that functions near a source of inputare most likely to contain a security vulnerability. Thesefunctions should be a small percentage of the total num-ber of functions in the system. To validate this hypothesis,we performed an experiment involving thirty one vulnera-bilities in open source software. This paper describes theexperiment, its outcome, and the tools used to conduct it.It also describes the FLF Finder, which is a tool that wasdeveloped using knowledge gathered from the outcome ofthe experiment. This tool automates the detection of highrisk functions. To demonstrate the effectiveness of the FLFFinder, three open source applications with known vulnera-bilities were tested. In addition to this test, a case study wasperformed on the privilege separation code in the OpenSSHserver daemon.1. IntroductionA software vulnerability is a fault in the specifica-tion, implementation, or configuration of a software systemwhose execution can violate an explicit or implicit securitypolicy [15]. Software maintainers typically focus on thefunctionality of software rather than on its security posture.Hence, vulnerabilities often escape their attention until thesoftware is exploited by specially written malicious code.A large percentage of software is developed using un-safe programming languages (e.g., C and C++) in the nameof cost effectiveness, programmer familiarity, and perfor-mance. Being unable to influence how others develop newsoftware, we must find ways to improve the maintenanceprocess to secure software against possible attacks.Code audits are one aspect of the maintenance processthat can focus on security vulnerabilities, and have beentried, with some success, on systems such as the OpenBSDoperating system [23]. Unfortunately, audits are expensiveand reoccurring. Each audit requires many man hours, andeach software revision requiresre-examinationtoverify thatnew faults have not been introduced.The quantity of code in many systems makes large-scaleauditing infeasible. In the case of OpenBSD, the auditingeffort only focuses on software that is enabled in the de-fault installation. This decision has resulted in overlookedvulnerabilities in other often used components of the distri-bution, such as telnetd.Beizer states that good source code will have one to threefaults for every one hundred lines of code [4]. However, itis not known which of those faults is a security vulnerabil-ity. Auditors would benefit from a tool that can reduce theamount of code that needs to be studied; enabling them tofocus their attention on areas of likely vulnerability.Our hypothesis is that a small percentage of functionsnear a source of input (e.g., file I/O) are the most likely tocontain a security vulnerability. Exactly what we mean by‘near’ is described in Section 3.3. We refer to these func-tions as FLFs (Front Line Functions), and the percentageof functions likely to contain a security vulnerability as theFLF density. We validate our hypothesis with an experi-ment that involves thirty one vulnerabilities in open sourcesoftware using two tools that were developed for this pur-pose. The results of this experiment are summarized in Ta-ble 1.Based on the validationof the hypothesis, the FLF Findertool was developed to identify areas of high vulnerabil-ity likelihood automatically. The effectiveness of the FLFFinder is demonstrated in two ways. First, it is applied tothree open source software systems, micq, elm, and dhcpd,with known (documented) vulnerabilities. Second, the FLFFinder is applied to the OpenSSH daemon software, whichdoes not have known vulnerabilities but has recently un-dergone a widely-publicized restructuring, called privilegeseparation, aimed at minimizing the amount of code thatruns with elevated privileges. By minimizing the amount ofprivileged code, it reduced the risk of a security vulnerabil-ity occurring within that code. Although the restructuringwas done manually, our case study shows that the resultsproduced by the FLF Finder are consistent with the designdecisions made by the security auditors.The remainder of this paper is structured as follows: Sec-tion 2 outlines related research, Section 3 presents the ex-periment, Section 4 describes how the experiment’s out-come was used to develop the FLF Finder, Section 5 detailsthe OpenSSH case study, and Section 6 outlines the limita-tions of this work and our future plans. The paper concludesin Section 7.2. Related ResearchThis work is related to two major areas of research. Thefirst is software security, specifically as it applies to securityvulnerabilities in code. The second is the use of source codeanalysis for software maintenance.2.1. Software Security2.1.1. Common Security VulnerabilitiesThere are many classifications of software security vulner-abilities [2, 14, 5]. This work considers only the categoriesof vulnerabilities that involve obtaining external input datathat causes a security exploit.Figure 1. Buffer OverflowBuffer overflows may occur when a fixed size mem-ory allocation is used to store a variable-size data entry.There are conflicts when the variable size data entry over-runs the bounds of the fixed-size memory, as show in Fig-ure 1. These overflows are typically exploited by entering astring which is larger than the buffer assigned to hold it. Ifthe return address (RA) is part of the overwritten run-timestack, the user may execute arbitrary code [22].Format string vulnerabilities target specific types of Cand C++ function calls such as printf and sprintf.This family of functions accepts an input string that speci-fies the format of the output along with an arbitrary numberof arguments that correspond to that string, called the vari-able argumentslist. The function pushes the arguments ontothe run-time stack and then reads the format string, poppingthe