Scalable Network-based Buffer Overflow AttackDetectionFu-Hau HsuDepartment of ComputerScience and InformationEngineeringNational Central UniversityTaoyuan, Taiwan, [email protected] GuoSymantec ResearchLaboratoryCupertino, CA, U.S.A.fanglu [email protected] ChiuehComputer ScienceDepartmentStony Brook UniversityStony Brook, NY, [email protected] overflow attack is the main attack method that most if notall existing malicious worms use to propagate themselves frommachine to machine. Although a great deal of research has beeninvested in defense mechanisms against buffer overflow attack,most of them require modifications to the network applicationsand/or the platforms that host them. Being an extension workof CTCP, this paper presents a network-based low performanceoverhead buffer overflow attack detection system called Nebula1,which can detect both known and zero-day buffer overflow at-tacks based solely on the packets observed without requiring anymodifications to the end hosts. Moreover, instead of derivinga specific signature for each individual buffer overflow attackinstance, Nebula uses a generalized signature that can captureall known variants of buffer overflow attacks while reducing thenumber of false positives to a negligible level. In addition, Neb-ula is built on a centralized TCP/IP architecture that effectivelydefeats all existing NIDS evasion techniques. Finally, Nebula in-corporates a payload type identification mechanism that reducesfurther the false positive rate and scales the proposed buffer over-flow attack detection scheme to gigabit network links.Categories and Subject DescriptorsC.2.0 [Computer-Communication Networks]: General—Se-curity and protection (e.g., firewalls)General TermsSecurityKeywordsBuffer Overflow Attacks, Return-into-libc Attacks, CTCP, Gen-eralized Attack Signatures, Payload Bypassing, Network-basedIntrusion Detection½NEtwork-based BUffer overfLow Attack detectionPermission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.ANCS’06, December 3–5, 2006, San Jose, California, USA.Copyright 2006 ACM 1-59593-580-0/06/0012 ...$5.00.1. INTRODUCTIONBuffer overflow attack is arguably the most widely used andthus most dangerous attack method used today. It accounts formore than 50% of all the security vulnerabilities recorded byCERT [20]. Many solutions to the buffer overflow attack prob-lem have been proposed in the last decade, including compilertransformation approaches that detect and/or prevent tamperingof control-sensitive data structures [2,3,19], library rewriting ap-proaches that ensure each incoming packet never steps beyondthe corresponding packet-receiving buffer’s bound, and operat-ing system approaches that prevent malicious code injected bybuffer overflow attacks from being executed. In theory, these ef-forts have largely solved the buffer overflow attack problem. Inpractice, however, new buffer overflow vulnerabilities are stilldiscovered and reported on a routine basis. This discrepancybetween theory and practice arises because almost all existingsolutions to the buffer overflow attack problem require substan-tial modification to the computing infrastructure in which net-work applications are developed or executed, and thus have metsubstantial resistance in actual deployment. One way to over-come this deployment problem is to develop a network-basedbuffer overflow attack detection mechanism that can detect ar-bitrary buffer overflow attacks without requiring any changes tothe network applications or the hosts they run on. This paperdescribes the design, implementation and evaluation of such asystem, called Nebula.Existing network-based intrusion detection systems (NIDS)compare incoming packets against an attack signature database,and raise an alert when one or multiple matches are found. Typi-cally, a separate signature is created for each distinct buffer over-flow attack. Obviously, this approach cannot effectively detectzero-day attacks, whose signature is unavailable by definition,or variants of known attacks. Moreover, under this approach,false positives are inevitable and tend to be numerous, mainlybecause the signature matching logic in NIDSs rarely takes intoaccount the context in which buffer overflow attacks take place.In contrast, the design goal of Nebula is to detect arbitrary bufferoverflow attacks, zero-day or not, based solely on the payload ofincoming packets. While Nebula is still signature-based, the sig-nature it uses is designed to capture all known buffer overflowattacks. Although the Nebula prototype described in this paperdoes not achieve its design goal completely, we believe it repre-sents an important step toward reaching that goal.There are two variants of buffer overflow attack: code-injection(CI) attack, where attackers insert a piece of malicious code intothe victim application’s address space and then steer the appli-cation’s control to the injected code; return to libc (RTL) attack,where attackers directly steer the control of the victim applica-tion to a function pre-existing in its address space, e.g., a libraryfunction. In both cases, attackers hijack the control of the vic-163tim application, by modifying a control-sensitive data structuresuch as a return address and changing it to either a location onthe stack (CI attack) or a location in the text or code region (RTLattack). From the above analysis, a buffer overflow attack packetmust include a 4-byte hijack destination word that correspondsto a memory address on the stack or in the text region. Fur-thermore, to increase the success probability and robustness ofa buffer overflow attack, attackers almost always replicate thehijack destination word in the packet so as to accommodate dif-ferences in the address of the target control-sensitive data struc-ture due to different combinations of compiler, loader, operatingsystem, and command-line arguments. In summary, the mainsignature that Nebula uses to detect buffer overflow attacks is asequence of identical 4-byte words that correspond to an addressin the stack region or text region.For all the buffer