Binary Obfuscation Using Signals∗Igor V. Popov, Saumya K. Debray, Gregory R. AndrewsDepartment of Computer Science, The University of Arizona, Tucson, AZ 85721, USAEmail: {ipopov, debray, greg}@cs.arizona.eduAbstractReverse engineering of software is the process of recov-ering higher-level structure and meaning from a lower-level program representation. It can be used for legit-imate purposes—e.g., to recover source code that hasbeen lost—but it is often used for nefarious purposes,e.g., to search for security vulnerabilities in binaries or tosteal intellectual property. This paper addresses the prob-lem of making it hard to reverse engineering binary pro-grams by making it difficult to disassemble machine codestatically. Binaries are obfuscated by changing manycontrol transfers into signals (traps) and inserting dummycontrol transfers and “junk” instructions after the signals.The resulting code is still a correct program, but eventhe best current disassemblers are unable to disassemble40%–60% of the instructions in the program. Further-more, the disassemblers have a mistaken understandingof over half of the control flow edges. However, the ob-fuscated program necessarily executes more slowly thanthe original. Experimental results quantify the degree ofobfuscation, stealth of the code, and effects on executiontime and code size.1 IntroductionSoftware is often distributed in binary form, withoutsource code. Many groups have developed technologythat enables one to reverse engineer binary programsand thereby reconstruct the actions and structure of theprogram. This is accomplished by disassembling ma-chine code into assembly code and then possibly de-compiling the assembly code into higher level repre-sentations [5, 6, 13]. While reverse-engineering tech-nology has many legitimate uses (in particular, an im-portant application of binary-level reverse engineeringis to analyse malware in order to understand its behav-ior [4, 16–18, 25, 27, 33]), it can also be used to dis-cover vulnerabilities, make unauthorized modifications,or steal intellectual property.∗This work was supported in part by NSF Grants EIA-0080123,CCR-0113633, and CNS-0410918.Since the first step in reverse engineering a binaryis disassembly, many approaches to binary obfuscationfocus on disrupting this step. This is typically doneby identifying assumptions made by disassemblers, thentransforming the program systematically so as to violatethese assumptions without altering program functional-ity. Two fundamental assumptions made by disassem-blers are that (1) the address where each instruction be-gins can be determined; and (2) control transfer instruc-tions can be identified and their targets determined. Thefirst assumption is used to identify the actual instructionsto disassemble; most modern disassemblers use the sec-ond assumption to determine which memory regions getdisassembled (see Section 2). In this context, this papermakes the following contributions:1. It shows how the second of these assumptions canbe violated, such that actual control transfers in theprogram cannot be identified by a static disassem-bler. This is done by replacing control transferinstructions—jumps, calls, and returns—by “ordi-nary” instructions whose execution raises traps atruntime; these traps are then fielded by signal han-dling code that carries out the appropriate controltransfer. The effect is to rep lace control transferinstructions either with apparently innocuous arith-metic or memory operations or with what appear tobe illegal instructions that suggest an erroneous dis-assembly.2. It shows how the code resulting from this first trans-formation can be further obfuscated to additionallyviolate the first assumption stated above. This isdone using a secondary transformation that inserts(unreachable) code, containing fake control trans-fers, after these trap-raising instructions, in order tomake it hard to find the beginning of the true nextinstructions.In earlier work, we showed how disassembly could bedisrupted by violating the first assumption [20]; this pa-per extends that work by showing a different way to ob-fuscate binaries by replacing control transfer instructions16th USENIX Security SymposiumUSENIX Association275with apparently-innocuous non-control-transfer instruc-tions. It is also very different from our earlier work onintrusion detection [21], which proposed a way to hindercertain kinds of mimicry attacks by obfuscating systemcall instructions. That work sought simply to disguise theinstruction (‘int$0x80’ in Intel x86 processors) used byapplications to trap into the OS kernel; more importantly,it required kernel modifications in order to to work. Bycontrast, the work described in this paper applies to arbi-trary control transfers in programs and requires no kernelmodifications. Taken together, these two differences leadto significant differences between the two approaches interms of goals, techniques, and effects.It is important to note that code obfuscation is merelya technique: just as it can be used to protect softwareagainst attackers, so too it can be used to hide maliciouscontent. The work presented here can therefore be seenfrom two perspectives: as a “defense model” of a newapproach for protecting intellectual property, or as an“attack model” of a new approach for hiding maliciouscontent. In either case, it goes well beyond current ap-proaches to hiding the content of executable code. Inparticular, the obfuscations cause the best existing dis-assemblers to miss 40%–60% of the instructions in testprograms and to make mistakes on over half of the con-trol flow edges.The remainder of the paper is organized as follows.Section 2 provides background information on static dis-assembly algorithms. Section 3 describes the new tech-niques for thwarting disassembly and explains how theyare implemented. Section 4 describes how we evaluatethe efficacy of our approach. Section 5 gives experimen-tal results for programs in the SPECint-2000 benchmarksuite. Section 6 describes related work, and Section 7contains concluding remarks.2 Disassembly AlgorithmsThis section summarizes the operation of disassemblersin order to provide the context needed to understandhow our obfuscation techniques work. Broadly speak-ing, there are two approaches to disassembly: static anddynamic, the difference between them being that the for-mer examines the program without execution, while thelatter monitors