slide 1 0x1A Great Papers in Computer Security Vitaly Shmatikov CS 380S http://www.cs.utexas.edu/~shmat/courses/cs380s/slide 2 After All Else Fails Intrusion prevention • Find buffer overflows and remove them • Use firewall to filter out malicious network traffic Intrusion detection is what you do after prevention has failed • Detect attack in progress • Discover telltale system modificationsslide 3 What Should Be Detected? Attempted and successful break-ins Attacks by legitimate users • Illegitimate use of root privileges, unauthorized access to resources and data … Malware • Trojan horses, rootkits, viruses, worms … Denial of service attacksslide 4 Intrusion Detection Systems (IDS) Host-based • Monitor activity on a single host • Advantage: better visibility into behavior of OS and individual applications running on the host Network-based (NIDS) • Often placed on a router, firewall, or network gateway • Monitor traffic, examine packet headers and payloads • Advantage: single NIDS can protect many hosts and look for global patternsslide 5 Intrusion Detection Techniques Misuse detection • Use attack “signatures” (need a model of the attack) – Sequences of system calls, patterns of network traffic, etc. • Must know in advance what attacker will do (how?) • Can only detect known attacks Anomaly detection • Using a model of normal system behavior, try to detect deviations and abnormalities • Can potentially detect unknown (zero-day) attacks Which is harder to do?slide 6 Misuse Detection (Signature-Based) Set of rules defining a behavioral signature likely to be associated with attack of a certain type • Example: buffer overflow – A setuid program spawns a shell with certain arguments – A network packet has lots of NOPs in it – Very long argument to a string function • Example: denial of service via SYN flooding – Large number of SYN packets without ACKs coming back …or is this simply a poor network connection? Attack signatures are usually very specific and may miss variants of known attacks • Why not make signatures more general?slide 7 Extracting Misuse Signatures Use invariant characteristics of known attacks • Bodies of known viruses and worms, RET addresses of memory exploits, port numbers of applications with known vulnerabilities • Hard to handle mutations – Polymorphic viruses: each copy has a different body Big research challenge: fast, automatic extraction of signatures of new attacksslide 8 Anomaly Detection Define a profile describing “normal” behavior • Works best for “small”, well-defined systems (single program rather than huge multi-user OS) Profile may be statistical • Build it manually (this is hard) • Use machine learning and data mining techniques – Log system activities for a while, then “train” IDS to recognize normal and abnormal patterns • Risk: attacker trains IDS to accept his activity as normal – Daily low-volume port scan may train IDS to accept port scans IDS flags deviations from the “normal” profileslide 9 Statistical Anomaly Detection Compute statistics of certain system activities Report an alert if statistics outside range Example: IDES (Denning, mid-1980s) • For each user, store daily count of certain activities – For example, fraction of hours spent reading email • Maintain list of counts for several days • Report anomaly if count is outside weighted norm Problem: the most unpredictable user is the most importantslide 10 “Self-Immunology” Approach Normal profile: short sequences of system calls • Use strace on UNIX … open,read,write,mmap,mmap,getrlimit,open,close … open,read,write,mmap read,write,mmap,mmap … write,mmap,mmap,getrlimit mmap,mmap,getrlimit,open … remember last K events Compute % of traces that have been seen before. Is it above the threshold? Y N normal abnormal Raise alarm if a high fraction of system call sequences haven’t been observed before [Forrest]slide 11 Level of Monitoring Which types of events to monitor? • OS system calls • Command line • Network data (e.g., from routers and firewalls) • Keystrokes • File and device accesses • Memory accesses Auditing / monitoring should be scalableslide 12 System Call Interposition Observation: all sensitive system resources are accessed via OS system call interface • Files, sockets, etc. Idea: monitor all system calls and block those that violate security policy • Inline reference monitors • Language-level: Java runtime environment inspects stack of the function attempting to access a sensitive resource to check whether it is permitted to do so • Common OS-level approach: system call wrapper – Want to do this without modifying OS kernel (why?)slide 13 Janus [Berkeley project, 1996]slide 14 Policy Design Designing a good system call policy is not easy When should a system call be permitted and when should it be denied? Example: ghostscript • Needs to open X windows • Needs to make X windows calls • But what if ghostscript reads characters you type in another X window?slide 16 Problems and Pitfalls Incorrectly mirroring OS state Overlooking indirect paths to resources • Inter-process sockets, core dumps Race conditions (TOCTTOU) • Symbolic links, relative paths, shared thread meta-data Unintended consequences of denying OS calls • Process dropped privileges using setuid but didn’t check value returned by setuid… and monitor denied the call Bugs in reference monitors and safety checks • What if runtime environment has a buffer overflow? [Garfinkel]slide 17 Incorrectly Mirroring OS State Policy: “process can bind TCP sockets on port 80, but cannot bind UDP sockets” 6 = socket(UDP, …) Monitor: “6 is UDP socket” 7 = socket(TCP, …) Monitor: “7 is TCP socket” close(7) dup2(6,7) Monitor’s state now inconsistent with OS bind(7, …) Monitor: “7 is TCP socket, Ok to bind” Oops! [Garfinkel]slide 18 TOCTTOU in Syscall Interposition User-level program makes a system call • Direct arguments in stack variables or registers • Indirect arguments are passed as pointers Wrapper enforces some security condition • Arguments are copied into kernel memory and analyzed and/or substituted by the syscall wrapper What if arguments change right here? If permitted by the wrapper,