Systems and Internet Infrastructure Security Network and Security Research Center Department of Computer Science and Engineering Pennsylvania State University University Park PA Advanced Systems Security Securing Commercial Systems Trent Jaeger Systems and Internet Infrastructure Security SIIS Lab Computer Science and Engineering Department Pennsylvania State University Systems and Internet Infrastructure Security SIIS Laboratory February 18 2010 Page 1 Commercial Systems Focus on Performance Flexibility and Protection Do not satisfy the reference monitor concept But lots of folks use them so big potential benefit to making a commercial system secure Not so easy so lots of lessons learned and new ideas Systems and Internet Infrastructure Security SIIS Laboratory Page 2 Retrofitting Security Make legacy code satisfy the reference monitor concept Did the rules that we set depend on whether we built the reference validation mechanism from scratch or from legacy code Which are hardest Is it worth trying Why Systems and Internet Infrastructure Security SIIS Laboratory Page 3 Commercial Reference Monitors 1970s 80s take an existing OS and add a reference validation mechanism KVM 370 VAX VMS Secure Xenix 1980s 90s Use microkernel architectures to deploy secure UNIX systems like security kernel DTMach DTOS Fluke Flask 1990s 2000s UNIX systems for secrecy and integrity IX DTE LOMAC Systems and Internet Infrastructure Security SIIS Laboratory Page 4 Early MLS Systems Data Secure UNIX and KSOS Compatible with the UNIX API Emulation UNIX calls implemented by KSOS kernel Emulation with MLS had a significant performance impact Scomp did not do emulation Insecure features of the UNIX interface presents problems too Fork when a new process is created can share file descriptors Why might this be a problem Systems and Internet Infrastructure Security SIIS Laboratory Page 5 Early MLS Systems KVM 370 Virtual machine monitor for MLS systems Retrofit of VM 370 for MLS Indirect security sensitive operations to the VMM Significant performance effect 25 or more Context switch to VMM Reuse of VM 370 code limited optimizations Systems and Internet Infrastructure Security SIIS Laboratory Page 6 Early MLS Systems VAX VMS DEC and Sandia Labs retrofit VAX VMS for MLS Retrofit identified several vulnerabilities Prototype system Systems and Internet Infrastructure Security SIIS Laboratory Page 7 Early MLS Systems Secure Xenix later Trusted Xenix Xenix PC version of UNIX from Microsoft Goal run Xenix apps without modification Two Problems How do two instances of the same program running at different secrecy levels access the file system securely Consider tmp should a less secret version use a temp file created by the more secret version How does a user know that she is communicating with the trusted computing base Systems and Internet Infrastructure Security SIIS Laboratory Page 8 Hidden Directories Now called Polyinstantiated File Systems Idea There is a version of a directory for each secrecy level When a process opens a file it opens the version based on its secrecy level How does this ensure MLS information flows Systems and Internet Infrastructure Security SIIS Laboratory Page 9 Trusted Path Mechanism that enables a user to communicate with the system s trusted computing base E g Key attention sequence ctrl alt del Communication from user to TCB and from TCB to user Tough for windowing systems as we ll see Systems and Internet Infrastructure Security SIIS Laboratory Page 10 Microkernels and Security Kernels Microkernel systems emerged at this time and they were gaining mindshare quickly as the way future OS s would be constructed Mach microkernel Security kernel requirements Verifiable design Map to implementation Should be easier for a microkernel than for a conventional kernel Systems and Internet Infrastructure Security SIIS Laboratory Page 11 Microkernel Architecture Operating system consists of a core kernel component microkernel and a set of servers that implement traditional OS function Microkernels provide base function needed by all processing Scheduling IPC Basic Device Access IRQs and others are discretionary OS Servers implement system specific function Memory managers file systems networking processes naming device drivers advanced scheduling Idea customize system on microkernel for function and performance oh yeah and security too Systems and Internet Infrastructure Security SIIS Laboratory Page 12 TMach DTMach DTOS MLS Mach systems Similar goals but built by competing companies Trusted Information Systems and Secure Computing Corp Approach built MLS aware servers on microkernel Applications would run at a single level Some also considered integrity A variety of innovations resulted we ll take about some in DTE also Systems and Internet Infrastructure Security SIIS Laboratory Page 13 Integrity The designers of these systems also considered protecting the integrity of computations For example they envisioned a Clark Wilson like model where high integrity data would be modified by a sequence of high integrity operations How would they ensure that only these operations in that sequence would modify high integrity data Systems and Internet Infrastructure Security SIIS Laboratory Page 14 Assured Pipelines A sequence of high integrity processes that take input high integrity data to output Use an MPS Input data is given a label Each process is given a unique label Each process s output is given a unique label Connect processes into a sequence based on data labels they input Use Type Enforcement for this Systems and Internet Infrastructure Security SIIS Laboratory Page 15 Mach Security Server Problem Enable multiple independent servers to enforce a coherent policy How do server work together on security Consider file opening When a process requests opening of a file Kernel authorizes process to access file server The file server asks the security server if this access is authorized The security server examines the policy and determines the answer Systems and Internet Infrastructure Security SIIS Laboratory Page 16 Kernel v Server Enforcement Mach allows a party with the permission to send an IPC to a server to send any message to that server Suppose a process can memory map files into its address space via a memory server Then it can send any request to the memory server Why is this a problem Choice between enforcement layers and complexity Kernel can enforce control if it understands the semantics