CS 380S - 0x1A Great Papers in Computer SecurityFall 2012MIDTERMNovember 1, 2012DO NOT OPEN UNTIL INSTRUCTEDYOUR NAME:Collaboration policyNo collaboration is permitted on this assignment. Any cheating (e.g., submitting anotherperson’s work as your own, or p ermitting your work to be copied) will automatically resultin a failing grade.1Midterm (45 points)Problem 1 (3 points)Why is calling free() twice on the same memory object in a C program a potential securityproblem?Problem 2In the DEC Alpha assembly language, all instructions are 4-bytes wide and must start onan aligned 4-byte boundary. Here are some examples:• br Ra, dispAn unconditional relative branch. This instruction stores the address of the next in-struction in Ra and then skips disp instructions, where disp may be negative. Forexample, br r13, -5 jumps back 5 instructions (this may happen in a loop, for ex-ample).• jmp Ra, (Rb)Jump to register. Stores the address of the next instruction in Ra, then jumps andstarts executing co de at address Rb.• ldq Rv, disp (Ra)Load. Takes the memory address contained in register Ra, adds disp to it, and loadsthe value of the memory location at this address into register Rv.• stq Rv, disp (Ra)Store. Takes the memory address contained in register Ra, adds disp to it, and storesthe value of register Rv into the memory location at this a ddress.• bis Ra, Rb, RcCompute bitwise OR of Ra and Rb and store it into Rc.• and Ra, Rb, RcCompute bitwise AND of Ra and Rb and store it into Rc.2Problem 2a (3 points)Fault isolation requires inserting special checking code before every unsafe instruction, i.e.,an instruction that may potentially write or execute memory outside the fault domain. Forexample, a store instruction stq Ra, 0(Rb) is unsafe if it cannot be statically checked thatthe address contained in Rb is within the fault domain’s data segment.In the following list, circle the instruction(s) which can be unsafe:• br Ra, disp where disp falls within the fault domain’s code segment.• jmp Ra, (Rb)• ldq Rv, disp (Ra)• bis Ra, Rb, RcProblem 2b (2 points)Suppose that the unsafe store instructions are “sandboxed” as follows. We use dedicatedregisters r20 and r21 to store, in the positions corresponding to the segment identifier partof a memory address, all-zero bits and the segment ID bits, respectively. If the code containsan unsafe store instruction stq r2, 0(r1), it is replaced by the following three instructions:and r1, r20, r1bis r1, r21, r1stq r2, 0(r1)How can you subvert the safety of the system that uses this sandboxing mechanism?Problem 2c (3 points)Suppose communication between fault domains is implemented as follows. For each fa ultdomain, the trusted execution environment inserts special “stubs” (little snippets of code)into a special region of that domain’s code segment . Because the code of the stubs is trusted,it may contain unsafe instructions. Furthermore, the stubs are the only part of the faultdomain’s code segment that is allowed to have instructions branching outside of this codesegment.When a trusted caller calls an untrusted function, it branches to the “entry” stub, whichcopies arguments, saves registers that must be changed when switching fault domains, and3passes control to the untrusted code. When the untrusted code returns, it jumps directly tothe “return” stub in its code segment, which restores the context and returns to the caller.How can you subvert the safety of the system t hat uses this cross-domain communicationmechanism?Problem 2d (3 points)How should you implement the “stubs” for cross-domain communication so that they cannotbe subverted? You may explain or draw a picture.Problem 3 (6 points)Imagine a static-analysis tool for checking source C code to ensure that it satisfies a certainset of rules. Each rule is expressed by a finite-state automaton, with a special ERRO R state.As the checker scans the code, it keeps track of the current state in the automaton. If astate labelled ERROR is ever reached, then the checker reports an error in the code.Draw finite-state automata represent ing the following security rules. If you believe t herule cannot be expressed by a finite-state automaton, explain why.• Immediately before each call to strcpy(dest,src), the program must check the lengthof src by calling strlen(src).4• Each tempo r ary file used by the program must be created using mkstemp(), written,and eventually closed.• The return value of every call to malloc must be immediately checked to ensure thatit is not NULL.Problem 4Problem 4a (2 points)Suppose an HTTPS page links to an HTTP ifra me where both are loaded from the samedomain. The browser shows a mixed content warning dialog. Explain the risk of clickingOK on this dialog.5Problem 4b (2 points)Suppose an HTTPS page links to an HTTP iframe where the two are loaded from differentdomains. Should the browser display a warning dialog? Explain.Problem 5 (3 points)Tatebayashi, Matsuzaki, and Newman (TMN) proposed the following proto col, which enablesAlice and Bob to establish a shared symmetric key K with the help of a trusted server S.Both Alice and Bob know the server’s public key KS. Alice randomly generates a t emporarysecret KA, while Bob randomly generates a new key K to be shared with Alice. The protocolthen proceeds as follows:Alice → Server encKS(KA)Bob → Server encKS(K)Server → Alice K ⊕ KAAlice recovers key K as KA⊕ (K ⊕ KA)In this protocol, Alice sends her secret to the Server encrypted with the Server’s publickey, while Bob sends to the Server the new key, also encrypted with the Server’s public key.The Server XORs the two values together and sends the result to Alice. Therefore, bothAlice and Bob know K.Suppose that evil Charlie eavesdropped on Bob’s message to the Server. How can he,with the help of his equally evil buddy Don, extract the key K that Alice and Bob are usingto protect their communications?Assume that Charlie and Don can engage in the TMN protocol with the Server, but theydon’t know the Server’s private key.6Problem 6 (4 points)Integrity is an important element of an information flow policy. Suppose there are two levelsof integrity, T for Trusted and U for Untrusted. Intuitively, untrusted data should not beallowed to corrupt trusted data. That is, data from untrusted variables should not be allowedto flow to trusted variables.Examine the following fo ur statements, which have integrity labels as