A race-cure case studyOur recent ‘race’ exampleNo interventions!Linux’s solutionBut how does it work?Is ‘open source’ enough?‘LXR’ can helpFrom: <arch/i386/kernel/time.c>Another approach…‘objdump’ can disassemble‘grep’ can do filtering‘System.map’Example on our machinesNow we know where to look…Slide 15What we discoverUse a text-editorThe complete functionSome ‘magic’ numbersThe ‘cmpxchg’ instruction‘spinlock’The ‘System-map’ againWhat is ‘per_cpu’ data?Role of segmentationEach CPU has its own GDTIn-class exercise #1What’s in register FS?In-class exercise #2‘virtual-to-physical’A race-cure case studyA look at how some standard software tools can illuminate what is happening inside LinuxOur recent ‘race’ example•Our ‘cmosram.c’ device-driver included a ‘race condition’ in its ‘read()’ and ‘write()’ functions, since accessing any CMOS memory-location is a two-step operation, and thus is a ‘critical section’ in our code:outb( reg_id, 0x70 );datum = inb( 0x71 );•Once the first step in this sequence is taken, the second step needs to followNo interventions!•To guarantee the integrity of each access to CMOS memory, we must prohibit every possibility that another control-thread may intervene and access that same i/o-port •The main ways in which an intervention by another ‘thread’ might happen are:–The current CPU could get ‘interrupted’; or–Another CPU could access the same i/o-portLinux’s solution•Linux provides a function that an LKM can call which is designed to insure ‘exclusive access’ to a CMOS memory-location:datum = rtc_cmos_read( reg_id );•By using this function, a programmer does not have to expend time and mental effort analyzing the race-condition and devising a suitable ‘cure’ for itBut how does it work?•As computer science students, we are not satisfied with just using convenient ‘black-box’ solutions which we don’t understand•Such purported ‘solutions’ may not always accomplish everything that they claim – if they perform correctly today, they still may fail in some way in the future (if hardware changes); we don’t want to be helpless!Is ‘open source’ enough?•In theory we could try to track down the actual behavior of the ‘rtc_cmos_read()’ function, by reading Linux’s source-code •But is that really a practical approach?•In some cases the answer might be ‘yes’, but in other situations it might be ‘no’!•Life is short, and the kernel source-files are very numerous – with many layers‘LXR’ can help•The Linux Cross-Reference tool offers a way to automate searching kernel source•This tool is online (see our website’s link under ‘Resources’) and it is hosted on a server in Norway: http://lxr.linux.no/•Here you just click on “Browse the Code”From: <arch/i386/kernel/time.c>unsigned char rtc_cmos_read(unsigned char addr) { unsigned char val; lock_cmos_prefix( addr ); outb_p( addr, RTC_PORT(0) ); val = inb_p( RTC_PORT(1) ; lock_cmos_suffix( addr ); return val; } EXPORT_SYMBOL( rtc_cmos_read );Another approach…•There is an alternative to searching kernel source files -- which may well be faster •We can use some standard command-line tools, including ‘objdump’ and ‘grep’•In this approach, we look at the compiled kernel’s object-file, named ‘vmlinux’, found normally in the ‘/usr/src/linux’ subdirectory•Using ‘objdump’ that file can be parsed!‘objdump’ can disassemble•Change the current working directory:$ cd /usr/src/linux•Then, to disassemble the ‘vmlinux’ kernel file we use can this command:$ objdump -d vmlinux •But the amount of output will be huge, so it’s hard to find the part we’re interested in‘grep’ can do filtering •If we want to see the ‘rtc_cmos_read’ code we could use ‘grep’ to eliminate irrelevant parts of the disassembly-output:$ objdump –d vmlinux | grep rtc_cmos_read•But we still see too many lines of output (because the ‘rtc_cmos_read()’ function gets called at many places in the kernel)‘System.map’•We can use a special textfile, located in the ‘/boot’ directory, which tells us where each ‘exported’ kernel-symbol will reside at run-time in the virtual address-space •You can use ‘cat’ to look at this textfile:$ cat /boot/System.map•And you can use ‘grep’ to find only the symbol you care about:$ cat /boot/System.map | grep rtc_cmos_readExample on our machines$ cat /boot/System.map-2.6.22.5cslabs | grep rtc_cmos_readc0105574 T rtc_cmos_readc029b8a8 r __ksymtab_rtc_cmos_readc02a0bff r __kstrtab_rtc_cmos_readNote that the usual ‘symbolic link’ is missing from the ‘/boot’ directoryon our class and lab machines -- so you have to type a longer nameWith superuser privileges this could be fixed using the ‘ln’ command: root# ln System.map-2.6.22.5cslabs System.mapNow we know where to look…•From the ‘System.map’ we learn where in the kernel our ‘rtc_cmos_read()’ function will reside•We can ‘extract’ that function’s code, for study purpose, using these steps:–Save the complete ‘vmlinux’ disassembly–Use ‘grep’ to find its starting-address–Use ‘vi’ to delete earlier and later instructions•Step 1: saving the ‘vmlinux’ disassembly$ objdump –d /usr/src/linux/vmlinux > ~/vmlinux.asm •Step 2: finding our function’s entry-point$ cat ~/vmlinux.asm | grep -n c0105574What we discoverFind the line that shows this virtual address (with colon)$ cat vmlinux.asm | grep -n c0105574:6812:c0105574: 53 push %ebx…and tell us which line-number it’s onOK, here’s that line…and this is it’s line-numberUse a text-editor•Remove all the lines in your ‘vmlinux.asm’ textfile whose line-numbers precede 6812•Scroll down, to find where your function ends (i.e., find its return-instruction ‘ret’):c01055b7: c3 ret•Delete all the lines that follow the ‘return’The complete functionc0105574 <rtc_cmos_read>:c0105574: 53 push %ebxc0105575: 9c pushfc0105576: 5b pop %ebxc0105577: fa clic0105578: 64 8b 15 08 20 30 c0 mov %fs:0xc0302008,%edxc010557f: 0f b6 c8 movzbl %al,%ecxc0105582: 42 inc %edxc0105583: c1 e2 08