Developing a good bedside manner The fastest and most effective way to debug hardware or software involves using a disciplined process. Here are six steps for debugging. By Jack Ganssle Embedded.com (09/23/09, 12:00:00 AM EDT) Why are debugging code and alien abduction so similar? Because in both cases large blocks of time are unaccounted for. Medicine, too, resembles debugging (and perhaps alien abduction). In an astonishing development, recently my doctor actually chatted with me for a few minutes. Maybe it was a slow H1N1 day, or perhaps he just wanted a break from all of the runny noses. So I asked him what the hardest part of his job was. Expecting to hear a rant against insurance companies, it was interesting to hear him talk about the difficulty in diagnosing diseases. Most people present with simple cases, but sometimes an individual will have a bewildering array of symptoms that suggest no single etiology. Physicians use differential diagnosis to try and weed causation from the complaints, which is made very complex since patients may ignore some important symptoms while focusing on those that are less critical. To add spice to the sauce of diagnosis, several illnesses may present at the same time. I was struck by how closely his comments mirror the art of debugging embedded systems. Our nascent product has a bug, which presents itself via some set of symptoms. Press the green button and nothing happens. Is the switch wired incorrectly? Could its Schmidt trigger gate be shot? Is the ISR invoked? Perhaps the ISR never passes a signal to the code that displays a result. Or maybe the power is off. In other cases, just as in medicine, one bug may present a variety of odd effects. Or a single symptom could stem from a combination of bugs all interacting in excruciating-complex, and hard to diagnose, manners. I wonder if physicians observe the infrequent symptoms we see, that appear in a system once, go away for weeks, and then randomly resurface? Then there are the bugs we know exist, but just cannot fix. The system gets shipped with the expectation that sometime someone will see a problem. That, too, is like medicine. "Doc, it hurts when I do this." "Don't do that." My health insurance will not cover kidney stones due to three prior episodes, one of which required an expensive lithotripsy. So that's one latent bug in my gut that will surely reappear at some unknown time in the future. Stretching the medicine analogy probably too far, the human body swarms with amazing self-repair facilities, rather like a well-designed product sporting watchdog timers, ECC, interlocks, and other protection and recovery mechanisms. Where medicine and debugging part ways, though, is that in most cases of illness the causes are known. The problem is to pare through the symptoms to correlate them with diseases listed in diagnostics manuals. We have no such manual since there are a huge number of ways a system can fail, and each of those can stem from a staggering number of root causes. So our problem set is vastly greater than the doctor's. Which seems odd, considering that even one of the trillions of cells that make up the human system is astronomically more complex than the biggest embedded system. But at least we can cut into the patient and change parts till we find the problem. A breakpoint stops its heart; a watchpoint instruments the brain and a recompile chainsaws through any part of the not-quite cadaver. Press "go" and the patient patiently restarts. Dr. Frankenstein could only drool with envy at our god-like powers of reanimation. Page 1 of 7Embedded Systems Design - Embedded.com10/18/2009http://www.embedded.com/220100899?printable=trueDebugging and troubleshooting, which I often use interchangeably since in an embedded system the software and hardware are synergistic, are both art and science. Art, because debugging uses skills built over a lifetime of experience. We draw on similar situations encountered in the past. It's craftsmanship, where much of our skill comes from years of hands-on work. Book learning helps, but like tradespeople, we must serve a long apprenticeship to get a visceral feel for where the clogs typically appear in the plumbing or why the HVAC unit isn't cooling zone three. Debugging is also science, because these are complex problems that will usually not yield to random efforts. We must be disciplined in how we chase down a bug. Troubleshooting comprises six steps which, in part, neatly parallels the scientific method: Observe collateral behavior Round up the usual suspects Generate a hypothesis Invent a fix Test the hypothesis Apply feedback First, we observe collateral behavior. The best embedded engineers I've had the pleasure of working with approach their work with a doctor's bedside manner. When a symptom appears, they take their hands off the keyboard and listen very closely to the patient--they look at the symptoms and see what other odd things might be happening. A single bug may manifest itself in several ways, which taken together can help nail it quickly. They use all of their senses. Does that smell like a burning resistor? Why does the stepper motor now sound faster? That processor never felt quite so hot before! Next, round up the usual suspects. Too many of us embedded people immediately start setting breakpoints instead of listening to the patient. I see engineers (and have done it myself) wandering into a labyrinth of debugging when the cause is simple. Just this weekend we sailed to Queenstown on Maryland's Eastern Shore and rafted with three other boats. When it was time to go, I fired up Voyager's diesel, but the instruments didn't come on. Out came the schematics, the VOM, and I started tracing wires and taking data. A friend (who happens to be a great embedded systems engineer) said: "Gee, my boat has a circuit breaker on the engine itself. It's hard to get to but could that be the cause?" It was. Check the simple stuff, first. And don't rely on assumptions. A very long time ago I was chasing a devilishly convoluted series of problems in my code. The symptoms kept changing. Days went by. My boss, who knew nothing about microcomputers, suggested checking the power supply. I sniggered --what an inane idea! He insisted, and the voltmeter showed a solid 5 volts. It felt good to be vindicated. Until, that is, he put a scope on Vcc and found a high frequency ripple. Swapping power supplies cured all of the myriad
View Full Document
Unlocking...