Arnold S. Berger - Debugging Embedded and Real-Time Systems: The Art, Science, Technology, and Tools of Real-Time System Debugging

Arnold S. Berger

Why write a book about debugging real-time systems? Good question, Im glad you asked. A lot has been written about debugging real-time systems or embedded systems, but what has been written has not, to the best of my knowledge, been collected into one resource, such as a book.

After having taught embedded system design for many years, Ive come to the conclusion that we are failing as teachers because our students can write a program in assembly, C, C ++, C#, some Arduino dialect, or Verilog, and get their program to compile. However, if problems crop up, as they invariably do, students lack the diagnostic skills to analyze the problems and, in a systematic way, zero in on the possible causes, and then find and fix the bugs. I hope to address this issue in the chapters to come.

What I observe with depressing regularity is that students take the shotgun approach. Try a bunch of changes at once and hope for the best. Even more disturbing, rather than try to find and fix a problem, students will just throw away their code or their prototype and start all over again, hoping beyond hope that will fix the problem.

You might assume when the students of today become the engineers of tomorrow and are totally immersed in product design, they will have developed the debugging skills they need to do their job in the most effective manner. Ive learned that assumption does not hold true.

Before I became an academic, I worked in industry, creating design and debug tools for embedded systems designers. In particular, I designed and led teams that designed logic analyzers, in-circuit emulators, and performance analyzers. These were and in many cases still are complex instruments designed to solve complex problems. Just learning to effectively use one of these instruments can be a chore that many engineers dont feel the desire to invest the time required to learn.

Maybe youve been there yourself. Do you do a mental cost/benefit analysis to invest the time to wade through a set of manuals, or just dive in and hope for the best? One of the most brilliant engineers I ever worked with, John Hansen, made this observation that came to be known as Hansens Law, which says:

If a customer doesnt know how to use a feature, the feature doesnt exist.

So, as vendors of these complex and expensive debugging tools, we certainly own a good part of the problem. We have not been able to effectively transfer technology to a user in a way that allows the user to take full advantage of the power of the tool to solve a problem.

Heres another example. I remember this one vividly because it led me to think a whole new way about technology and how to transfer it. Well come back to it in a later chapter, but this is a good time to introduce the problem. It involves logic analyzers. For many years, the logic analyzer has been one of the premier tools for real-time system analysis. Theres some evidence that dominance may be changing, but for now, well assume the logic analyzer still holds a position of prominence.

Suppose you are trying to debug a complex, real-time system with many high-priority tasks running in parallel. Stopping the processor to single step through your code is not an option, although many excellent debuggers are task-aware, so they may be able to single step in a particular task without stopping other tasks from running at full speed.

The logic analyzer sits between the processor or processors and the rest of the system, and records in real time the state of every address bit, data bit, and status bit output from the processor, then inputs to the processor as they occur in real-time. Once the buffer or recording memory is full, the engineer can then trace through it and see what exactly transpired during the time interval of interest.

But how do you define the time interval of interest? Your memory buffer is not infinitely large, and your processor is clipping along at 10 s to 100 s of millions of bus cycles every second. This is where the logic analyzer really shines as a tool. The user can define a sequence of events through the code, very much like a sequence of states in a finite state machine. In some logic analyzers, these states can be defined in terms of the high-level C ++ code with which the engineer is accustomed to programming.

If the user gets the sequence of states correctly defined, the logic analyzer will trigger (capture) the trace at just the right time in the code sequence to show where the fault occurs. Here is where it gets interesting. At the time, below.