QTools  5.9.3
QP/Spy™ Software Tracing

In any real-life project, getting the code written, compiled, and successfully linked is only the first step. The system still needs to be tested, validated, and tuned for best performance and resource consumption. A single-step debugger is frequently not helpful because it stops the system and exactly hinders seeing live interactions within the application. Clogging up high-performance code with printf statements is usually too intrusive and simply unworkable in most embedded systems, which typically don't have adequate screens to print to. So the questions are: How can you monitor the behavior of a running real-time system without degrading the system itself? How can you discover and document elusive, intermittent bugs that are caused by subtle interactions among concurrent components? How do you design and execute repeatable #ref qutest "unit tests" and integration tests of your system? How do you ensure that a system runs reliably for long periods of time and achieves optimal performance?

Techniques based on software tracing can answer many of these questions. Software tracing is a method for obtaining diagnostic information in a live environment without the need to stop the application to get the system feedback. Software tracing always involves some form of a target system instrumentation to log interesting discrete events for subsequent retrieval from the system and analysis.

What is it?

QP/Spy™ is a software tracing and testing system specifically designed for embedded systems, such as single chip microcontrollers. The job of QP/Spy is to capture information about an embedded code's execution and send it to the host computer with minimal impact on the real-time performance of the embedded code. QP/Spy can also send commands and data to the embedded target, which serves as the basis for Unit Testing and for Visualization subsystems of the QP/Spy system.

Software tracing is particularly effective and powerful in combination with the event-driven reactive programming model, such as the one implemented in QP active object frameworks. Due to the inversion of a control, a reactive framework controls almost all interesting interactions in the system, so an instrumented event-driven framework can provide much more comprehensive and detailed information than any traditional RTOS.

How it works?

In a nutshell, working with QP/Spy™ is similar to peppering the code with printf statements for logging and debugging, except that QP/Spy™ is much less intrusive, and more lightweight, portable and selective than the primitive printf.

The main difference between the QP/Spy™ tracing system and peppering the code with printf statements is where the data formatting and sending is done. When you use printfs, the data formatting and sending occur in the time-critical paths through the embedded code. In contrast, the QP/Spy system produces raw binary data, so all the time-consuming formatting is removed from the embedded system and is done after the fact in the host computer. Additionally, in QP/Spy, data logging and sending to the host are separated so that the embedded system can typically perform the transmission outside of the time-critical path, for example in the idle processing of the embedded CPU. Finally, the code size of the target-resident component in QP/Spy is merely a few hundred bytes, which contrasts with several kilobytes of code required by a full-blown printf formatter.

The picture below shows a typical setup for software tracing. The embedded Target system is executing instrumented code, which logs the trace data into a RAM buffer inside the Target. From that buffer the trace data is sent over a data link to a Host computer, which stores, displays, and analyzes the information. This configuration means that a software tracing always requires two components: a "Target resident component" for generating and sending the trace data (QS in QP/Spy™), and a "Host resident component" to receive, decompress, visualize, and analyze the data (QSPY in QP/SPy™).

Typical setup for software tracing with QP/Spy™
Software tracing instrumentation logs interesting discrete events that occur in the target system. These discrete events will be called trace records, to avoid confusing them with the application-level events.

A good tracing solution, such as QP/Spy, is minimally intrusive, which means that it can provide visibility into the running code with minimal impact on the target system behavior. Properly implemented and used, it will let you diagnose a live system without interrupting or significantly altering the behavior of the system under investigation.

Of course, it's always possible that the overhead of software tracing, no matter how small, will have some effect on the target system behavior, which is known as the probe effect (a.k.a. the "Heisenberg effect"). To help you determine whether that is occurring, you must be able to configure the instrumentation in and out both at compile-time as well as at run-time.

To minimize the "probe effect", a good trace system performs efficient, selective logging of trace records using as little processing and memory resources of the target as possible. Selective logging means that the tracing system provides user-definable, fine granularity filters so that the target-resident component only collects events of interest you can filter as many or as few instrumented events as you need. That way you can make the tracing as noninvasive as necessary.

To minimize the RAM usage, the target-resident trace component typically uses a circular trace buffer that is continuously updated, and new data overwrites the old when the buffer "wraps around" due to limited size or transmission rate to the host. This reflects the typically applied last-is-best policy in collecting the trace data. In order to focus on certain periods of time, software trace provides configurable software triggers that can start and stop trace collection before the new data overwrites the old data of interest in the circular buffer.

To further maximize the amount of data collected in the trace buffer, the Target-resident component typically applies some form of data compression to squeeze more trace information into the buffer and to minimize the bandwidth required to uplink the data to the Host.

However, perhaps the most important characteristic of a flexible software tracing system is the separation of trace logging (what is being traced) from the data transmission mechanism (how and when exactly the data is sent to the Host). This separation of concerns allows the transmissions to occur in the least time-critical paths of the code, such as the idle loop. Also, clients should be able to employ any data transmission mechanism available on the Target, meaning both the physical transport layer (e.g., serial port, SPI, USB, Ethernet, etc.) as well as implementation strategy (polling, interrupt, DMA, etc.). The tracing facility should tolerate and be able to detect any RAM buffer overruns due to bursts of tracing data production rate or insufficient transmission rate to the host.

Finally, the tracing facility must allow consolidating data from all parts of the system, including concurrently executing threads and interrupts. This means that the instrumentation facilities must be reentrant (i.e., both thread-safe and interrupt-safe). Also, to be able to correlate all this data, most tracing systems provide precise time-stamping of the trace records.

Bi-Directional Connection to the Target

While traditional software tracing systems support only uni-directional output of trace data from the embedded Target to a Host computer, QP/Spy supports bi-directional communication to the target as well. This capability allows users to send commands and data to the Target and form the basis for Unit Testing and Visualization and Monitoring of the embedded Target.

UDP Socket Extension

The QP/Spy system provides a UDP socket, which is open for communication with various Front-Ends (GUI-based or "headless"). Currently, the UDP connection point is used by the QUTest headless (console-based) front-end and GUI-based QSpyView front-end.

QP/Spy Session Example

To give you a better idea how QP/Spy works, the listing below shows an example output from a QP/Spy session. The left-hand side shows the raw, binary output generated by the target-resident component (QS). The right-hand side shows the human-readable format generated from the same data by the host-resident component (QSPY). The compression ratio between the binary and textual outputs in this data sample is about 3.7.

Example of the QP/Spy output

The following sections explain the concepts and components of QP/Spy™:

Next: QP/Spy™ Data Protocol