QTools  5.9.3
QUTest™ Unit Testing Harness
qutest_banner.jpg

About QUTest™

QUTest™ (pronounced 'cutest') is a unit testing harness, which is specifically designed for deeply embedded systems, but also supports unit testing of embedded code on host computers ("dual targeting"). QUTest™ is the fundamental tooling for Test-Driven Development (TDD) of QP/C/C++ applications, which is a highly recommended best-practice.

Note
Even though QUTest™ has been primarily designed for testing of QP state machines, it can also be used to test any embedded C or C++ code. To demonstrate this capability, the QUTest™ comes with several examples of testing embedded code from the book "Test-Driven Development for Embedded C" by James W. Grenning. These tests include examples of such techniques as using test doubles and mocks with QUTest™.


qutest_ex.png
Example of a QUTest™ session on an embedded board (EFM32-SLSTK)

What's Special about QUTest™?

Unlike other existing unit testing harnesses for embedded systems (e.g., Unity or CppUTest) QUTest™ is not based on xUnit, which was originally designed to run tests on host computers. Instead, QUTest™ is geared towards unit testing of deeply embedded systems. Here is a list of QUTest™ unique features, specifically designed for this purpose:

  • QUTest™ separates the execution of the CUT (Code Under Test) from checking of the "test assertions". The embedded target is concerned only with running a test fixture that exercises the CUT and produces QP/Spy™ trace, but it does not check the "test assertions". Checking the "test assertions" against the expectations is performed on the host computer by means of test scripts. This approach is more intuitive for embedded developers and actually simplifies the development of all sorts of test doubles, including mocks, without breaking encapsulation of the CUT.
  • QUTest™ is a unique test harness on the embedded market that supports scripting. QUTest test scripts run on the Host, which skips compilation and uploading the code to the Target and thus shortens the TDD micro-cycle.
  • QUTest™ supports resetting the Target for each individual test, if needed. This goes far beyond providing test setup() and teardown() functions that other test fixtures offer (and of course QUTest supports as well). Clean reset of the Target avoids erroneous tests that implicitly rely on side effects from previously executed code. This is particularly important for embedded systems and for state machines, so that each test can start from a known reset condition.
  • QUTest™ supports testing Design by Contract (assertions in C or C++, not to be confused with "test assertions") in the CUT. This is a carefully designed, unique feature of QUTest not available in other test harnesses. A successful test of DbC might actually mean breaking an assertion in the Target code.
  • QUTest™ test fixtures that run on the Target do not require dynamic memory allocation (malloc()/free() in C or new/delete in C++). This means that you don't need to commit any of your precious embedded RAM to the heap (you can set the heap size to zero) and you don't need to link the heap management code. Avoiding dynamic memory allocation is one of the best practices of real-time embedded programming, which you don't need to compromise to run QUTest.
  • QUTest™ test fixtures that run on the Target do not require non-local jumps (setjmp()()/longjmp() in C or throw/catch in C++), which are needed by other test harnesses to discontinue failing tests. QUTest™ test fixtrues do not need to discontinue failing tests, because they don't check "testing assertions", so a test fixture does not "know" if it is failing or passing. Should a test fixture crash on the Target, it simply waits for the target reset commanded by a test script.
  • QUTest™ test fixtures can be based on the actual application code. For example you can reuse the same main() function in a test fixture and in your final application. This means that you can either grow your test fixture into a final application through TDD, or you can more easily add unit tests to an existing application.
Note
Even though QUTest™ is particularly suitable for running tests on deeply embedded targets, it also fully supports running the same tests on your host computer. In fact, running the tests as much as possible on the host and thus avoiding the target-hardware bottleneck is the highly recommended best-practice of embedded TDD. QUTest™ supports fully-automated unit testing, both on the embedded target and on the host computer.

Installing QUTest™ and Tcl

QUTest™ is bundled with QSPY and is installed automatically when you install QTools. Specifically, QUTest consists of the qspy.tcl and qutest.tcl Tcl scripts located in the qspy_files qtools/qspy/tcl folder.

Attention
Even though QUTest™ is implemented in Tcl (Tool Command Language), you don't need to know Tcl to write effective tests with QUTest™. The actual tests that you will write utilize a small, carefully designed set of commands that form a small DSL (Domain Specific Language) for unit testing. The Tcl interpreter just happened to be an effective vehicle to implement this "Unit Testing DSL".

Windows

In order to run QUTest, you need the Tcl interpreter called tclsh installed on your machine. tclsh (version 8.4) is included in the QTools collection for Windows, so it will be available if you install QTools™. Also, the Tcl/Tk distribution included in QTools has been already extended with the UDP-Sockets for Tcl, so you can use the tclsh and wish interpreters "as is" without any additional modifications.

Linux

On Linux, Tcl/Tk is usually installed, but you need to augment the standard Tcl/Tk distribution with the UDP Sockets for Tcl.

Note
In order to communicate with the QSPY back-end, QUTest requires the Tcl interpreter extended with UDP Sockets. Istructions of adding UDP Sockets to Tcl are included in the QTools collection for Linux, in the tcludp folder (see QSPY directories and files)

Running QUTest™

From the description so far it should be clear that before starting QUTest, you need to make sure that the QSPY host application is running on this machine with the -u command-line option. QSPY can be launched from a command-prompt and should connect to the Target system of your choice. You can also use the -t to open a TCP/IP socket for connecting host executables as Targets. For example, to start QSPY and open a TCP/IP connection, you can open a command prompt and type:

qspy -u -t

Second, because QUTest™ is implemented as a Tcl script, it requires the Tcl interpreter called tclsh (Tcl shell) to run. This Tcl interpreter needs to be augmented with the UDP socket extension, because the standard Tcl/Tk distributions typically don't support UDP. (NOTE: The Tcl interpreters included in the QTools collection for Windows are already augmented with the UDP socket extension, so you don't need to do anything to use them for QSpyView).

Once QSPY is running, you can run QUTest from a Windows command-prompt by typing:

tclsh %QTOOLS%\qspy\tcl\qutest.tcl

This would launch qutest in the current directory and look for test scripts (*.tcl files) to execute them as tests.

Attention
Typically, the qutest.tcl script is started from a Makefile, which also builds the code under test (CUT) and loads it into the Target. The following QUTest™ Tutorial introduces QUTest™ concepts while the QUTest™ Reference goes into more detail.

QUTest™ Examples

The QUTest™ examples are co-located with the QP frameworks to which they pertain.

  • QUTest™ examples for the QP/C framework are located in the QP/C installation directory, in the sub-directory qpc\examples\qutest.
  • QUTest™ examples for the QP/C++ framework are located in the QP/C++ installation directory, in the sub-directory qpcpp\examples\qutest

QUTest™ Structure

Testing with QUTest™ involves the following components:

  1. A test fixture code that is written in C (or C++) and runs in your embedded Target (or on the host as a host-executable). The test fixture is linked with the stub of the QP framework (see qutest.c), which provides the complete QP API to your CUT (Code Under Test), but instead of actually running the code as a QP application, it waits for and executes commands from the QSPY host application. The design of test fixtures is explained in the QUTest™ Tutorial.
  2. The QSPY host application started with the -u command-line option (UDP socket) and with the proper communication link for your Target (e.g., -t command-line option (TCP/IP) link for a host executable);
  3. A test script that drives the tests and checks the generated QSPY output against the expectations ("test-assertions"). The design of test scripts is explained in the QUTest™ Tutorial.

The sequence diagram below shows the communication between these QUTest™ components. The embedded Target is running a test fixture code that communicates with QSPY over the Target data link (red arrows). This communication is based on the QP/Spy Protocol. The QUTest™ (Tcl script) attaches to the UDP text channel of the QSPY host application. This communication uses the QSPY text format (green arrows) for data forwarded from the Target and binary format for commands sent to the Target (blue arrows).

Note
The QUTest Front-End attaches to the text-channel of the UDP socket served by QSPY.
qspy_qutest.gif
Communication between Target, QSPY, and QUTest
  • A The QUTest™ front-end sends commands to the QSPY (e.g., target reset), which are relayed to the Target

  • B The Target responds by sending QS trace records to QSPY, which parses them and sends the text-format to QUTest. The QUTest then compares the produced output to the expected output (checks the "test assertions")

  • C UDP packets with Record-ID in the range 128..255 are not relayed to the Target, but instead are used for communication between the QUTest™ Front-End and the QSPY Back-End.
Note
As you can see, the QUTest structure separates test execution from checking the test expectations (a.k.a. "test assertions"). This leads to a testing strategy most suitable to deeply embedded systems, where the embedded Target needs to execute only a minimal test fixture code, while actual driving of the tests and checking "test assertions" is performed on a powerful host computer. In practice, a well designed test fixture can serve many tests, which greatly increases the flexibility of testing, because tests are just scripts, which run in seconds and don't require compilation and uploading to the Target.

Host Executable

The general QUTest™ structure just described corresponds to running tests on an embedded Target. But QUTest™ can also execute tests on the host computer. In that case (shown in the figure below), the test fixture is a host executable that communicates with the QSPY host application via a TCP/IP socket (QSPY started with the -t command-line option). In this case all QUTest™ components executes on the host computer.

qspy_host.gif
QUTest with Host Executable

Next: QUTest™ Tutorial