QTools  7.3.4
Collection of Host-Based Tools
Loading...
Searching...
No Matches
Installation & Use

QUTest™ Unit Testing HarnessQUTest Concepts

In order to run tests in the QUTest test system, you need the following three components:

  1. The test fixture running on a remote target (or the host computer)
  2. The QSPY host application running and connected to the target
  3. The qutest.py script runner and some test scripts

This section describes how to install and use the "qutest.py script runner" component.

Installation

The "qutest.py script runner" can be used standalone, without installation in your Python system (see Examples below).

Note
The qutest.py script is included in the QTools™ collection. Also, the QTools™ collection for Windows already includes Python (3.x), so you don't need to install anything extra.

Alternatively, you can use your own Python installation, into which you can install the latest QUTest™ with pip from GitHub by executing the following command:

pip install https://github.com/QuantumLeaps/qtools/releases/latest/download/qutest.tar.gz

If you are using QUTest™ as a standalone Python script, you invoke it as follows:

python3 <path-to-qutest-script>/qutest.py <command-line-options>

Alternatively, if you've installed qutest.py with pip, you invoke it as follows:

qutest <command-line-options>

Command-line Options

The Python test scripts are executed by the QUTest test script runner qutest.py (located in qtools/qutest/ folder), with the following usage:

Attention
The qutest.py script runner command-line options have been expanded and changed at version 7.2.0. Unfortunately, it was not possible to preserve the backwards compatibility with the earlier versions.
usage: python qutest.py [-h] [-v] [-e [EXE]] [-q [QSPY]] [-l [LOG]] [-o [OPT]] [scripts ...]

QUTest test script runner

positional arguments:
  scripts               List (comma-separated) of test scripts to run

options:
  -h, --help            show this help message and exit
  -v, --version         Display QUTest version
  -e [EXE], --exe [EXE]
                        Optional host executable or debug/DEBUG
  -q [QSPY], --qspy [QSPY]
                        optional qspy host, [:ud_port][:tcp_port]
  -l [LOG], --log [LOG]
                        Optional log directory (might not exist yet)
  -o [OPT], --opt [OPT]
                        xciob: x:exit-on-fail,i:inter,c:qspy-clear,o:qspy-save-txt,b:qspy-save-bin

More info: https://www.state-machine.com/qtools/qutest.html

where:

Parameters
[scripts]optional specification of the Python test scripts to run. If not specified, qutest.py will try to run all *.py files in the current directory as test scripts.
-e[exe]optional specification of the host executable to launch for testing embedded code on the host computer. If exe is not specified a remote (embedded) target is assumed. The special value debug or DEBUG means that qutest.py will start in the debug mode, in which it will NOT launch the host executable and it will wait for the Target reset and other responses from the Target.
-q[host][:ud_port][:tcp_port]optional host-name/IP-address:udp_port:tcp_port for the machine running the QSpy utility. If not specified, the default is localhost:7701:6601.
-l[log]Optional directory for output of the log file (directory will be created if it does not exist yet).
-o[opt]options for the QUTest script runner to perform the following actions:
x : exit on first failed test
i : interactive mode
c : clear the QSPY screen just after attaching
o : save the QSPY text output to a file (in the QSPY launch directory)
b : save the QSPY binary output to a file (in the QSPY launch directory)
Note
For reliable operation it is recommended to apply the short options without a space between the option and the parameter (e.g., -q192.168.1.100, -oci).

Examples

Windows Hosts

[1] python3 %QTOOLS%\qutest\qutest.py
[2] python3 %QTOOLS%\qutest\qutest.py -- test_mpu.py
[3] python3 %QTOOLS%\qutest\qutest.py -ebuild/test_dpp.exe
[4] python3 %QTOOLS%\qutest\qutest.py -ebuild/test_dpp.exe -q192.168.1.100 -l../log -oco
[5] qutest -qlocalhost:7702 -oxc --  test_qk.py,test_mpu.py
[6] python3 %QTOOLS%\qutest\qutest.py -eDEBUG -- test_mpu.py

[1] runs all test scripts (*.py) in the current directory on a remote target connected to QSPY host utility.

[2] runs the test script test_mpu.py in the current directory on a remote target connected to QSPY host utility.

[3] runs all test scripts (*.py) in the current directory and uses the host executable build/test_dpp.exe (test fixture).

[4] runs all test scripts (*.py) in the current directory, uses the host executable build/test_dpp.exe (test fixture), and connects to QSPY running on a machine with IP address 192.168.1.100. Also produces QUTest log (-l) in the directory ../log. Also clears the QUTest screen before the run (-oc) and causes QSPY to save the text output to a file (-oo)

[5] runs "qutest" (installed with pip) to execute the test scripts test_qk.py,test_mpu.py in the current directory, and connects to QSPY at UDP-host:port localhost:7701.

[6] runs "qutest" in the DEBUG mode to execute the test script test_mpu.py in the current directory.


Linux/macOS Hosts

[1] python3 $(QTOOLS)/qutest/qutest.py
[2] python3 $(QTOOLS)/qutest/qutest.py -- test_mpu.py
[3] python3 $(QTOOLS)/qutest/qutest.py -ebuild/test_dpp
[4] python3 $(QTOOLS)/qutest/qutest.py -ebuild/test_dpp -q192.168.1.100 -l../log -oco
[5] qutest -qlocalhost:7702 -oxc --  test_qk.py,test_mpu.py
[6] python3 $(QTOOLS)/qutest/qutest.py -eDEBUG -- test_mpu.py

[1] runs all test scripts (*.py) in the current directory on a remote target connected to QSPY host utility.

[2] runs the test script test_mpu.py in the current directory on a remote target connected to QSPY host utility.

[3] runs all test scripts (*.py) in the current directory and uses the host executable build/test_dpp (test fixture).

[4] runs all test scripts (*.py) in the current directory, uses the host executable build/test_dpp (test fixture), and connects to QSPY running on a machine with IP address 192.168.1.100. Also produces QUTest log (-l) in the directory ../log. Also clears the QUTest screen before the run (-oc) and causes QSPY to save the text output to a file (-oo)

[5] runs "qutest" (installed with pip) to execute the test scripts test_qk.py,test_mpu.py in the current directory, and connects to QSPY at UDP-host:port localhost:7701.

[6] runs "qutest" in the DEBUG mode to execute the test script test_mpu.py in the current directory.

Interactive Mode

Similar to the Python interpreter itself, the qutest.py test runner has two basic modes: script (default) and interactive. In the default script mode, the test runner executes the test scripts as fast as possible and exits. In the interactive mode, the test runner pauses and executes commands interactively from the terminal input. The interactive mode is activated by providing the -oi command-line option. Interactive mode is a good way to play around and try the system's response to various commands. Also, interactive mode can be used to provide stimuli for debugging the test fixture.

Once activated, the interactive mode works as follows:

  • If any test scripts are provided, after each test group (test script) the test runner enters an interactive session, where the user can manually enter any supported test commands. These commands are executed in the context of the last test in the group. It is also possible to start a new test interactively (by providing the test() command). In that case, the interactively entered commands will be executed in the context of the new test.
  • If no test scripts are provided, the test runner enters an interactive session immediately. In that case, the user must provide the test() command manually (perhaps after defining the test callbacks).

In either case, the interactive mode is shown by the presence of the >>> command prompt. During an interactive session the output produced by the text fixture is displayed after each interactive command entered. The output is not checked against any expectations. In fact, the expect() command itself makes no sense in the interactive mode. Also, all tests that are interactive are considered "SIKPPED" and are not evaluated as "PASS" or "FAIL".

Note
You can use the Up-arrow and Down-arrow keys on your keyboard to navigate through the list of previous commands. You end the interactive session by providing an "empty" command, that is by pressing the <Enter> key right at the >>> command prompt

Here is an example of an interactive session with test scripts provided. (Note the annotations added after the // comments).

C:\qp\qpc\examples\qutest\qhsmtst\test>make OPT=i   // <--- NOTE OPT=i

python3 C:\qp\qtools/qutest/qutest.py -ebuild/test_qhsmtst.exe -q -l -oci --

QUTest unit testing front-end 7.2.1 running on Python 3.10.1
Copyright (c) 2005-2022 Quantum Leaps, www.state-machine.com
Attaching to QSpy (localhost:7701) ... OK

Run ID    : 230111_174610
Target    : build/test_qhsmtst.exe

==================================[Group  1]==================================
test_qhsm-funct.py
[ 1]--------------------------------------------------------------------------
QHsmTst_init
                                                             [ PASS (  0.1s) ]
[ 2]^-------------------------------------------------------------------------
QHsmTst dispatch
>>> dispatch('A_SIG')      // <--- interactive command
0000000127 BSP_DISPLAY s1-A;
0000000128 BSP_DISPLAY s11-EXIT;
0000000129 BSP_DISPLAY s1-EXIT;
0000000130 BSP_DISPLAY s1-ENTRY;
0000000131 BSP_DISPLAY s1-INIT;
0000000132 BSP_DISPLAY s11-ENTRY;
0000000133 Trg-Done QS_RX_EVENT
>>> dispatch('B_SIG')      // <--- interactive command
0000000134 BSP_DISPLAY s1-B;
0000000135 BSP_DISPLAY s11-EXIT;
0000000136 BSP_DISPLAY s11-ENTRY;
0000000137 Trg-Done QS_RX_EVENT
>>>                        // <--- end of interactive session
                          interactive test shows as "skipped" ---> [ SKIPPED ]

==================================[Group  2]==================================
test_qhsm-struct.py
[ 3]--------------------------------------------------------------------------
QHsmTst init
                                                             [ PASS (  0.1s) ]
[ 4]^-------------------------------------------------------------------------
QHsmTst dispatch
>>>                        // <--- end of interactive session
                                                                   [ SKIPPED ]

==================================[ SUMMARY ]=================================

Target ID : 230111_170758 (QP-Ver=721)
Log file  :
Groups    : 2
Tests     : 4
Skipped   : 2 [ 2 4 ]
Failed    : 0

==============================[  OK  ( 20.1s) ]===============================

Generating Test Logs

As required for safety certification, the qutest.py test runner can generate permanent records of the runs by producing log files. This feature is enabled by the -l<log-dir> command-line option.

Remarks
The various make-files supplied in QP/C and QP/C++ allow you to supply the command-line options for saving QUTest logs (by defining the LOG= symbol while invoking make), for example: 
[1] make LOG=.
[2] make LOG=../log
[3] make LOG=c:/cert/logs
[1] generates QUTest log file in the current directory (.)

[2] generates QUTest log file in the ../log directory (relative to the current directory)

[3] generates QUTest log file in the absolute directory c:/cert/logs

The following following listing shows the generated log file:

The following screen shot shows the qutest.py screen output in which a log file has been produced (with the -l. command-line option):

And the following listing shows the generated log file:

Run ID    : 221221_161550
Target    : build/test_qutest.exe

===================================[group]====================================
test_assert.py

This test group contains tests that intenionally FAIL,
to exercise failure modes of the QUTest system.

[ 1]--------------------------------------------------------------------------
Expected assertion
                                                             [ PASS (  0.1s) ]
[ 2]--------------------------------------------------------------------------
Unexpected assertion (should FAIL!)
  @test_assert.py:22
exp: "0000000002 COMMAND CMD_A 0"
got: "0000000002 =ASSERT= Mod=test_qutest,Loc=100"
! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ![ FAIL (  0.2s) ]
[ 3]--------------------------------------------------------------------------
Simple passing test
                                                             [ PASS (  0.1s) ]
[ 4]--------------------------------------------------------------------------
Wrong assertion expectation (should FAIL!)
  @test_assert.py:32
exp: "0000000002 =ASSERT= Mod=test_qutest,Loc=200"
got: "0000000002 =ASSERT= Mod=test_qutest,Loc=100"
! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ![ FAIL (  1.1s) ]
[ 5]--------------------------------------------------------------------------
Simple passing test
                                                             [ PASS (  0.1s) ]

=================================[ SUMMARY ]==================================

Target ID : 221221_161031 (QP-Ver=720)
Log file  : ./qutest221221_161550.txt
Groups    : 1
Tests     : 5
Skipped   : 0
FAILED    : 2 [ 2 4 ]

==============================[ FAIL (  2.7s) ]===============================

Generating QSPY Logs

Additionally, it is also possible to generate logs files from the QSPY output. This feature is enabled by the -oo and -ob command-line options.

  • the option -oo saves the QSPY text output generated during the qutest.py run. QSPY saves the output in the directory from which it was launched.
  • the option -ob saves the QSPY raw binary output received from the test fixture during the qutest.py run. QSPY saves the output in the directory from which it was launched.
Remarks
The various make-files supplied in QP/C and QP/C++ allow you to supply the command-line options for saving QSPY logs (by defining the OPT= symbol while invoking make), for example: 
[1] make OPT=o
[2] make OPT=b
[3] make OPT=ob
[1] generates QSPY text output (in the QSPY launch directory)
[2] generates QSPY binary output (in the QSPY launch directory)
[3] generates both QSPY text and binary outputs (in the QSPY launch directory)

QUTest™ Unit Testing HarnessQUTest Concepts