Integration Testing using MAVSDK

PX4 can be tested end to end to using integration tests based on MAVSDK.

The tests are primarily developed against SITL and run in continuous integration (CI). In future we plan to generalise them for any platform/hardware.

The instructions below explain how to setup and run the tests locally.

Prerequisites

Setup Developer Environment

If you haven't done so already:

• Install the development toolchain for Linux or macOS (Windows not supported). Gazebo Classic is required, and should be installed by default.

• Get the PX4 source code:

  git clone https://github.com/PX4/PX4-Autopilot.git --recursive
  cd PX4-Autopilot

Build PX4 for Testing

To build PX4 source code for simulator testing, use:

DONT_RUN=1 make px4_sitl gazebo-classic mavsdk_tests

Install the MAVSDK C++ Library

The tests need the MAVSDK C++ library installed system-wide (e.g. in /usr/lib or /usr/local/lib).

Install either from binaries or source:

• MAVSDK > C++ > C++ QuickStart: Install as a prebuilt library on supported platforms (recommended)
• MAVSDK > C++ Guide > Building from Source: Build C++ library from source.

Run All PX4 Tests

To run all SITL tests as defined in sitl.json, do:

test/mavsdk_tests/mavsdk_test_runner.py test/mavsdk_tests/configs/sitl.json --speed-factor 10

This will list all of the tests and then run them sequentially.

To see all possible command line arguments use the -h argument:

test/mavsdk_tests/mavsdk_test_runner.py -h

usage: mavsdk_test_runner.py [-h] [--log-dir LOG_DIR] [--speed-factor SPEED_FACTOR] [--iterations ITERATIONS] [--abort-early] [--gui] [--model MODEL]
                             [--case CASE] [--debugger DEBUGGER] [--verbose]
                             config_file

positional arguments:
  config_file           JSON config file to use

optional arguments:
  -h, --help            show this help message and exit
  --log-dir LOG_DIR     Directory for log files
  --speed-factor SPEED_FACTOR
                        how fast to run the simulation
  --iterations ITERATIONS
                        how often to run all tests
  --abort-early         abort on first unsuccessful test
  --gui                 display the visualization for a simulation
  --model MODEL         only run tests for one model
  --case CASE           only run tests for one case
  --debugger DEBUGGER   choice from valgrind, callgrind, gdb, lldb
  --verbose             enable more verbose output

Run a Single Test

Run a single test by specifying the model and test case as command line options. For example, to test flying a tailsitter in a mission you might run:

test/mavsdk_tests/mavsdk_test_runner.py test/mavsdk_tests/configs/sitl.json --speed-factor 10 --model tailsitter --case 'Fly VTOL mission'

The easiest way to find out the current set of models and their associated test cases is to run all PX4 tests as shown above (note, you can then cancel the build if you wish to test just one).

At time of writing the list generated by running all tests is:

About to run 39 test cases for 3 selected models (1 iteration):
  - iris:
    - 'Land on GPS lost during mission (baro height mode)'
    - 'Land on GPS lost during mission (GPS height mode)'
    - 'Continue on mag lost during mission'
    - 'Continue on baro lost during mission (baro height mode)'
    - 'Continue on baro lost during mission (GPS height mode)'
    - 'Continue on baro stuck during mission (baro height mode)'
    - 'Continue on baro stuck during mission (GPS height mode)'
    - 'Takeoff and Land'
    - 'Fly square Multicopter Missions including RTL'
    - 'Fly square Multicopter Missions with manual RTL'
    - 'Fly straight Multicopter Mission'
    - 'Offboard takeoff and land'
    - 'Offboard position control'
    - 'Fly forward in position control'
    - 'Fly forward in altitude control'
  - standard_vtol:
    - 'Land on GPS lost during mission (baro height mode)'
    - 'Land on GPS lost during mission (GPS height mode)'
    - 'Continue on mag lost during mission'
    - 'Continue on baro lost during mission (baro height mode)'
    - 'Continue on baro lost during mission (GPS height mode)'
    - 'Continue on baro stuck during mission (baro height mode)'
    - 'Continue on baro stuck during mission (GPS height mode)'
    - 'Takeoff and Land'
    - 'Fly square Multicopter Missions including RTL'
    - 'Fly square Multicopter Missions with manual RTL'
    - 'Fly forward in position control'
    - 'Fly forward in altitude control'
  - tailsitter:
    - 'Land on GPS lost during mission (baro height mode)'
    - 'Land on GPS lost during mission (GPS height mode)'
    - 'Continue on mag lost during mission'
    - 'Continue on baro lost during mission (baro height mode)'
    - 'Continue on baro lost during mission (GPS height mode)'
    - 'Continue on baro stuck during mission (baro height mode)'
    - 'Continue on baro stuck during mission (GPS height mode)'
    - 'Takeoff and Land'
    - 'Fly square Multicopter Missions including RTL'
    - 'Fly square Multicopter Missions with manual RTL'
    - 'Fly forward in position control'
    - 'Fly forward in altitude control'

Implementation Notes

• The tests are invoked from the test runner script mavsdk_test_runner.py, which is written in Python.

  In addition to MAVSDK, this runner starts px4 as well as Gazebo for SITL tests, and collects the logs of these processes.

• The test runner is a C++ binary that contains:

  • The main function to parse the arguments.
  • An abstraction around MAVSDK called autopilot_tester.
  • The actual tests using the abstraction around MAVSDK as e.g. test_multicopter_mission.cpp.
  • The tests use the catch2 unit testing framework. The reasons for using this framework are:
    • Asserts (REQUIRE) which are needed to abort a test can be inside of functions (and not just in the top level test as is the case with gtest).
    • Dependency management is easier because catch2 can just be included as a header-only library.
    • Catch2 supports tags, which allows for flexible composition of tests.

Terms used:

• "model": This is the selected Gazebo model, e.g. iris.
• "test case": This is a catch2 test case.