History

Juanje Ojeda b27795a03e test/system: Refactor tests using bats-support and bats-assert The bats-support[0] and bats-assert[1] libraries extend the capabilities of bats[2]. Mainly, bats-assert is very useful for clean checking of values/outputs/return codes. Apart from updating the cases to use the libraries, the test cases have been restructured in a way that they don't depend on each other anymore. This required major changes in the helpers.bats file. Overall, the tests are cleaner to read and easier to extend due to the test cases being independent. Some slight changes were made to the test cases themselves. Should not alter their final behaviour. There will be a follow up commit that will take care of downloading of the tested images locally and caching them using Skopeo to speedup the tests and try to resolve network problems when pulling the images that we experienced in the past. [0] https://github.com/bats-core/bats-support [1] https://github.com/bats-core/bats-assert [2] https://github.com/bats-core/bats-core		2021-02-12 14:02:08 +01:00
..
libs	test/system: Refactor tests using bats-support and bats-assert	2021-02-12 14:02:08 +01:00
001-version.bats	test/system: Refactor tests using bats-support and bats-assert	2021-02-12 14:02:08 +01:00
002-help.bats	test/system: Refactor tests using bats-support and bats-assert	2021-02-12 14:02:08 +01:00
101-create.bats	test/system: Refactor tests using bats-support and bats-assert	2021-02-12 14:02:08 +01:00
102-list.bats	test/system: Refactor tests using bats-support and bats-assert	2021-02-12 14:02:08 +01:00
103-run.bats	test/system: Refactor tests using bats-support and bats-assert	2021-02-12 14:02:08 +01:00
104-rm.bats	test/system: Refactor tests using bats-support and bats-assert	2021-02-12 14:02:08 +01:00
105-rmi.bats	test/system: Refactor tests using bats-support and bats-assert	2021-02-12 14:02:08 +01:00
README.md	test/system: Refactor tests using bats-support and bats-assert	2021-02-12 14:02:08 +01:00

README.md

System tests

These tests are built with BATS (Bash Automated Testing System).

The tests are meant to ensure that Toolbox's functionality remains stable throughout updates of both Toolbox and Podman/libpod.

Warning: The tests are not executed in an isolated environment. They affect the system where they are run.

Dependencies

These tests use a few standard libraries for BATS which help with clarity and consistency. In order to use it you need to download them to the libs directory:

# Go to the Toolbox root folder
$ git clone https://github.com/ztombol/bats-assert test/system/libs/bats-assert
$ git clone https://github.com/ztombol/bats-support test/system/libs/bats-support

Convention

All tests should follow the nomenclature: [command]: <test description>...
When the test is expected to fail, start the test description with "Try to..."
When the test is to give a non obvious output, it should be put in parenthesis at the end of the title

Examples:

@test "create: Create the default container"
@test "rm: Try to remove a non-existent container"

All the tests start with a clean system (no images or containers) to make sure that there are no dependencies between tests and they are really isolated. Use the setup() and teardown() functions for that purpose.

How to run the tests

Make sure you have bats, podman and toolbox installed on your system. And have the test dependencies prepared.

Enter the toolbox root folder
Invoke command bats ./test/system/ and the test suite should fire up

By default the test suite uses the system versions of podman and toolbox.

If you have a podman or toolbox installed in a nonstandard location then you can use the PODMAN and TOOLBOX environmental variables to set the path to the binaries. So the command to invoke the test suite could look something like this: PODMAN=/usr/libexec/podman TOOLBOX=./toolbox bats ./test/system/.