History

Debarshi Ray e8512828c1 cmd/list, test/system: Ignore the problem of UBI not being listed Not having the corresponding image for UBI toolbox containers show up in 'toolbox list' is a rough edge. However, the whole UBI feature is a bit experimental. It's about a gratis RHEL environment getting created in a jiffy on any host, which is something that hasn't been done before, and those containers also suffer from various shortcomings because of the limited package set of UBI. So it's not that big of a problem if it takes a release or two to hammer out the details. Especially since it's likely that there will be a special Toolbox-specific image that's created out of the UBI RPM repositories, which will likely have the com.github.containers.toolbox label. There's also the issue that 0.1.0 needs to be finished, and for that the the churn needs to be kept down. Changing the labels can very likely lead to compatibility issues in the future, because of which it either can't be removed for a while or the wrong images start to get listed. Some of the older labels have finally been removed, so it's better not to add more to the list. In short, this problem will likely fix itself in the coming months, so it's wise not to create complications trying to rush through a fix. This reverts commits `1df36591d0` and `e09de9f3e5`. https://github.com/containers/toolbox/issues/753		2021-06-19 01:12:08 +02:00
..
libs	cmd/list, test/system: Ignore the problem of UBI not being listed	2021-06-19 01:12:08 +02:00
000-setup.bats	cmd/list, test/system: Ignore the problem of UBI not being listed	2021-06-19 01:12:08 +02:00
001-version.bats	test/system: Use env var for invoking Toolbox	2021-05-26 22:52:40 +02:00
002-help.bats	test/system: Use env var for invoking Toolbox	2021-05-26 22:52:40 +02:00
101-create.bats	test/system: Decouple image caching from Zuul	2021-06-01 00:41:20 +02:00
102-list.bats	cmd/list, test/system: Ignore the problem of UBI not being listed	2021-06-19 01:12:08 +02:00
103-run.bats	test/system: Use env var for invoking Toolbox	2021-05-26 22:52:40 +02:00
104-rm.bats	test/system: Use env var for invoking Toolbox	2021-05-26 22:52:40 +02:00
105-rmi.bats	test/system: Use env var for invoking Toolbox	2021-05-26 22:52:40 +02:00
999-teardown.bats	test/system: Decouple image caching from Zuul	2021-06-01 00:41:20 +02:00
README.md	test/system: Update documentation	2021-02-19 20:37:29 +01:00

README.md

System tests

These tests are built with BATS (Bash Automated Testing System).

The tests are meant to ensure that Toolbox's functionality remains stable throughout updates of both Toolbox and Podman/libpod.

Warning: The tests are not executed in an isolated environment. Running the tests will clear all podman state (delete all containers, images, etc).

Dependencies

bats
podman
skopeo
toolbox

These tests use a few standard libraries for bats which help with clarity and consistency. In order to use it you need to download them to the libs directory:

# Go to the Toolbox root folder
$ git clone https://github.com/bats-core/bats-assert test/system/libs/bats-assert
$ git clone https://github.com/bats-core/bats-support test/system/libs/bats-support

Convention

All tests should follow the nomenclature: [command]: <test description>...
When the test is expected to fail, start the test description with "Try to..."
When the test is to give a non obvious output, it should be put in parenthesis at the end of the title

Examples:

@test "create: Create the default container"
@test "rm: Try to remove a non-existent container"

All the tests start with a clean system (no images or containers) to make sure that there are no dependencies between tests and they are really isolated. Use the setup() and teardown() functions for that purpose.

How to run the tests

First, make sure you have all the dependencies installed.

Enter the toolbox root folder
Prepare container images. See playbooks/setup-env.yaml
Invoke command bats ./test/system/ and the test suite should fire up

Mocking of images is done to prevent potential networking issues and to speed up the cases.

Currently, this is done in a playbook that is a part of Toolbox's CI. In the future, this should be converted and become part of the test suite itself to allow local execution.

By default the test suite uses the system versions of podman and toolbox.

If you have a podman or toolbox installed in a nonstandard location then you can use the PODMAN and TOOLBOX environmental variables to set the path to the binaries. So the command to invoke the test suite could look something like this: PODMAN=/usr/libexec/podman TOOLBOX=./toolbox bats ./test/system/.