toolbox/test/system
Ondřej Míchal 8f6deadaef test/system: Drop extra pull when caching images
In 54a2ca1 image caching has been done by first pulling using Podman and
then moving the image from the local container store to a directory. The
pull to the local container store can be skipped and instead we can use
Skopeo to directly save the pulled image into a directory.

On my machine this reduced the time of the system test setup "test" by
about 50 seconds. This speed-up largely depends on the available network
connection, though.
2022-09-21 14:49:21 +02:00
..
libs test/system: Drop extra pull when caching images 2022-09-21 14:49:21 +02:00
000-setup.bats cmd/create: Add option --authfile 2022-03-20 18:08:42 +02:00
001-version.bats test/system: Add function for setting up environment 2021-12-01 01:09:34 +02:00
002-help.bats cmd, test/system: Make the behaviour of 'toolbox' conditional 2021-12-10 01:33:24 +01:00
101-create.bats cmd/create: Improve the error messages for mutually exclusive options 2022-09-07 19:20:17 +02:00
102-list.bats test/system: Add function for setting up environment 2021-12-01 01:09:34 +02:00
103-container.bats test/system: Adjust the return value of container start test 2022-03-20 17:11:33 +02:00
104-run.bats cmd/utils, pkg/utils: Improve the error messages for the distro option 2022-09-02 14:52:44 +02:00
105-enter.bats cmd/utils, pkg/utils: Improve the error messages for the distro option 2022-09-02 14:52:44 +02:00
106-rm.bats test/system: Add function for setting up environment 2021-12-01 01:09:34 +02:00
107-rmi.bats test/system: Add function for setting up environment 2021-12-01 01:09:34 +02:00
999-teardown.bats cmd/create: Add option --authfile 2022-03-20 18:08:42 +02:00
README.md cmd/create: Add option --authfile 2022-03-20 18:08:42 +02:00

System tests

These tests are built with BATS (Bash Automated Testing System).

The tests are meant to ensure that Toolbox's functionality remains stable throughout updates of both Toolbox and Podman/libpod.

The tests are set up in a way that does not affect the host environment. Running them won't remove any existing containers or images.

Dependencies

  • awk
  • bats
  • GNU coreutils
  • httpd-tools
  • openssl
  • podman
  • skopeo
  • toolbox

These tests use a few standard libraries for bats which help with clarity and consistency. The libraries are bats-support and bats-assert. These libraries are provided as git submodules in the libs directory. Make sure both are present.

How to run the tests

First, make sure you have all the dependencies installed.

  • Enter the toolbox root folder
  • Invoke command bats ./test/system/ and the test suite should fire up

Mocking of images is done automatically to prevent potential networking issues and to speed up the cases.

By default the test suite uses the system versions of podman, skopeo and toolbox.

If you have a podman, skopeo or toolbox installed in a nonstandard location then you can use the PODMAN, SKOPEO and TOOLBOX environmental variables to set the path to the binaries. So the command to invoke the test suite could look something like this: PODMAN=/usr/libexec/podman TOOLBOX=./toolbox bats ./test/system/.

When running the tests, make sure the test suite: [job] jobs are successful. These jobs set up the whole environment and are a strict requirement for other jobs to run correctly.

Writing tests

Environmental variables

  • Inspect top part of libs/helpers.bats for a list of helper environmental variables

Naming convention

  • All tests should follow the nomenclature: [command]: <test description>...
  • When the test is expected to fail, start the test description with "Try to..."
  • When the test is to give a non obvious output, it should be put in parenthesis at the end of the title

Examples:

  • @test "create: Create the default container"
  • @test "rm: Try to remove a non-existent container"

Test case environment

  • All the tests start with a clean system (no images or containers) to make sure that there are no dependencies between tests and they are really isolated. Use the setup() and teardown() functions for that purpose.

Image registry

  • The system tests set up an OCI image registry for testing purposes - localhost:50000. The registry requires authentication. There is one account present: user (password: user)

  • The registry contains by default only one image: fedora-toolbox:32

Example pull of the fedora-toolbox:32 image:

$PODMAN login --username user --password user "$DOCKER_REG_URI"
$PODMAN pull "$DOCKER_REG_URI/fedora-toolbox:32"