wbrawner/toolbox
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
toolbox/test/system
History
Debarshi Ray 61e2c970f8 test/system: Make it easier to spot failures to download & cache images 
Currently, if 'skopeo copy ...' fails to download and cache an OCI image
during setup_suite(), the test suite doesn't immediately fail, but
continues.  It only fails later when trying to set up the Docker
registry and contains a lot of noise:
  not ok 1 setup_suite
  # (from function `assert_success' in file
       test/system/libs/bats-assert/src/assert.bash, line 114,
  #  from function `_setup_docker_registry' in file
       test/system/libs/helpers.bash, line 211,
  #  from function `setup_suite' in test file
       test/system/setup_suite.bash, line 59)
  #   `_setup_docker_registry' failed
  # Failed to cache image registry.fedoraproject.org/fedora-toolbox:38
      to /tmp/bats-run-GyTP7A/image-cache/fedora-toolbox-38
  #
  # -- command failed --
  # status : 1
  # output : time="2023-09-25T12:19:52+02:00" level=fatal
      msg="initializing source
        docker://registry.fedoraproject.org/fedora-toolbox:38-foo:
        reading manifest 38-foo in
        registry.fedoraproject.org/fedora-toolbox: manifest unknown"
  # --
  #
  # Failed to cache image quay.io/toolbx/arch-toolbox:latest to
      /tmp/bats-run-GyTP7A/image-cache/arch-toolbox-latest
  #
  # -- command failed --
  # status : 1
  # output : time="2023-09-25T12:20:48+02:00" level=fatal
      msg="initializing source
        docker://quay.io/toolbx/arch-toolbox:latest-foo: reading
        manifest latest-foo in quay.io/toolbx/arch-toolbox: manifest
        unknown"
  # --
  #
  # Failed to cache image registry.fedoraproject.org/fedora-toolbox:34
      to /tmp/bats-run-GyTP7A/image-cache/fedora-toolbox-34
  #
  # -- command failed --
  # status : 1
  # output : time="2023-09-25T12:21:42+02:00" level=fatal
      msg="initializing source
        docker://registry.fedoraproject.org/fedora-toolbox:34-foo:
        reading manifest 34-foo in
        registry.fedoraproject.org/fedora-toolbox: manifest unknown"
  # --
  #
  # ...
  #
  # -- command failed --
  # status : 1
  # output : time="2023-09-25T12:26:33+02:00" level=fatal
      msg="determining manifest MIME type for
        dir:/tmp/bats-run-GyTP7A/image-cache/fedora-toolbox-34: open
        /tmp/bats-run-GyTP7A/image-cache/fedora-toolbox-34/manifest.json:
        no such file or directory"
  # --
  #
  # docker-registry
  # 27fa141e291e64e4c7a148c88ddab219ff2bfb5802a2982dc4188dc11f41692d
  # Untagged: quay.io/toolbox_tests/registry:latest
  # Deleted: fea5a12cde107bb407bc44ede6dd9edea1d2b4171cd8e52b0cb330bf45e517e1

It makes it look as if the root cause of the failure is related to
setting up the Docker registry, which it isn't, and all that noise makes
it difficult to spot the actual problem.

Instead, from now on, it will be more obvious:
  not ok 1 setup_suite
  # (from function `setup_suite' in test file
       test/system/setup_suite.bash, line 44)
  #   `_pull_and_cache_distro_image "$system_id" "$system_version" ||
         false' failed
  # Failed to cache image registry.fedoraproject.org/fedora-toolbox:38
      to /tmp/bats-run-62b8CU/image-cache/fedora-toolbox-38
  # time="2023-09-25T13:55:42+02:00" level=fatal msg="initializing
      source docker://registry.fedoraproject.org/fedora-toolbox:38-foo:
      reading manifest 38-foo in
      registry.fedoraproject.org/fedora-toolbox: manifest unknown"

Note that Bats' 'run' helper [1] isn't designed to work inside
setup_suite().  eg., 'run --separate-stderr' doesn't work because
BATS_TEST_TMPDIR isn't defined.

[1] https://bats-core.readthedocs.io/en/stable/writing-tests.html

https://github.com/containers/toolbox/pull/1377
2023-09-25 16:02:25 +02:00
..
libs test/system: Make it easier to spot failures to download & cache images 2023-09-25 16:02:25 +02:00
001-version.bats test/system: Silence SC2154 2023-09-20 01:33:29 +02:00
002-help.bats test/system: Silence SC2086 2023-09-19 14:29:17 +02:00
101-create.bats test/system: Silence SC2155 2023-09-20 10:24:29 +02:00
102-list.bats test/system: Silence SC2154 2023-09-20 01:33:29 +02:00
103-container.bats test/system: Silence SC2155 2023-09-20 10:24:29 +02:00
104-run.bats test/system: Silence SC2155 2023-09-20 10:24:29 +02:00
105-enter.bats test/system: Silence SC2155 2023-09-20 10:24:29 +02:00
106-rm.bats test/system: Silence SC2086 2023-09-19 14:29:17 +02:00
107-rmi.bats test/system: Silence SC2154 2023-09-20 01:33:29 +02:00
108-completion.bats test/system: Silence SC2154 2023-09-20 01:33:29 +02:00
201-ipc.bats test/system: Unbreak the line count checks with Bats >= 1.10.0 2023-08-18 06:21:40 +02:00
203-network.bats test/system: Unbreak the line count checks with Bats >= 1.10.0 2023-08-18 06:21:40 +02:00
206-user.bats test/system: Test that group and user IDs work 2023-08-22 16:01:08 +02:00
210-ulimit.bats test/system: Unbreak the line count checks with Bats >= 1.10.0 2023-08-18 06:21:40 +02:00
211-dbus.bats test/system: Test that D-Bus works 2023-08-22 17:11:59 +02:00
meson.build test/system: Test that D-Bus works 2023-08-22 17:11:59 +02:00
README.md test/system: Clarify the use of Git submodules 2023-06-21 12:34:08 +02:00
setup_suite.bash test/system: Silence SC2155 2023-09-20 10:24:29 +02:00

README.md

System tests

These tests are built with BATS (Bash Automated Testing System).

The tests are meant to ensure that Toolbox's functionality remains stable throughout updates of both Toolbox and Podman/libpod.

The tests are set up in a way that does not affect the host environment. Running them won't remove any existing containers or images.

Dependencies

  • awk
  • bats
  • GNU coreutils
  • httpd-tools
  • openssl
  • podman
  • skopeo
  • toolbox

These tests use a few standard libraries for bats which help with clarity and consistency. The libraries are bats-support and bats-assert. These libraries are provided as git submodules in the libs directory. Make sure both are present by running git submodule init and git submodule update.

How to run the tests

First, make sure you have all the dependencies installed.

  • Enter the toolbox root folder
  • Invoke command bats ./test/system/ and the test suite should fire up

Mocking of images is done automatically to prevent potential networking issues and to speed up the cases.

By default the test suite uses the system versions of podman, skopeo and toolbox.

If you have a podman, skopeo or toolbox installed in a nonstandard location then you can use the PODMAN, SKOPEO and TOOLBOX environmental variables to set the path to the binaries. So the command to invoke the test suite could look something like this: PODMAN=/usr/libexec/podman TOOLBOX=./toolbox bats ./test/system/.

When running the tests, make sure the test suite: [job] jobs are successful. These jobs set up the whole environment and are a strict requirement for other jobs to run correctly.

Writing tests

Environmental variables

  • Inspect top part of libs/helpers.bats for a list of helper environmental variables

Naming convention

  • All tests should follow the nomenclature: [command]: <test description>...
  • When the test is expected to fail, start the test description with "Try to..."
  • When the test is to give a non obvious output, it should be put in parenthesis at the end of the title

Examples:

  • @test "create: Create the default container"
  • @test "rm: Try to remove a non-existent container"

Test case environment

  • All the tests start with a clean system (no images or containers) to make sure that there are no dependencies between tests and they are really isolated. Use the setup() and teardown() functions for that purpose.

Image registry

  • The system tests set up an OCI image registry for testing purposes - localhost:50000. The registry requires authentication. There is one account present: user (password: user)

  • The registry contains by default only one image: fedora-toolbox:34

Example pull of the fedora-toolbox:34 image:

$PODMAN login --username user --password user "$DOCKER_REG_URI"
$PODMAN pull "$DOCKER_REG_URI/fedora-toolbox:34"