7ab2f9b866
Fedora 32 reached End of Life on 25th May 2021: https://docs.fedoraproject.org/en-US/releases/eol/ That's quite old because right now Fedora 35 is nearing its End of Life. Since the tests are intended for Toolbx, not the Fedora infrastructure, it will be better to use a newer image, because images that are too old can get lost from registry.fedoraproject.org. The fedora-toolbox:34 image can be a drop-in replacement for the fedora-toolbox:32 image for the purposes of this test suite, and has the advantage of being newer. Note that fedora-toolbox:34 is also old enough to test that the toolbox binary runs against it's build-time ABI from the host, and not the Toolbx container's ABI, when it's invoked as the entry point of the container [1,2]. This is important because the subsequent commit will add a test to ensure that. [1] Commit6063eb27b9https://github.com/containers/toolbox/issues/821 [2] Commit6ad9c63180https://github.com/containers/toolbox/issues/529 https://github.com/containers/toolbox/pull/1187
88 lines
2.9 KiB
Markdown
88 lines
2.9 KiB
Markdown
# System tests
|
|
|
|
These tests are built with BATS (Bash Automated Testing System).
|
|
|
|
The tests are meant to ensure that Toolbox's functionality remains stable
|
|
throughout updates of both Toolbox and Podman/libpod.
|
|
|
|
The tests are set up in a way that does not affect the host environment.
|
|
Running them won't remove any existing containers or images.
|
|
|
|
## Dependencies
|
|
|
|
- `awk`
|
|
- `bats`
|
|
- `GNU coreutils`
|
|
- `httpd-tools`
|
|
- `openssl`
|
|
- `podman`
|
|
- `skopeo`
|
|
- `toolbox`
|
|
|
|
These tests use a few standard libraries for `bats` which help with clarity
|
|
and consistency. The libraries are [bats-support](https://github.com/bats-core/bats-support)
|
|
and [bats-assert](https://github.com/bats-core/bats-assert). These libraries are
|
|
provided as git submodules in the `libs` directory. Make sure both are present.
|
|
|
|
## How to run the tests
|
|
|
|
First, make sure you have all the dependencies installed.
|
|
|
|
- Enter the toolbox root folder
|
|
- Invoke command `bats ./test/system/` and the test suite should fire up
|
|
|
|
Mocking of images is done automatically to prevent potential networking issues
|
|
and to speed up the cases.
|
|
|
|
By default the test suite uses the system versions of `podman`, `skopeo` and
|
|
`toolbox`.
|
|
|
|
If you have a `podman`, `skopeo` or `toolbox` installed in a nonstandard
|
|
location then you can use the `PODMAN`, `SKOPEO` and `TOOLBOX` environmental
|
|
variables to set the path to the binaries. So the command to invoke the test
|
|
suite could look something like this: `PODMAN=/usr/libexec/podman TOOLBOX=./toolbox bats ./test/system/`.
|
|
|
|
When running the tests, make sure the `test suite: [job]` jobs are successful.
|
|
These jobs set up the whole environment and are a strict requirement for other
|
|
jobs to run correctly.
|
|
|
|
## Writing tests
|
|
|
|
### Environmental variables
|
|
|
|
- Inspect top part of `libs/helpers.bats` for a list of helper environmental
|
|
variables
|
|
|
|
### Naming convention
|
|
|
|
- All tests should follow the nomenclature: `[command]: <test description>...`
|
|
- When the test is expected to fail, start the test description with "Try
|
|
to..."
|
|
- When the test is to give a non obvious output, it should be put in parenthesis
|
|
at the end of the title
|
|
|
|
Examples:
|
|
|
|
* `@test "create: Create the default container"`
|
|
* `@test "rm: Try to remove a non-existent container"`
|
|
|
|
### Test case environment
|
|
|
|
- All the tests start with a clean system (no images or containers) to make sure
|
|
that there are no dependencies between tests and they are really isolated. Use
|
|
the `setup()` and `teardown()` functions for that purpose.
|
|
|
|
### Image registry
|
|
|
|
- The system tests set up an OCI image registry for testing purposes -
|
|
`localhost:50000`. The registry requires authentication. There is one account
|
|
present: `user` (password: `user`)
|
|
|
|
- The registry contains by default only one image: `fedora-toolbox:34`
|
|
|
|
Example pull of the `fedora-toolbox:34` image:
|
|
|
|
```bash
|
|
$PODMAN login --username user --password user "$DOCKER_REG_URI"
|
|
$PODMAN pull "$DOCKER_REG_URI/fedora-toolbox:34"
|
|
```
|