These files aren't marked as executable, and shouldn't be, because they
aren't meant to be standalone executable scripts. They're meant to be
part of a test suite driven by Bats. Therefore, it doesn't make sense
for them to have shebangs, because it gives the opposite impression.
The shebangs were actually being used by external tools like Coverity to
deduce the shell when running shellcheck(1). Shellcheck's inline
'shell' directive is a more obvious way to achieve that.
https://github.com/containers/toolbox/pull/1363
Bats 1.7.0 emits a warning if a feature that is only available starting
from a certain version of Bats onwards is used without specifying that
version [1]:
BW02: Using flags on `run` requires at least BATS_VERSION=1.5.0. Use
`bats_require_minimum_version 1.5.0` to fix this message.
(from function `bats_warn_minimum_guaranteed_version' in file
/usr/lib/bats-core/warnings.bash, line 32,
from function `run' in file
/usr/lib/bats-core/test_functions.bash, line 227,
in test file test/system/001-version.bats, line 27)
Note that bats_require_minimum_version itself is only available from
Bats 1.7.0 [2]. Hence, even though the specific feature here (using
flags on 'run') only requires Bats >= 1.5.0, in practice Bats >= 1.7.0
is needed. Fortunately, commit e22a82fec8 already added a
dependency on Bats >= 1.7.0. So, there's nothing to worry about.
[1] Bats commit 82002bb6c1a5c418
https://github.com/bats-core/bats-core/issues/556https://bats-core.readthedocs.io/en/stable/warnings/BW02.html
[2] Bats commit 71d6b71cebc3d32b
https://github.com/bats-core/bats-core/issues/556https://bats-core.readthedocs.io/en/stable/warnings/BW02.htmlhttps://github.com/containers/toolbox/pull/1315
Instead of typing out two function names to set up the test environment,
type out only one. We never know if a new set up function will show up.
https://github.com/containers/toolbox/pull/818
The system test refactor[0] replaced the 'run_toolbox' helper function
with 'run toolbox', which is a normal invocation of Toolbox. This makes
it impossible to override Toolbox used during the tests using env var.
[0] https://github.com/containers/toolbox/pull/693
The bats-support[0] and bats-assert[1] libraries extend the
capabilities of bats[2]. Mainly, bats-assert is very useful for clean
checking of values/outputs/return codes.
Apart from updating the cases to use the libraries, the test cases have
been restructured in a way that they don't depend on each other anymore.
This required major changes in the helpers.bats file.
Overall, the tests are cleaner to read and easier to extend due to the
test cases being independent.
Some slight changes were made to the test cases themselves. Should not
alter their final behaviour.
There will be a follow up commit that will take care of downloading of
the tested images locally and caching them using Skopeo to speedup the
tests and try to resolve network problems when pulling the images that
we experienced in the past.
[0] https://github.com/bats-core/bats-support
[1] https://github.com/bats-core/bats-assert
[2] https://github.com/bats-core/bats-core
In the Go implementation, when the 'rm' and 'rmi' commands fail to
remove a container or image, they don't use a non-zero exit code.
There's currently no nice fix for this. So, the tests have been
adjusted as a temporary measure.
https://github.com/containers/toolbox/pull/507
The tests introduced by commit b5cdc57ae3 have proven to be
rather unstable due to mistakes in their design. The tests were quite
chaotically structured, and because of that images were deleted and
pulled too often, causing several false positives [1, 2].
This changes the structure of the tests in a major way. The tests
(resp. commands) are now run in a manner that better simulates the way
Toolbox is actually used. From a clean state, through creating
containers, using them and in the end deleting them. This should
reduce the strain on the bandwidth and possibly even speed up the
tests themselves.
[1] https://github.com/containers/toolbox/pull/372
[2] https://github.com/containers/toolbox/pull/374https://github.com/containers/toolbox/pull/375
2020-02-18 14:00:59 +01:00
Renamed from test/system/001-basics.bats (Browse further)
