Note that 'run --keep-empty-lines' counts the trailing newline on the
last line as a separate line.
Until Bats 1.7.0, 'run --keep-empty-lines' had a bug where even when a
command produced no output, it would report a line count of one [1] due
to a stray line feed character. This needs to be conditionalized, since
Fedora 35 has Bats 1.5.0.
[1] https://github.com/bats-core/bats-core/issues/573https://github.com/containers/toolbox/pull/1192
This was making it difficult to read the Bats assertions on test
failures, by polluting it with unexpected and irrelevant output from
'podman images'. For example [1]:
not ok 39 list: Images with and without names in 12332ms
# (from function `assert' in file test/system/libs/bats-assert/src/assert.bash, line 46,
# in test file test/system/102-list.bats, line 126)
# `assert [ ${#stderr_lines[@]} -eq 0 ]' failed
# REPOSITORY TAG IMAGE ID CREATED SIZE
# registry.fedoraproject.org/fedora-toolbox 35 862705390e8b 4 weeks ago 332 MB
# REPOSITORY TAG IMAGE ID CREATED SIZE
# registry.fedoraproject.org/fedora-toolbox 35 862705390e8b 4 weeks ago 332 MB
# registry.fedoraproject.org/fedora-toolbox 34 70cbe2ce60ca 7 months ago 354 MB
#
# -- assertion failed --
# expression : [ 1 -eq 0 ]
# --
#
Fallout from 7973181136
[1] https://github.com/containers/toolbox/pull/1192https://github.com/containers/toolbox/pull/1193
Fedora 32 reached End of Life on 25th May 2021:
https://docs.fedoraproject.org/en-US/releases/eol/
That's quite old because right now Fedora 35 is nearing its End of Life.
Since the tests are intended for Toolbx, not the Fedora infrastructure,
it will be better to use a newer image, because images that are too old
can get lost from registry.fedoraproject.org. The fedora-toolbox:34
image can be a drop-in replacement for the fedora-toolbox:32 image for
the purposes of this test suite, and has the advantage of being newer.
Note that fedora-toolbox:34 is also old enough to test that the toolbox
binary runs against it's build-time ABI from the host, and not the
Toolbx container's ABI, when it's invoked as the entry point of the
container [1,2]. This is important because the subsequent commit will
add a test to ensure that.
[1] Commit 6063eb27b9https://github.com/containers/toolbox/issues/821
[2] Commit 6ad9c63180https://github.com/containers/toolbox/issues/529https://github.com/containers/toolbox/pull/1187
In 54a2ca1 image caching has been done by first pulling using Podman and
then moving the image from the local container store to a directory. The
pull to the local container store can be skipped and instead we can use
Skopeo to directly save the pulled image into a directory.
On my machine this reduced the time of the system test setup "test" by
about 50 seconds. This speed-up largely depends on the available network
connection, though.
When a command is executed with toolbox run and it returns a non-zero
exit code, it is just ignored if that exit code is not handled. This
prevents users to identify errors when executing commands in toolbox.
With this fix, the exit codes of the invoked command are propagated
and returned by 'toolbox run'. This includes even exit codes returned
by Podman on error.
https://github.com/containers/toolbox/pull/1013
Co-authored-by: Ondřej Míchal <harrymichal@seznam.cz>
Calling 'podman system cleanup' causes problems with containers/images
in a separate Podman root. Despite being stored elsewhere, they are
still under Podman's influence and the cleanup removes them. Also,
running containers (outside the scope of the tests) still got affected
by this call and e.g., lost the ability to follow terminal size changes.
Despite the raised concerns, to ensure proper cleanup of any Podman
state, the reset still needs to be done. Thus, do it only once during
the test suite teardown, moving the potential source of problems to a
single position..
https://github.com/containers/toolbox/pull/1024
We need to know if the latest changes in the libc (that is dynamically
linked to the binary) causes problems in containers based on older
releases of Fedora.
The estimate of the version numbers is very crude and does not follow
the upstream schedule. That should not be a problem, though.
A part of an existing test has been reused and made into a helper
function to implement this.
This increases the run time of the test suite on Rawhide which already
takes longer than the same test suite on released versions of Fedora.
Make up for it by increasing the timeout by 2 minutes.
https://github.com/containers/toolbox/pull/899
Instead of typing out two function names to set up the test environment,
type out only one. We never know if a new set up function will show up.
https://github.com/containers/toolbox/pull/818
This allows to run the test suite without having to worry about blasting
the whole local state of Podman.
This is done by creating a configuration file with a custom path for the
storage of Podman and specifying the config file using an env var.
The used location for the temporary storage is located either under
XDG_CACHE_HOME and if the one is not defined, $HOME/.cache is used
instead. The data are namespaced. This follows the XDG Base Directory
Specification[0]. Other locations could be /tmp or /run but those
locations usually use tmpfs and that filesystem can not be used by
Podman[1] due to missing features in tmpfs.
https://github.com/containers/toolbox/pull/818
[0] https://specifications.freedesktop.org/basedir-spec/latest/index.html
[1] https://github.com/containers/podman/issues/10693#issuecomment-863007516
Due to docker rate limiting we can not rely in docker.io for
retrieving the images.
This was detected when executing our tests for podman fedora
gating pipeline. Our busybox image was not downloaded and
one of the list tests was failing.
Using the current working directory for cache is not a good solution
since the test files may reside in a location that is unwritable (e.g.,
/usr/share). The `BATS_RUN_TMPDIR` variable should point to a location
that is sure to be writeable from the test suite.
https://github.com/containers/toolbox/pull/850
Not having the corresponding image for UBI toolbox containers show up
in 'toolbox list' is a rough edge. However, the whole UBI feature is
a bit experimental. It's about a gratis RHEL environment getting
created in a jiffy on any host, which is something that hasn't been
done before, and those containers also suffer from various shortcomings
because of the limited package set of UBI.
So it's not that big of a problem if it takes a release or two to
hammer out the details. Especially since it's likely that there will
be a special Toolbox-specific image that's created out of the UBI RPM
repositories, which will likely have the com.github.containers.toolbox
label.
There's also the issue that 0.1.0 needs to be finished, and for that
the the churn needs to be kept down. Changing the labels can very
likely lead to compatibility issues in the future, because of which it
either can't be removed for a while or the wrong images start to get
listed. Some of the older labels have finally been removed, so it's
better not to add more to the list.
In short, this problem will likely fix itself in the coming months, so
it's wise not to create complications trying to rush through a fix.
This reverts commits 1df36591d0 and
e09de9f3e5.
https://github.com/containers/toolbox/issues/753
UBI[0] does not have the recommend Toolbox labels used to track whether
an image/container is truly a toolbox image/container. Thankfully, they
have a number of labels to choose from that we can use to identify the
image. The "com.redhat.component=ubi8-container" seems to be ideal.
The approach of using the UBI8 label introduces one problem though. If
we were to use only one set of labels for both images and containers,
containers created with Podman and not Toolbox from UBI8 would also be
marked as toolbox containers. This is not desired and therefore there
are now two sets of labels. Ones for images where the new label has been
added and other for containers that stays the same.
Since the rewrite of the system test suite[0] we've relied on the Zuul
playbooks for taking care of caching images using Skopeo for increasing
the reliability of the tests (in the past the instability of the Fedora
registry caused problems). This state is problematic if we want to use
the tests in other environments than the Zuul CI. This moves the caching
from Zuul into the system tests.
Currently, Bats does not support officially suite-wide setup and
teardown functions. The solution I chose was to add two new test files
that are executed before and after all tests. This may complicate the
execution of cherry-picked tests but that is not a very common use case
anyway.
The tests are now to some extent capable of adjusting to the host
environment. This is meant in the sense of: I'm running on RHEL, the
"default image" is UBI; I'm running on Fedora, the "default image" is
fedora-toolbox. This mechanism relies on os-release, which is the same
as what Toolbox itself uses.
[0] https://github.com/containers/toolbox/pull/517https://github.com/containers/toolbox/pull/774
The fedora-toolbox:32 image is the first of images in the renamed
toolbox image repository[0]. With the change we can drop the
pull_image_old() function because it was kept only for the old image.
Seems like newer version of ShellCheck checks the validity of variable
names (SC2153). This caused a false positive, so I silenced it.
[0] https://github.com/containers/toolbox/pull/615https://github.com/containers/toolbox/pull/780
Since commit b27795a03e, each section of the test suite starts
and ends with a clean Podman state. This includes removing all images
from the local containers storage. Therefore, the images get downloaded
multiple times during the course of the test suite.
This commit restores the earlier behaviour where the images would get
downloaded only once, by copying them to separate directories outside
the local containers storage and then restoring them when the tests
are run.
https://github.com/containers/toolbox/pull/517https://github.com/containers/toolbox/pull/704
The bats-support[0] and bats-assert[1] libraries extend the
capabilities of bats[2]. Mainly, bats-assert is very useful for clean
checking of values/outputs/return codes.
Apart from updating the cases to use the libraries, the test cases have
been restructured in a way that they don't depend on each other anymore.
This required major changes in the helpers.bats file.
Overall, the tests are cleaner to read and easier to extend due to the
test cases being independent.
Some slight changes were made to the test cases themselves. Should not
alter their final behaviour.
There will be a follow up commit that will take care of downloading of
the tested images locally and caching them using Skopeo to speedup the
tests and try to resolve network problems when pulling the images that
we experienced in the past.
[0] https://github.com/bats-core/bats-support
[1] https://github.com/bats-core/bats-assert
[2] https://github.com/bats-core/bats-core
