The following caveats must be noted:
* Podman sets the Toolbx container's soft limit for the maximum number
of open file descriptors to the host's hard limit, which is often
greater than the host's soft limit [1].
* The ulimit(1) options -P, -T, -b, and -k don't work on Fedora 38
because the corresponding resource arguments for getrlimit(2) are
absent from the operating system. These are RLIMIT_NPTS,
RLIMIT_PTHREAD, RLIMIT_SBSIZE and RLIMIT_KQUEUES respectively.
[1] https://github.com/containers/podman/issues/17681https://github.com/containers/toolbox/issues/213
The current approach of extracting the VERSION_ID field from
os-release(5) assumes that the value is not quoted. There's no
guarantee that this will be the case. It only happens to be so on
Fedora by chance, and is different on Ubuntu:
$ cat /etc/os-release
...
VERSION_ID="22.04"
...
This means that "22.04", including the double quotes, is read as the
value of VERSION_ID on Ubuntu, not 22.04. This is wrong because this
value can't be used as is in image and container names. There's no
image called quay.io/toolbx/ubuntu-toolbox:"22.04" and double quotes are
not allowed in container names.
Instead, use the same approach as profile.d/toolbox.sh and the old POSIX
shell implementation that doesn't rely on the quoting of the
os-release(5) values.
Fallout from b27795a03ehttps://github.com/containers/toolbox/pull/1320
The current approach of selecting all the os-release(5) fields that have
'ID' in their name (eg., ID, VERSION_ID, PLATFORM_ID, VARIANT_ID, etc.)
and then picking the first one, assumes that the ID field will always be
placed above the others in os-release(5). There's no guarantee that
this will be the case. It only happens to be so on Fedora by chance,
and is different on Ubuntu:
$ cat /etc/os-release
...
VERSION_ID="22.04"
...
ID=ubuntu
ID_LIKE=debian
...
This means that "22.04" is read as the value of ID on Ubuntu, which is
clearly wrong.
Instead, use the same approach as profile.d/toolbox.sh and the old POSIX
shell implementation that doesn't rely on the order of the os-release(5)
fields.
Fallout from 54a2ca1eadhttps://github.com/containers/toolbox/pull/1320
We wasted some time trying to get the tests running locally, when all we
were missing were the 'git submodule ...' commands.
Add some more obvious hints about this possible stumbling block.
Note that Bats cautions against printing outside the @test, setup* or
teardown* functions [1]. In this case, doing so leads to the first line
of the error output going missing, when using the pretty formatter for
human consumption:
$ bats --formatter pretty ./test/system
✗ setup_suite
Forgot to run 'git submodule init' and 'git submodule update' ?
bats warning: Executed 1 instead of expected 191 tests
191 tests, 1 failure, 190 not run
[1] https://bats-core.readthedocs.io/en/stable/writing-tests.htmlhttps://github.com/containers/toolbox/pull/1298
Signed-off-by: Matthias Clasen <mclasen@redhat.com>
The 000-setup.bats and 999-teardown.bats files were added [1] at a time
when Bats didn't offer any hooks for suite-wide setup and teardown.
That changed in Bats 1.7.0, which introduced the setup_suite and
teardown_suite hooks. These hooks make it easier to run a subset of the
tests, which is a good thing.
In the past, to run a subset of the tests, one had to do:
$ bats ./test/system/000-setup.bats ./test/system/002-help.bats \
./test/system/999-teardown.bats
Now, one only has to do:
$ bats ./test/system/002-help.bats
Commit e22a82fec8 already added a dependency on Bats >= 1.7.0.
Therefore, it should be exploited wherever possible to simplify things.
[1] Commit 54a2ca1eadhttps://github.com/containers/toolbox/issues/751
[2] Bats commit fb467ec3f04e322a
https://github.com/bats-core/bats-core/issues/39https://bats-core.readthedocs.io/en/stable/writing-tests.htmlhttps://github.com/containers/toolbox/pull/1317
Bats 1.7.0 emits a warning if a feature that is only available starting
from a certain version of Bats onwards is used without specifying that
version [1]:
BW02: Using flags on `run` requires at least BATS_VERSION=1.5.0. Use
`bats_require_minimum_version 1.5.0` to fix this message.
(from function `bats_warn_minimum_guaranteed_version' in file
/usr/lib/bats-core/warnings.bash, line 32,
from function `run' in file
/usr/lib/bats-core/test_functions.bash, line 227,
in test file test/system/001-version.bats, line 27)
Note that bats_require_minimum_version itself is only available from
Bats 1.7.0 [2]. Hence, even though the specific feature here (using
flags on 'run') only requires Bats >= 1.5.0, in practice Bats >= 1.7.0
is needed. Fortunately, commit e22a82fec8 already added a
dependency on Bats >= 1.7.0. So, there's nothing to worry about.
[1] Bats commit 82002bb6c1a5c418
https://github.com/bats-core/bats-core/issues/556https://bats-core.readthedocs.io/en/stable/warnings/BW02.html
[2] Bats commit 71d6b71cebc3d32b
https://github.com/bats-core/bats-core/issues/556https://bats-core.readthedocs.io/en/stable/warnings/BW02.htmlhttps://github.com/containers/toolbox/pull/1315
Commit e22a82fec8 already added a dependency on Bats >= 1.7.0,
which is present on Fedora >= 36. Therefore, it should be exploited
wherever possible to simplify things.
Earlier, when the line counts were checked only with Bats >= 1.7.0,
there was a need to separately check the whole standard error and
output streams with 'assert_output' for the tests to be useful on
Fedora 35, which only had Bats 1.5.0. Now that the line counts are
being checked unconditionally, there's no need for that anymore.
Note that bats_require_minimum_version itself is only available from
Bats 1.7.0 [1].
[1] Bats commit 71d6b71cebc3d32b
https://github.com/bats-core/bats-core/issues/556https://bats-core.readthedocs.io/en/stable/warnings/BW02.htmlhttps://github.com/containers/toolbox/pull/1314
This allows using the 'distro' option to create and enter Arch Linux
containers. Due to Arch's rolling-release model, the 'release' option
isn't required. If 'release' is used, the accepted values are 'latest'
and 'rolling'.
https://github.com/containers/toolbox/pull/1311
It's quite obvious what the corresponding code is doing, and it isn't
any harder to understand than the rest of the code that's not commented.
https://github.com/containers/toolbox/pull/1282
This is a quick sanity check with 'podman images' to ensure that all the
images are in place before running 'list'. Other tests already do this,
so this change makes these two tests consistent with the rest.
https://github.com/containers/toolbox/pull/1273
This is the 'simple' case of having a well-known Toolbx image (ie.,
not a copy, not an image without a name, not a non-Toolbx image). It's
good to ensure that the default image works as expected with 'list'
before moving on to more complex scenarios.
https://github.com/containers/toolbox/pull/1278
Currently, some of the names of the tests were too long, and had
inconsistent and verbose wording. This made it difficult to look at
them and get a gist of all the scenarios being tested. The names are
like headings. They shouldn't be too long, should capture the primary
objective of the test and be consistent in their wording.
https://github.com/containers/toolbox/pull/1276
Currently, some of the names of the tests were too long, and had
inconsistent and verbose wording. This made it difficult to look at
them and get a gist of all the scenarios being tested. The names are
like headings. They shouldn't be too long, should capture the primary
objective of the test and be consistent in their wording.
https://github.com/containers/toolbox/pull/1271
Currently, some of the names of the tests were too long, and had
inconsistent and verbose wording. This made it difficult to look at
them and get a gist of all the scenarios being tested. The names are
like headings. They shouldn't be too long, should capture the primary
objective of the test and be consistent in their wording.
https://github.com/containers/toolbox/pull/1265
Currently, the titles of the manuals are rendered with a pair of empty
parentheses and no section title:
toolbox(1)() toolbox(1)()
NAME
toolbox - Tool for containerized command line environments...
However, they should be:
toolbox(1) General Commands Manual toolbox(1)
NAME
toolbox - Tool for containerized command line environments...
This is because the troff generated by go-md2man from Markdown has a
faulty invocation of the .TH macro [1]:
.nh
.TH toolbox(1)
.SH NAME
.PP
toolbox - Tool for containerized command line environments on Linux
It should be:
.nh
.TH toolbox 1
.SH NAME
.PP
toolbox - Tool for containerized command line environments on Linux
Original patch from Andrew Denton for Podman [2].
[1] https://www.gnu.org/software/groff/manual/groff.html
[2] Podman commit 63c779a857b55b00
https://github.com/containers/podman/pull/15621https://github.com/containers/toolbox/pull/1210
Otherwise codespell would complain:
: @test "create: Try to create a container with invalid custom name...
> run $TOOLBOX -y create "ßpeci@l.Nam€"
:
./test/system/101-create.bats:57: Nam ==> Name
CentOS Stream 9 has codespell-2.2.1, while so far the 'unit tests' were
being run on Fedora 36, which only has codespell-2.1.0.
This is a step towards testing on CentOS Stream 9.
https://github.com/containers/toolbox/pull/1200
CentOS Stream 9 has codespell-2.2.1, while so far the 'unit tests' were
being run on Fedora 36, which only has codespell-2.1.0.
This is a step towards testing on CentOS Stream 9.
Fallout from ecd1ced719https://github.com/containers/toolbox/pull/1200
Note that 'run --keep-empty-lines' counts the trailing newline on the
last line as a separate line.
Until Bats 1.7.0, 'run --keep-empty-lines' had a bug where even when a
command produced no output, it would report a line count of one [1] due
to a stray line feed character. This needs to be conditionalized, since
Fedora 35 has Bats 1.5.0.
[1] https://github.com/bats-core/bats-core/issues/573https://github.com/containers/toolbox/issues/1043
Currently, if an image was copied with:
$ skopeo copy \
containers-storage:registry.fedoraproject.org/fedora-toolbox:36 \
containers-storage:localhost/fedora-toolbox:36
... or:
$ podman tag \
registry.fedoraproject.org/fedora-toolbox:36 \
localhost/fedora-toolbox:36
... then it would show up twice in 'list' with the same name, and in the
wrong order.
Either as:
$ toolbox list --images
IMAGE ID IMAGE NAME CREATED
2110dbbc33d2 localhost/fedora-toolbox:36 1 day...
e085805ade4a registry.access.redhat.com/ubi8/toolbox:latest 1 day...
2110dbbc33d2 localhost/fedora-toolbox:36 1 day...
70cbe2ce60ca registry.fedoraproject.org/fedora-toolbox:34 1 day...
... or as:
$ toolbox list --images
IMAGE ID IMAGE NAME CREATED
2110dbbc33d2 registry.fedoraproject.org/fedora-toolbox:36 1 day...
e085805ade4a registry.access.redhat.com/ubi8/toolbox:latest 1 day...
2110dbbc33d2 registry.fedoraproject.org/fedora-toolbox:36 1 day...
70cbe2ce60ca registry.fedoraproject.org/fedora-toolbox:34 1 day...
The correct output should be similar to 'podman images', and be sorted
in ascending order of the names:
$ toolbox list --images
IMAGE ID IMAGE NAME CREATED
2110dbbc33d2 localhost/fedora-toolbox:36 1 day...
e085805ade4a registry.access.redhat.com/ubi8/toolbox:latest 1 day...
70cbe2ce60ca registry.fedoraproject.org/fedora-toolbox:34 1 day...
2110dbbc33d2 registry.fedoraproject.org/fedora-toolbox:36 1 day...
The problem is that, in these situations, 'podman images --format json'
returns separate identical JSON collections for each copy of the image,
and all of those copies have multiple names:
[
{
"Id": "2110dbbc33d2",
...
"Names": [
"localhost/fedora-toolbox:36",
"registry.fedoraproject.org/fedora-toolbox:36"
],
...
},
{
"Id": "e085805ade4a",
...
"Names": [
"registry.access.redhat.com/ubi8/toolbox:latest"
],
...
},
{
"Id": "2110dbbc33d2",
...
"Names": [
"localhost/fedora-toolbox:36",
"registry.fedoraproject.org/fedora-toolbox:36"
],
...
}
{
"Id": "70cbe2ce60ca",
...
"Names": [
"registry.fedoraproject.org/fedora-toolbox:34"
],
...
},
]
The image objects need to be flattened to have only one unique name per
copy, but with the same ID, and then sorted to ensure the right order.
Note that the ordering was already broken since commit 2369da5d31,
which started using 'podman images --sort repository'. Podman can sort
by either the image's repository or tag, but not by the unified name,
which is what Toolbx needs. Therefore, even without copied images,
Toolbx really does need to sort the images itself.
Prior to commit 2369da5d31, the ordering was correct, but copied
images would only show up once.
Fallout from 2369da5d31
This reverts parts of commit 67e210378e.
https://github.com/containers/toolbox/issues/1043
If an image was copied with:
$ skopeo copy \
containers-storage:registry.fedoraproject.org/fedora-toolbox:36 \
containers-storage:localhost/fedora-toolbox:36
... or:
$ podman tag \
registry.fedoraproject.org/fedora-toolbox:36 \
localhost/fedora-toolbox:36
... then the image ID is only showed once in 'podman images --quiet',
not twice.
A subsequent commit will use this to write tests to ensure that copied
images are correctly handled.
https://github.com/containers/toolbox/issues/1043
Note that 'run --keep-empty-lines' counts the trailing newline on the
last line as a separate line.
Until Bats 1.7.0, 'run --keep-empty-lines' had a bug where even when a
command produced no output, it would report a line count of one [1] due
to a stray line feed character. This needs to be conditionalized, since
Fedora 35 has Bats 1.5.0.
[1] https://github.com/bats-core/bats-core/issues/573https://github.com/containers/toolbox/pull/1192
Note that 'run --keep-empty-lines' counts the trailing newline on the
last line as a separate line.
Until Bats 1.7.0, 'run --keep-empty-lines' had a bug where even when a
command produced no output, it would report a line count of one [1] due
to a stray line feed character. This needs to be conditionalized, since
Fedora 35 has Bats 1.5.0.
[1] https://github.com/bats-core/bats-core/issues/573https://github.com/containers/toolbox/pull/1192
A subsequent commit will test the order in which images with and without
names are listed. It's logical for that test to come after the one
about the basic support for images without names.
https://github.com/containers/toolbox/pull/1192
This was making it difficult to read the Bats assertions on test
failures, by polluting it with unexpected and irrelevant output from
'podman images'. For example [1]:
not ok 39 list: Images with and without names in 12332ms
# (from function `assert' in file test/system/libs/bats-assert/src/assert.bash, line 46,
# in test file test/system/102-list.bats, line 126)
# `assert [ ${#stderr_lines[@]} -eq 0 ]' failed
# REPOSITORY TAG IMAGE ID CREATED SIZE
# registry.fedoraproject.org/fedora-toolbox 35 862705390e8b 4 weeks ago 332 MB
# REPOSITORY TAG IMAGE ID CREATED SIZE
# registry.fedoraproject.org/fedora-toolbox 35 862705390e8b 4 weeks ago 332 MB
# registry.fedoraproject.org/fedora-toolbox 34 70cbe2ce60ca 7 months ago 354 MB
#
# -- assertion failed --
# expression : [ 1 -eq 0 ]
# --
#
Fallout from 7973181136
[1] https://github.com/containers/toolbox/pull/1192https://github.com/containers/toolbox/pull/1193
Commit ae43560d45 had added a test with a similar intention. When
the test suite is run on a Fedora Rawhide host, it tests whether the
containers for the two previous stable Fedora releases start or not.
Fedora N-2 reaches End of Life 4 weeks after Fedora N is released [1].
So, testing the containers for Fedora Rawhide and the two previous
stable releases on a Fedora Rawhide host is a decent test of general
backwards compatibility.
However, as seen recently [2], this isn't enough to catch some known
ABI compatibility issues [3,4]. These involve toolbox binaries built
on hosts with newer toolchains that aren't meant to be run against
containers with older runtimes. A targeted test is needed to defend
against these scenarios.
The fedora-toolbox:34 image has glibc-2.33, which is old enough to be
unable to run binaries compiled on Fedora 35 with glibc-2.34 and newer.
[1] https://docs.fedoraproject.org/en-US/releases/
[2] https://github.com/containers/toolbox/pull/1180
[3] Commit 6063eb27b9https://github.com/containers/toolbox/issues/821
[4] Commit 6ad9c63180https://github.com/containers/toolbox/issues/529https://github.com/containers/toolbox/pull/1187
Fedora 32 reached End of Life on 25th May 2021:
https://docs.fedoraproject.org/en-US/releases/eol/
That's quite old because right now Fedora 35 is nearing its End of Life.
Since the tests are intended for Toolbx, not the Fedora infrastructure,
it will be better to use a newer image, because images that are too old
can get lost from registry.fedoraproject.org. The fedora-toolbox:34
image can be a drop-in replacement for the fedora-toolbox:32 image for
the purposes of this test suite, and has the advantage of being newer.
Note that fedora-toolbox:34 is also old enough to test that the toolbox
binary runs against it's build-time ABI from the host, and not the
Toolbx container's ABI, when it's invoked as the entry point of the
container [1,2]. This is important because the subsequent commit will
add a test to ensure that.
[1] Commit 6063eb27b9https://github.com/containers/toolbox/issues/821
[2] Commit 6ad9c63180https://github.com/containers/toolbox/issues/529https://github.com/containers/toolbox/pull/1187
This is a precursor to checking that higher valued exit codes from the
command running inside the container are retained, and commands like
test(1) can be used with 'toolbox run ...' in subsequent test cases.
https://github.com/containers/toolbox/pull/1163
Currently, some of the names of the tests were too long, and had
inconsistent and verbose wording. This made it difficult to look at
them and get a gist of all the scenarios being tested. The names are
like headings. They shouldn't be too long, should capture the primary
objective of the test and be consistent in their wording.
https://github.com/containers/toolbox/pull/1161
Commit a22d7821cb ensured that a nested pseudo-terminal device is
only created for the process running inside the container, if the Toolbx
binary's standard input and output streams are connected to a terminal.
Therefore, 'echo ...' no longer ends with an unwanted extra carriage
return when terminal devices are absent - there's only a line feed for
the trailing newline. Hence, there's no need to use the -n flag to skip
the trailing newline.
This reverts parts of commit 16b0c5d88f.
https://github.com/containers/toolbox/issues/157
It seems that as new test cases got developed they got appended towards
the end of the file. Now that there are a non-trivial number of test
cases, it's difficult to look at the file and get a gist of all the
scenarios being tested.
It will be better to have some logical grouping -- starting with the
most basic functionality, then moving on to more advanced features,
and then finally the errors.
This is a step towards that.
https://github.com/containers/toolbox/pull/1155
This needs a directory that's going to be present on the host operating
system across various configurations of all supported distributions,
such as the hosts running the CI, but not inside the Toolbx containers.
It looks like /etc/kernel is present on both Debian and Fedora, but
absent from the fedora-toolbox images. On a Debian 10 server, it's
owned by several packages:
$ dpkg-query --search /etc/kernel
dkms, systemd, grub2-common, initramfs-tools, apt: /etc/kernel
... while on Fedora 36 Workstation:
$ rpm --file --query /etc/kernel
systemd-udev-250.8-1.fc36.x86_64
Currently, there's no way to get assert_line to use the stderr_lines
array [1]. This is worked around by assigning stderr_lines to the
'lines' array.
[1] https://github.com/bats-core/bats-assert/issues/42https://github.com/containers/toolbox/pull/1153
It seems that as new test cases got developed they got appended towards
the end of the file. Now that there are a non-trivial number of test
cases, it's difficult to look at the file and get a gist of all the
scenarios being tested.
It will be better to have some logical grouping -- starting with the
most basic functionality, then moving on to more advanced features,
and then finally the errors.
This a step towards that.
https://github.com/containers/toolbox/pull/1152
Currently, commands invoked using 'toolbox run' have a different
environment than the interactive environment offered by 'toolbox enter'.
This is because 'toolbox run' was invoking the commands using something
like this:
$ bash -c 'exec "$@"' bash [COMMAND]
... whereas, 'toolbox enter' was using something like this:
$ bash -c 'exec "$@"' bash bash --login
In the first case, the helper Bash shell is a non-interactive non-login
shell. This means that it doesn't read any of the usual start-up files,
and, hence, it doesn't pick up anything that's specified in them. It
runs with the default environment variables set up by Podman and the
Toolbx image, plus the environment variables set by Toolbx itself.
In the second case, even though the helper Bash shell is still the same
as the first, it eventually invokes a login shell, which runs the usual
set of start-up files and picks up everything that's specified in them.
Therefore, to ensure parity, 'toolbox run' should always have a login
shell in the call chain inside the Toolbx container.
The easiest option is to always use a helper shell that's a login shell
with 'toolbox run', but not 'toolbox enter' so as to avoid reading the
same start-up files twice, due to two login shells in the call chain.
It will still end up reading the same start-up files twice, if someone
tried to invoke a login shell through 'toolbox run', which is fine.
It's very difficult to be sure that the user is invoking a login shell
through 'toolbox run', and it's not what most users will be doing.
https://github.com/containers/toolbox/issues/1076
For the most part, this fixes a minor cosmetic issue for users, but it
does make the code less misleading to read for those hacking on Toolbx.
Further details below.
Commands are invoked inside a Toolbx from a helper shell invoked by
capsh(1). Unless capsh(1) is built with custom options, the helper
shell is always bash, not /bin/sh:
$ capsh --caps="" -- -c 'echo "$(readlink /proc/$$/exe)"'
/usr/bin/bash
( The possibility of capsh(1) using a different shell, other than Bash,
through a custom build option is ignored for the time being. If there
really are downstream distributors who do that, then this can be
addressed one way or another. )
Secondly, the name assigned to the embedded command string's '$0' should
only be the basename of the helper shell's binary, not the full path, to
match the usual behaviour:
$ bash -c 'exec foo'
bash: line 1: exec: foo: not found
With 'toolbox run' it was:
$ toolbox run foo
/bin/sh: line 1: exec: foo: not found
Error: command foo not found in container fedora-toolbox-36
https://github.com/containers/toolbox/pull/1147
Using 'true' is likely going to be quicker than launching the entire
shell (ie., /bin/sh).
Note that 'toolbox run' already invokes a wrapper shell via capsh(1)
before invoking the user-specified command. So, this was the second
instance of a shell.
https://github.com/containers/toolbox/pull/1145
In 54a2ca1 image caching has been done by first pulling using Podman and
then moving the image from the local container store to a directory. The
pull to the local container store can be skipped and instead we can use
Skopeo to directly save the pulled image into a directory.
On my machine this reduced the time of the system test setup "test" by
about 50 seconds. This speed-up largely depends on the available network
connection, though.
Currently, the container name and release are only validated if they
were specified as command line options. Neither the value of release
in the configuration file nor the container name generated from an
image are validated.
There's also a lot of repeated code in the command front-ends to
validate the container name and release. This opens the door for
mistakes. Any adjustment to the code must be repeated elsewhere, and
there are subtle interactions and overlaps between the validation code
and the code to resolve container and image names.
It's worth noting that the container and image name resolution happens
for both the command line and configuration file options, and generates
the container name from the image when necessary.
Therefore, validating everything while resolving cleans up the command
front-ends and increases the coverage of the validation.
This introduces the use of sentinel error values and custom error
implementations to identify the different errors that can occur while
resolving the container and images, so that they can be appropriately
shown to the user.
https://github.com/containers/toolbox/pull/1101