This needs a directory that's going to be present on the host operating
system across various configurations of all supported distributions,
such as the hosts running the CI, but not inside the Toolbx containers.
It looks like /etc/kernel is present on both Debian and Fedora, but
absent from the fedora-toolbox images. On a Debian 10 server, it's
owned by several packages:
$ dpkg-query --search /etc/kernel
dkms, systemd, grub2-common, initramfs-tools, apt: /etc/kernel
... while on Fedora 36 Workstation:
$ rpm --file --query /etc/kernel
systemd-udev-250.8-1.fc36.x86_64
Currently, there's no way to get assert_line to use the stderr_lines
array [1]. This is worked around by assigning stderr_lines to the
'lines' array.
[1] https://github.com/bats-core/bats-assert/issues/42https://github.com/containers/toolbox/pull/1153
It seems that as new test cases got developed they got appended towards
the end of the file. Now that there are a non-trivial number of test
cases, it's difficult to look at the file and get a gist of all the
scenarios being tested.
It will be better to have some logical grouping -- starting with the
most basic functionality, then moving on to more advanced features,
and then finally the errors.
This a step towards that.
https://github.com/containers/toolbox/pull/1152
Currently, commands invoked using 'toolbox run' have a different
environment than the interactive environment offered by 'toolbox enter'.
This is because 'toolbox run' was invoking the commands using something
like this:
$ bash -c 'exec "$@"' bash [COMMAND]
... whereas, 'toolbox enter' was using something like this:
$ bash -c 'exec "$@"' bash bash --login
In the first case, the helper Bash shell is a non-interactive non-login
shell. This means that it doesn't read any of the usual start-up files,
and, hence, it doesn't pick up anything that's specified in them. It
runs with the default environment variables set up by Podman and the
Toolbx image, plus the environment variables set by Toolbx itself.
In the second case, even though the helper Bash shell is still the same
as the first, it eventually invokes a login shell, which runs the usual
set of start-up files and picks up everything that's specified in them.
Therefore, to ensure parity, 'toolbox run' should always have a login
shell in the call chain inside the Toolbx container.
The easiest option is to always use a helper shell that's a login shell
with 'toolbox run', but not 'toolbox enter' so as to avoid reading the
same start-up files twice, due to two login shells in the call chain.
It will still end up reading the same start-up files twice, if someone
tried to invoke a login shell through 'toolbox run', which is fine.
It's very difficult to be sure that the user is invoking a login shell
through 'toolbox run', and it's not what most users will be doing.
https://github.com/containers/toolbox/issues/1076
For the most part, this fixes a minor cosmetic issue for users, but it
does make the code less misleading to read for those hacking on Toolbx.
Further details below.
Commands are invoked inside a Toolbx from a helper shell invoked by
capsh(1). Unless capsh(1) is built with custom options, the helper
shell is always bash, not /bin/sh:
$ capsh --caps="" -- -c 'echo "$(readlink /proc/$$/exe)"'
/usr/bin/bash
( The possibility of capsh(1) using a different shell, other than Bash,
through a custom build option is ignored for the time being. If there
really are downstream distributors who do that, then this can be
addressed one way or another. )
Secondly, the name assigned to the embedded command string's '$0' should
only be the basename of the helper shell's binary, not the full path, to
match the usual behaviour:
$ bash -c 'exec foo'
bash: line 1: exec: foo: not found
With 'toolbox run' it was:
$ toolbox run foo
/bin/sh: line 1: exec: foo: not found
Error: command foo not found in container fedora-toolbox-36
https://github.com/containers/toolbox/pull/1147
Using 'true' is likely going to be quicker than launching the entire
shell (ie., /bin/sh).
Note that 'toolbox run' already invokes a wrapper shell via capsh(1)
before invoking the user-specified command. So, this was the second
instance of a shell.
https://github.com/containers/toolbox/pull/1145
It was decided in commit 950f510872 that golang.org/x/* would be
used for the IsTerminal() API, not github.com/mattn/go-isatty. However,
github.com/mattn/go-isatty had crept in through commits f49df914f4
and a22d7821cb.
The size savings seem to have been lost, because with Go 1.18.6, the
binary size actually grew from 9410616 bytes to 9410912. However, it
seems better to stick to packages from the golang.org domain, whenever
possible.
https://github.com/containers/toolbox/pull/1144
... at https://containertoolbx.org/install/
There are some minor benefits to always invoking meson(1), as opposed to
directly invoking the underlying build backend, like 'ninja'.
It's one less command to be aware of. Secondly, in theory, Meson can be
used with backends other than Ninja (see 'meson configure'), even though
Ninja is the most likely option for building Toolbx because it's only
supported on Linux.
https://github.com/containers/toolbox/pull/1142
If 'systemd-tmpfiles --create' is called as a non-root user, then it
causes:
--- stdout ---
Calling systemd-tmpfiles --create ...
--- stderr ---
Failed to open directory 'cryptsetup': Permission denied
Failed to open directory 'certs': Permission denied
Failed to create directory or subvolume "/var/spool/cups/tmp":
Permission denied
...
...
...
Traceback (most recent call last):
File "toolbox/meson_post_install.py", line 26, in <module>
subprocess.run(['systemd-tmpfiles', '--create'], check=True)
File "/usr/lib64/python3.10/subprocess.py", line 524, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['systemd-tmpfiles',
'--create']' returned non-zero exit status 73.
Since, systemd-tmpfiles(8) can't be used like this as a non-root user,
there's no point in calling it and needlessly failing the build.
Unfortunately, Meson doesn't seem to offer a way to get the process'
effective UID inside its scripts. Therefore, this leaves a spurious
build-time dependency on systemd when building as a non-root user.
https://github.com/containers/toolbox/pull/1140
The bash-completion and fish dependencies were already optional - the
shell completions for Bash and fish won't be generated and installed if
they are absent; and there's no dependency required for Z shell. So the
install_completions build option wasn't reducing the dependency burden.
The build option was a way to disable the generation and installation of
the shell completions, regardless of whether the necessary dependencies
are present or not. The only use-case for this is when installing to a
non-system-wide prefix while hacking on Toolbox as a non-root user,
because the locations for the completions advertised by the shells' APIs
might not be accessible. Being able to disable the completions prevents
the installation from failing.
A different way of ensuring a smooth developer experience for a Toolbx
hacker is to offer a way to change the locations where the shell
completions are installed, which is necessary and beneficial for other
use-cases.
Z shell, unlike Bash's bash-completion.pc and fish's fish.pc, doesn't
offer an API to detect the location for the shell completions. This
means that Debian and Fedora use different locations [1, 2]. Namely,
/usr/share/zsh/vendor-completions and /usr/share/zsh/site-functions.
An option to specify the locations for the shell completions can
optimize the build, if there's an alternate API for the location that
doesn't involve using bash-completion.pc and fish.pc as build
dependencies. eg., Fedora provides the _tmpfilesdir RPM macro to
specify the location for vendor-supplied tmpfiles.d(5) files, which
makes it possible to avoid having systemd.pc as a build dependency [3].
Fallout from bafbbe81c9
[1] Debian zsh commit bf0a44a8744469b5
https://salsa.debian.org/debian/zsh/-/commit/bf0a44a8744469b5https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=620452
[2] https://src.fedoraproject.org/rpms/zsh/blob/f37/f/zsh.spec
[3] Fedora toolbox commit 9bebde5bb60f36e3
https://src.fedoraproject.org/rpms/toolbox/c/9bebde5bb60f36e3https://github.com/containers/toolbox/pull/1123https://github.com/containers/toolbox/pull/840
Noticed today that `man xargs` was returning the POSIX manpage instead
of the one shipped by `findutils`.
Signed-off-by: Jonathan Lebon <jonathan@jlebon.com>
The following packages have also been added to Fedora 38 image:
mesa-dri-drivers
mesa-vulkan-drivers
vulkan-loader
Fixing up fedora 38 image to match the changes made earlier on fedora 37.
Signed-off-by: Nieves Montero <nmontero@redhat.com>
This new packet allows the user to set a locale inside the
toolbox and make locale dependent commands work
https://github.com/containers/toolbox/issues/60
Signed-off-by: Nieves Montero <nmontero@redhat.com>
In 54a2ca1 image caching has been done by first pulling using Podman and
then moving the image from the local container store to a directory. The
pull to the local container store can be skipped and instead we can use
Skopeo to directly save the pulled image into a directory.
On my machine this reduced the time of the system test setup "test" by
about 50 seconds. This speed-up largely depends on the available network
connection, though.
The following packages have also been added to images f36 and f35:
mesa-dri-drivers
mesa-vulkan-drivers
vulkan-loader
https://github.com/containers/toolbox/pull/1124
Signed-off-by: Nieves Montero <nmontero@redhat.com>
The following packages have been added to the
image to make OpenGL and Vulkan work:
mesa-dri-drivers
mesa-vulkan-drivers
vulkan-loader
https://github.com/containers/toolbox/issues/1110
Signed-off-by: Nieves Montero <nmontero@redhat.com>
If systemd-tmpfiles(8) couldn't be spawned, then the attempted command
is already included in the traceback:
Traceback (most recent call last):
File "toolbox/meson_post_install.py", line 26, in <module>
subprocess.run(['systemd-tmpfiles', '--create'], check=True)
File "/usr/lib64/python3.10/subprocess.py", line 524, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['systemd-tmpfiles',
'--create']' returned non-zero exit status 73.
https://github.com/containers/toolbox/pull/1122
In short, it's a lot of effort to cover all possible exceptions that can
be thrown, and things work reasonably well even without handling them.
Since this is just part of the build, there's no point in complicating
things for aesthetic reasons.
More details below.
First, not every runtime error leads to a subprocess.CalledProcessError.
It's only thrown if the spawned process returns with a non-zero exit
code. There can be other problems. eg., if the gofmt file isn't
executable then a PermissionError is thrown that's currently not
handled, and the wrapper Python script returns with a non-zero exit
code:
Traceback (most recent call last):
File "toolbox/src/meson_go_fmt.py", line 28, in <module>
gofmt = subprocess.run(['gofmt', '-d', source_dir],
capture_output=True, check=True)
File "/usr/lib64/python3.10/subprocess.py", line 501, in run
with Popen(*popenargs, **kwargs) as process:
File "/usr/lib64/python3.10/subprocess.py", line 969, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/lib64/python3.10/subprocess.py", line 1845, in
_execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
PermissionError: [Errno 13] Permission denied: 'gofmt'
Second, when a subprocess.CalledProcessError is thrown, the wrapper
Python script will still return with a non-zero exit code with an
understandable error message, even if the exception isn't handled. eg.,
if 'meson install' is called without the adequate permissions, then
systemd-tmpfiles(8) will return with a non-zero exit code, which shows
up as:
--- stdout ---
Calling systemd-tmpfiles --create ...
--- stderr ---
Failed to open directory 'cryptsetup': Permission denied
Failed to open directory 'certs': Permission denied
Failed to create directory or subvolume "/var/spool/cups/tmp":
Permission denied
...
...
...
Traceback (most recent call last):
File "toolbox/meson_post_install.py", line 26, in <module>
subprocess.run(['systemd-tmpfiles', '--create'], check=True)
File "/usr/lib64/python3.10/subprocess.py", line 524, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['systemd-tmpfiles',
'--create']' returned non-zero exit status 73.
Similarly, if there problems generating the shell completions:
--- stderr ---
Error: unknown command "__completion" for "toolbox"
Run 'toolbox --help' for usage.
exit status 1
Traceback (most recent call last):
File "toolbox/completion/generate_completions.py", line 35, in
<module>
output = subprocess.run(['go', 'run', '.', '__completion',
completion_type], check=True)
File "/usr/lib64/python3.10/subprocess.py", line 524, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['go', 'run', '.',
'__completion', 'bash']' returned non-zero exit status 1.
https://github.com/containers/toolbox/pull/1122
Cobra provides a default command 'completion' that is always visible.
The reverted change caused an additional command 'completion' to show up
in the list because the then called command '__completion' didn't
override the default one. This became apparent due to d69ce6794b
dynamically generating completion arguments for the 'help' command.
This reverts commit 4469774fb1.
https://github.com/containers/toolbox/pull/1055https://github.com/containers/toolbox/pull/1121
By default, the value of the 'build_by_default' argument is determined
by the value of the 'install' argument, which was set to 'true' once the
Go implementation was considered stable enough for end users.
Fallout from 0b3c66434ehttps://github.com/containers/toolbox/pull/1116
When describing the --authfile option, the word 'private' is used to
refer to images needing authentication. Using the same word shortens
the text so that the word 'custom' can be used in the same way as in the
other examples.
https://github.com/containers/toolbox/pull/1107
It turns out that Viper's custom error implementations use non-pointer
receivers, whereas often people assume pointer receivers. This can
cause confusion when trying to use errors.As(...) with those errors [1].
Secondly, Viper may or may not throw ConfigFileNotFoundError depending
on its build tags.
[1] https://github.com/spf13/viper/issues/1139https://github.com/containers/toolbox/pull/1105
Currently, the container name and release are only validated if they
were specified as command line options. Neither the value of release
in the configuration file nor the container name generated from an
image are validated.
There's also a lot of repeated code in the command front-ends to
validate the container name and release. This opens the door for
mistakes. Any adjustment to the code must be repeated elsewhere, and
there are subtle interactions and overlaps between the validation code
and the code to resolve container and image names.
It's worth noting that the container and image name resolution happens
for both the command line and configuration file options, and generates
the container name from the image when necessary.
Therefore, validating everything while resolving cleans up the command
front-ends and increases the coverage of the validation.
This introduces the use of sentinel error values and custom error
implementations to identify the different errors that can occur while
resolving the container and images, so that they can be appropriately
shown to the user.
https://github.com/containers/toolbox/pull/1101
Currently, if an invalid or unsupported string is specified as the
distro on the command line or in the configuration file, then it would
silently fallback to Fedora. This shouldn't happen.
It should only fallback to Fedora when no distro was specified and
there's no supported Toolbox image matching the host operating system.
If a distro was explicitly specified then it should either be supported
or it should error out.
The test cases were resurrected from commit 8b6418d8aa.
https://github.com/containers/toolbox/issues/937https://github.com/containers/toolbox/pull/1080
