Currently, some of the names of the tests were too long, and had
inconsistent and verbose wording. This made it difficult to look at
them and get a gist of all the scenarios being tested. The names are
like headings. They shouldn't be too long, should capture the primary
objective of the test and be consistent in their wording.
https://github.com/containers/toolbox/pull/1265
This uses 'skopeo inspect' to get the size of the image on the registry,
which is usually less than the size of the image in a local
containers/storage image store after download (eg., 'podman images'),
because they are kept compressed on the registry. Skopeo >= 1.10.0 is
needed to retrieve the sizes [1].
However, this doesn't add a hard dependency on Skopeo to accommodate
size-constrained operating systems like Fedora CoreOS. If skopeo(1) is
missing or too old, then the size of the image won't be shown, but
everything else would continue to work as before.
Some changes by Debarshi Ray.
[1] Skopeo commit d9dfc44888ff71a6
https://github.com/containers/skopeo/commit/d9dfc44888ff71a6https://github.com/containers/skopeo/issues/641https://github.com/containers/toolbox/issues/752
Signed-off-by: Nieves Montero <nmontero@redhat.com>
Bind mounting the locations at runtime doesn't really have anything to
do with whether /run/host/etc is present inside the Toolbx container.
The only possible exception could have been /etc/machine-id, but it
isn't, because the bind mount is only performed if the source at
/run/host/etc/machine-id is present.
This is a historical mistake that has persisted for a long time, since,
in practice, /run/host/etc will almost always exist inside the Toolbx
container. It's time to finally correct it.
Fallout from 9436bbece0https://github.com/containers/toolbox/pull/1255
The --monitor-host option was added to the 'init-container' command in
commit 8b84b5e460 to accommodate Podman versions older than 1.2.0
that didn't have the '--dns none' and '--no-hosts' options for
'podman create'. These options are necessary to keep the Toolbx
container's /etc/resolv.conf and /etc/hosts files synchronized with
those of the host.
Note that Podman 1.2.0 was already available a few months before
commit 8b84b5e460 introduced the --monitor-host option. The
chances of someone using an older Podman back then was already on the
decline, and it's very unlikely that a container created with such a
Podman has survived till this date.
Commit b6b484fa79 raised the minimum required Podman version to
1.4.0, and made the '--dns none' and '--no-hosts' options a hard
requirement. The minimum required Podman version was again raised
recently in commit 8e80dd5db1 to 1.6.4. Therefore, these days,
there's no need to separately use the --monitor-host option of
'init-container' for newly created containers to indicate that the
Podman version wasn't older than 1.2.0.
Given all this, it's time to stop using the --monitor-host option of
'init-container', and assume that it's always set. The option is still
accepted to retain compatibility with existing Toolbx containers.
For containers that were created with the --monitor-host option, a
deprecation notice will be shown as:
$ podman start --attach CONTAINER
Flag --monitor-host has been deprecated, it does nothing
...
https://github.com/containers/toolbox/pull/617
So far the minimum required Podman version was 1.4.0, based on what used
to be available in RHEL 7. These days, Podman 1.6.4 is old enough to be
in RHEL 7.9. Hence it's time to bump the baseline.
https://github.com/containers/toolbox/pull/1253
This is meant to roughly replicate the build environments used by
downstream distributors to build toolbox(1). These can be restricted in
odd ways compared to a fully featured environment where toolbox(1) is
actually going to be used. eg., the inability to use podman(1) in the
case of Fedora or not having subordinate user and group ID ranges in the
case of openSUSE.
It's important to ensure that toolbox(1) can be built by downstream
distributors without any unnecessary hassle.
https://github.com/containers/podman/issues/17657https://github.com/containers/toolbox/issues/1246
Ever since commit bafbbe81c9, the shell completions are generated
while building Toolbx using the 'completion' command. This involves
running toolbox(1) itself, and hence validating the subordinate user and
group ID ranges.
Unfortunately, some build environments, like openSUSE's, don't have
subordinate ID ranges set up. Therefore, it's better to not validate
the subordinate ID ranges when generating the shell completions, since
they are generated by Cobra itself and subordinate ID ranges are not
involved at all.
Note that subordinate ID ranges may be needed when the generated shell
completions are actually used in interactive command line environments.
The shell completions invoke the hidden '__complete' command to get the
results that are presented to the user, and, if needed, the subordinate
ID ranges will continue to be used by podman(1) as part of that.
Some changes by Debarshi Ray.
https://github.com/containers/toolbox/issues/1246https://github.com/containers/toolbox/pull/1249
Having a separate convenience function reduces the indentation levels by
at least one, and sometimes two, and makes it easy to have more detailed
debug logs.
This will make the subsequent commit easier to read.
https://github.com/containers/toolbox/issues/1246
Ever since commit bafbbe81c9, the shell completions are generated
while building Toolbx using the 'completion' command. This involves
running toolbox(1) itself, and hence invoking 'podman version' to decide
if 'podman system migrate' is needed or not.
Unfortunately, some build environments, like Fedora's, are set up inside
a chroot(2) or systemd-nspawn(1) or similar, where 'podman version' may
not work because it does various things with namespaces(7) and clone(2)
that can, under certain circumstances, encounter an EPERM.
Therefore, it's better to avoid using podman(1) when generating the
shell completions, especially, since they are generated by Cobra itself
and podman(1) is not involved at all.
Note that podman(1) is needed when the generated shell completions are
actually used in interactive command line environments. The shell
completions invoke the hidden '__complete' command to get the results
that are presented to the user, and, if needed, 'podman system migrate'
will continue to be run as part of that.
This partially reverts commit f3e005d014
because podman(1) is now only an optional runtime dependency for the
system tests.
https://github.com/containers/podman/issues/17657
It's better not to use the global flag variables beyond the top-level
RunE functions, because sometimes the lower-level functions are re-used
from other files within the 'cmd' package. In this case,
createContainer(), and hence pullImage(), is also used in src/cmd/run.go
to implement the 'run' command. However, the 'run' command doesn't have
a --authflags option.
Since the default value of the flag is the zero value of the type, which
is a NOP in the code, it's likely that the code was still correct, but
it will be better to maintain some discipline here to highlight the
inputs needed by the lower-level functions. Otherwise, things can get
tangled up.
Fallout from ecd1ced719https://github.com/containers/toolbox/pull/1240
Just like /run/systemd/sessions makes it possible to get the seat for a
session ID, /run/systemd/users can make it possible to get the seat and
the session ID for a user's UID.
The absence of /run/systemd/users inside Toolbx containers isn't
currently causing problems for any use-case, but it seems very close
to the sort of things that were necessary to run a non-nested display
server from within a Toolbx container on a virtual terminal. It's not
impossible that in future some implementation details of the display
server stack may make /run/systemd/users necessary.
https://github.com/containers/toolbox/issues/992
Not having sd_booted(3) work inside Toolbx containers isn't currently
causing problems for any use-case. However, it did come in handy when
investigating how to run a non-nested display server from within a
Toolbx container on a virtual terminal, because it's necessary for
'systemd --user' to realize that the host operating system was booted
with systemd.
https://github.com/containers/toolbox/issues/992
Signed-off-by: Sebastian Wick <sebastian.wick@redhat.com>
Podman creates a private cgroup namespace for containers on cgroups v2
by default. The host's cgroupfs is mounted at /sys/fs/cgroup giving an
inconsistent view of the cgroups. Toolbx doesn't intend to provide a
segregated security domain. So, there is no need for a cgroup namespace
and Toolbx containers can just use the host's namespace.
Having a private cgroup namespace for containers isn't currently causing
problems for any use-case, but it did come in handy when investigating
how to run a non-nested display server from within a Toolbx container on
a virtual terminal. Since this requires a change to the 'podman create'
arguments, it's not going to have an effect on existing containers, and
re-creating containers is annoying for users. So, it might be better to
get ahead of the curve and do it preemptively.
https://github.com/containers/toolbox/issues/992
Signed-off-by: Sebastian Wick <sebastian.wick@redhat.com>
This is needed by display servers for creating udev device enumerators
that matches against tags.
https://github.com/containers/toolbox/issues/992
Signed-off-by: Jonas Ådahl <jadahl@gmail.com>
This is the full definition of the UBI-based toolbox image published for
RHEL 9.1 [1] at registry.access.redhat.com/ubi9/toolbox:9.1. Note that
the Dockerfile used to build this image was already available to the
public [2], but didn't include all the files necessary to build it.
However, this has some minor deviations from the published image. The
FROM line has been changed to registry.access.redhat.com/ubi9:9.1 so
that it can be built outside Red Hat's build system and always points to
the desired RHEL version. The extra-packages file doesn't have
gnupg2-smime because it doesn't seem to be actually part of the UBI RPM
repositories, and it's not clear how it works inside Red Hat's build
system. Otherwise, 'podman build' fails with:
STEP 11/14: RUN dnf -y install $(<extra-packages)
...
Last metadata expiration check: 0:00:23 ago on Tue Feb 7 18:50:13...
...
No match for argument: gnupg2-smime
...
Error: Unable to find a match: gnupg2-smime
Error: building at STEP "RUN dnf -y install $(<extra-packages)": while
running runtime: exit status 1
[1] https://catalog.redhat.com/software/containers/ubi9/toolbox/61532d7dd2c7f84a4d2ed86b
[2] https://catalog.redhat.com/software/containers/ubi9/toolbox/61532d7dd2c7f84a4d2ed86b?container-tabs=dockerfilehttps://github.com/containers/toolbox/pull/1232
The canonical copy of README.md contains banners and labels in the
header that aren't useful when the file is shipped as part of the
images. Hence, those were removed.
Only the images for currently maintained Fedoras (ie., 36, 37 and 38)
were updated.
https://github.com/containers/toolbox/pull/1231
It turns out that at least since Fedora 30 [1], the gnupg2 package has
been part of the fedora base image, because it's required by the dnf
package:
dnf -> python3-dnf -> python3-libdnf -> libdnf -> gpgme -> gnupg2
Hence, the need to restore the gnupg2 documentation that was stripped
out in the base image.
Only the images for currently maintained Fedoras (ie., 36, 37 and 38)
were updated.
[1] It's difficult to find out if the gnupg2 package wasn't part of the
fedora base image before Fedora 30, because those images are no
longer available from registry.fedoraproject.org.
https://github.com/containers/toolbox/pull/1228
Building an OCI image leads to so much spew that it's hard to notice if
something unexpected happened, and as seen in the previous commit [1],
unexpected things do happen.
Therefore, this adds a built-in test to ensure that the desired files
are actually present in the final image. Right now it only checks the
presence of some representative manuals to ensure that the packages
listed in the 'missing-docs' file really do get reinstalled, and the
documentation that was stripped out in the base image really does get
restored.
Only the images for currently maintained Fedoras (ie., 36, 37 and 38)
were updated.
[1] Commit 1fc50176c9https://github.com/containers/toolbox/pull/1226https://github.com/containers/toolbox/pull/1226
The RPM packages in the base 'fedora' image can be older than the those
currently available in the DNF 'updates' repository [1], but at the same
time newer than those available in the DNF 'fedora' repository [1]. The
first part happens because the base image isn't updated as often as the
individual packages, so the 'updates' repository can have newer RPMs.
The second part happens because the base image does get updated after a
stable Fedora has been released, and hence can have newer RPMs than the
'fedora' repository.
This is complicated by the fact that packages can get pulled directly
from Fedora's Koji build system into the base 'fedora' image before
they make it to one of the well-known repositories like 'fedora' or
'updates' [1]. These packages are marked as having come from the
koji-override-0 repository.
All that combined can lead to unexpected behaviour when DNF is invoked
to reinstall or swap the RPM packages in the base image. Some examples
below.
The base fedora:36 image contains glibc-minimal-langpack-2.35-20.fc36
that came from koji-override-0, while 'fedora' and 'updates' have
glibc-all-langpacks-2.35-4.fc36 and glibc-all-langpacks-2.35-22.fc36
respectively. This leads to:
STEP 8/15: RUN dnf -y swap glibc-minimal-langpack glibc-all-langpacks
Last metadata expiration check: 0:00:03 ago on Wed Feb 1 12:37:04...
Dependencies resolved.
======================================================================
Package Arch Version Repository
======================================================================
Installing:
glibc-all-langpacks x86_64 2.35-4.fc36 fedora
Removing:
glibc-minimal-langpack x86_64 2.35-20.fc36 @koji-override-0
Downgrading:
glibc x86_64 2.35-4.fc36 fedora
glibc-common x86_64 2.35-4.fc36 fedora
That's unexpected. Instead of upgrading all the glibc sub-packages to
the latest version from 'updates', it's downgrading them to the older
version from 'fedora'.
Similarly, the base fedora:36 image has bash-5.2.9-2.fc36.x86_64 from
koji-override-0, and there is bash-5.2.15-1.fc36.x86_64 in 'updates'.
This leads to:
STEP 10/15: RUN dnf -y reinstall $(<missing-docs)
Last metadata expiration check: 0:00:06 ago on Wed Feb 1 12:37:04...
Package acl available, but not installed.
No match for argument: acl
Installed package bash-5.2.9-2.fc36.x86_64 (from koji-override-0) not
available.
That's unexpected. Instead of upgrading bash to the latest version from
'updates', it's simply skipping the 'reinstall', which means that the
documentation that was stripped out in the base image doesn't get
restored.
Updating all the RPM packages in the base 'fedora' image to match the
contents of the 'updates' repository before making any changes to the
image's package set will avoid such unexpected behaviour.
Only the images for currently maintained Fedoras (ie., 36, 37 and 38)
were updated.
[1] https://docs.fedoraproject.org/en-US/quick-docs/repositories/https://github.com/containers/toolbox/pull/1226
The URLs for the RHEL Toolbx images based on the Red Hat Universal Base
Images (or UBI) are a bit more complicated to construct, in comparison
to the URLs for Fedora's fedora-toolbox images. It's not enough to just
concatenate the registry, the image's basename and the release. Some
parts of the URL depend on the release's major number, which requires
custom code.
So far, the release's major number was hard coded to 8 since only RHEL 8
Toolbx containers were supported.
To support other RHEL major releases, it's necessary to have custom code
to construct the URLs for the Toolbx images.
https://github.com/containers/toolbox/issues/1065
On enterprise FreeIPA set-ups, the subordinate user and group IDs are
provided by SSSD's sss plugin for the GNU Name Service Switch (or NSS)
functionality of the GNU C Library. They are not listed in /etc/subuid
and /etc/subgid. Therefore, its necessary to use libsubid.so to check
the subordinate ID ranges.
The CGO interaction with libsubid.so is loosely based on 'readSubid' in
github.com/containers/storage/pkg/idtools [1].
However, unlike 'readSubid', this code considers the absence of any
range (ie., nRanges == 0) to be an error as well.
More importantly, this code uses dlopen(3) and friends to dynamically
load the symbols from libsubid.so, instead of linking to libsubid.so at
build-time and having the dependency noted in the /usr/bin/toolbox
binary. This is done because libsubid.so itself depends on several
other shared libraries, and indirect dependencies can't be influenced
by the RUNPATH [2] embedded in the /usr/bin/toolbox binary [3]. Hence,
when the binary is used inside Toolbx containers (eg., as the entry
point), those indirect dependencies won't be picked from the host's
runtime against which the binary was built. This can render the binary
useless due to ABI compatibility issues. Using dlopen(3) avoids this
problem, especially because libsubid.so is only used when running on the
host.
Care was taken to not load and link libsubid.so twice to separately
validate the subordinate ID ranges for the user and the group. Note
that libsubid_init() must be passed a FILE pointer for logging.
Otherwise, it will create it's own for logging, and there's no way to
close it during dlclose(3).
Version 4 of the libsubid.so API/ABI [4] was released in Shadow 4.10,
which is newer than the versions shipped on RHEL 8 and Debian 10 [5],
and even that newer version had some problems [6]. Therefore, support
for older versions, with the relevant workarounds, is necessary.
Fortunately, the oldest that needs to be support is Shadow 4.9 because
that's when libsubid.so was introduced [7].
Note that SUBID_ABI_VERSION was only introduced with version 4 of the
libsubid.so API/ABI released in Shadow 4.10 [8]. The first release of
libsubid.so in Shadow 4.9 already had an ABI version of 3.0.0 [9], since
it was bumped a few times during development, so that's what's assumed
when SUBID_ABI_VERSION is absent.
This code doesn't set the public variables Prog and shadow_logfd that
older Shadow versions used to expect for logging, because from Shadow
4.9 onwards there's a separate function [4,10] to specify these. This
can be changed if there are libsubid.so versions in the wild that really
do need those public variables to be set.
Finally, ISO C99 is required because of the use of <stdbool.h> in the
libsubid.so API.
Some changes by Debarshi Ray.
[1] https://github.com/containers/storage/blob/main/pkg/idtools/idtools_supported.go
[2] https://man7.org/linux/man-pages/man8/ld.so.8.html
[3] Commit 6063eb27b9https://github.com/containers/toolbox/issues/821
[4] Shadow commit 32f641b207f6ddff
https://github.com/shadow-maint/shadow/commit/32f641b207f6ddffhttps://github.com/shadow-maint/shadow/issues/443
[5] https://packages.debian.org/source/buster/shadow
[6] Shadow commit 79157cbad87f42cd
https://github.com/shadow-maint/shadow/commit/79157cbad87f42cdhttps://github.com/shadow-maint/shadow/issues/465
[7] Shadow commit 0a7888b1fad613a0
https://github.com/shadow-maint/shadow/commit/0a7888b1fad613a0https://github.com/shadow-maint/shadow/issues/154
[8] Shadow commit 0c9f64140852e8d5
https://github.com/shadow-maint/shadow/commit/0c9f64140852e8d5https://github.com/shadow-maint/shadow/pull/449
[9] Shadow commit 3d670ba7ed58f910
https://github.com/shadow-maint/shadow/commit/3d670ba7ed58f910https://github.com/shadow-maint/shadow/issues/339
[10] Shadow commit 2b22a6909dba60d
https://github.com/shadow-maint/shadow/commit/2b22a6909dba60dhttps://github.com/shadow-maint/shadow/issues/325https://github.com/containers/toolbox/issues/1074
Signed-off-by: Martin Jackson <martjack@redhat.com>
Building Toolbx requires a C compiler [1], which defaults to GCC on
Fedora and CentOS Stream. It's good to explicitly require it, so that
it doesn't go missing from the build.
Showing the version of the C compiler is a big help when debugging weird
build problems involving the toolchain. A following commit will use CGO
to link to libsubid.so, which will only increase the relevance of the C
compiler.
[1] Commit c8aaed52c5https://github.com/containers/toolbox/pull/923https://github.com/containers/toolbox/pull/1218
