This uses 'skopeo inspect' to get the size of the image on the registry,
which is usually less than the size of the image in a local
containers/storage image store after download (eg., 'podman images'),
because they are kept compressed on the registry. Skopeo >= 1.10.0 is
needed to retrieve the sizes [1].
However, this doesn't add a hard dependency on Skopeo to accommodate
size-constrained operating systems like Fedora CoreOS. If skopeo(1) is
missing or too old, then the size of the image won't be shown, but
everything else would continue to work as before.
Some changes by Debarshi Ray.
[1] Skopeo commit d9dfc44888ff71a6
https://github.com/containers/skopeo/commit/d9dfc44888ff71a6https://github.com/containers/skopeo/issues/641https://github.com/containers/toolbox/issues/752
Signed-off-by: Nieves Montero <nmontero@redhat.com>
Bind mounting the locations at runtime doesn't really have anything to
do with whether /run/host/etc is present inside the Toolbx container.
The only possible exception could have been /etc/machine-id, but it
isn't, because the bind mount is only performed if the source at
/run/host/etc/machine-id is present.
This is a historical mistake that has persisted for a long time, since,
in practice, /run/host/etc will almost always exist inside the Toolbx
container. It's time to finally correct it.
Fallout from 9436bbece0https://github.com/containers/toolbox/pull/1255
The --monitor-host option was added to the 'init-container' command in
commit 8b84b5e460 to accommodate Podman versions older than 1.2.0
that didn't have the '--dns none' and '--no-hosts' options for
'podman create'. These options are necessary to keep the Toolbx
container's /etc/resolv.conf and /etc/hosts files synchronized with
those of the host.
Note that Podman 1.2.0 was already available a few months before
commit 8b84b5e460 introduced the --monitor-host option. The
chances of someone using an older Podman back then was already on the
decline, and it's very unlikely that a container created with such a
Podman has survived till this date.
Commit b6b484fa79 raised the minimum required Podman version to
1.4.0, and made the '--dns none' and '--no-hosts' options a hard
requirement. The minimum required Podman version was again raised
recently in commit 8e80dd5db1 to 1.6.4. Therefore, these days,
there's no need to separately use the --monitor-host option of
'init-container' for newly created containers to indicate that the
Podman version wasn't older than 1.2.0.
Given all this, it's time to stop using the --monitor-host option of
'init-container', and assume that it's always set. The option is still
accepted to retain compatibility with existing Toolbx containers.
For containers that were created with the --monitor-host option, a
deprecation notice will be shown as:
$ podman start --attach CONTAINER
Flag --monitor-host has been deprecated, it does nothing
...
https://github.com/containers/toolbox/pull/617
So far the minimum required Podman version was 1.4.0, based on what used
to be available in RHEL 7. These days, Podman 1.6.4 is old enough to be
in RHEL 7.9. Hence it's time to bump the baseline.
https://github.com/containers/toolbox/pull/1253
Ever since commit bafbbe81c9, the shell completions are generated
while building Toolbx using the 'completion' command. This involves
running toolbox(1) itself, and hence validating the subordinate user and
group ID ranges.
Unfortunately, some build environments, like openSUSE's, don't have
subordinate ID ranges set up. Therefore, it's better to not validate
the subordinate ID ranges when generating the shell completions, since
they are generated by Cobra itself and subordinate ID ranges are not
involved at all.
Note that subordinate ID ranges may be needed when the generated shell
completions are actually used in interactive command line environments.
The shell completions invoke the hidden '__complete' command to get the
results that are presented to the user, and, if needed, the subordinate
ID ranges will continue to be used by podman(1) as part of that.
Some changes by Debarshi Ray.
https://github.com/containers/toolbox/issues/1246https://github.com/containers/toolbox/pull/1249
Having a separate convenience function reduces the indentation levels by
at least one, and sometimes two, and makes it easy to have more detailed
debug logs.
This will make the subsequent commit easier to read.
https://github.com/containers/toolbox/issues/1246
Ever since commit bafbbe81c9, the shell completions are generated
while building Toolbx using the 'completion' command. This involves
running toolbox(1) itself, and hence invoking 'podman version' to decide
if 'podman system migrate' is needed or not.
Unfortunately, some build environments, like Fedora's, are set up inside
a chroot(2) or systemd-nspawn(1) or similar, where 'podman version' may
not work because it does various things with namespaces(7) and clone(2)
that can, under certain circumstances, encounter an EPERM.
Therefore, it's better to avoid using podman(1) when generating the
shell completions, especially, since they are generated by Cobra itself
and podman(1) is not involved at all.
Note that podman(1) is needed when the generated shell completions are
actually used in interactive command line environments. The shell
completions invoke the hidden '__complete' command to get the results
that are presented to the user, and, if needed, 'podman system migrate'
will continue to be run as part of that.
This partially reverts commit f3e005d014
because podman(1) is now only an optional runtime dependency for the
system tests.
https://github.com/containers/podman/issues/17657
It's better not to use the global flag variables beyond the top-level
RunE functions, because sometimes the lower-level functions are re-used
from other files within the 'cmd' package. In this case,
createContainer(), and hence pullImage(), is also used in src/cmd/run.go
to implement the 'run' command. However, the 'run' command doesn't have
a --authflags option.
Since the default value of the flag is the zero value of the type, which
is a NOP in the code, it's likely that the code was still correct, but
it will be better to maintain some discipline here to highlight the
inputs needed by the lower-level functions. Otherwise, things can get
tangled up.
Fallout from ecd1ced719https://github.com/containers/toolbox/pull/1240
Just like /run/systemd/sessions makes it possible to get the seat for a
session ID, /run/systemd/users can make it possible to get the seat and
the session ID for a user's UID.
The absence of /run/systemd/users inside Toolbx containers isn't
currently causing problems for any use-case, but it seems very close
to the sort of things that were necessary to run a non-nested display
server from within a Toolbx container on a virtual terminal. It's not
impossible that in future some implementation details of the display
server stack may make /run/systemd/users necessary.
https://github.com/containers/toolbox/issues/992
Not having sd_booted(3) work inside Toolbx containers isn't currently
causing problems for any use-case. However, it did come in handy when
investigating how to run a non-nested display server from within a
Toolbx container on a virtual terminal, because it's necessary for
'systemd --user' to realize that the host operating system was booted
with systemd.
https://github.com/containers/toolbox/issues/992
Signed-off-by: Sebastian Wick <sebastian.wick@redhat.com>
Podman creates a private cgroup namespace for containers on cgroups v2
by default. The host's cgroupfs is mounted at /sys/fs/cgroup giving an
inconsistent view of the cgroups. Toolbx doesn't intend to provide a
segregated security domain. So, there is no need for a cgroup namespace
and Toolbx containers can just use the host's namespace.
Having a private cgroup namespace for containers isn't currently causing
problems for any use-case, but it did come in handy when investigating
how to run a non-nested display server from within a Toolbx container on
a virtual terminal. Since this requires a change to the 'podman create'
arguments, it's not going to have an effect on existing containers, and
re-creating containers is annoying for users. So, it might be better to
get ahead of the curve and do it preemptively.
https://github.com/containers/toolbox/issues/992
Signed-off-by: Sebastian Wick <sebastian.wick@redhat.com>
This is needed by display servers for creating udev device enumerators
that matches against tags.
https://github.com/containers/toolbox/issues/992
Signed-off-by: Jonas Ådahl <jadahl@gmail.com>
The URLs for the RHEL Toolbx images based on the Red Hat Universal Base
Images (or UBI) are a bit more complicated to construct, in comparison
to the URLs for Fedora's fedora-toolbox images. It's not enough to just
concatenate the registry, the image's basename and the release. Some
parts of the URL depend on the release's major number, which requires
custom code.
So far, the release's major number was hard coded to 8 since only RHEL 8
Toolbx containers were supported.
To support other RHEL major releases, it's necessary to have custom code
to construct the URLs for the Toolbx images.
https://github.com/containers/toolbox/issues/1065
On enterprise FreeIPA set-ups, the subordinate user and group IDs are
provided by SSSD's sss plugin for the GNU Name Service Switch (or NSS)
functionality of the GNU C Library. They are not listed in /etc/subuid
and /etc/subgid. Therefore, its necessary to use libsubid.so to check
the subordinate ID ranges.
The CGO interaction with libsubid.so is loosely based on 'readSubid' in
github.com/containers/storage/pkg/idtools [1].
However, unlike 'readSubid', this code considers the absence of any
range (ie., nRanges == 0) to be an error as well.
More importantly, this code uses dlopen(3) and friends to dynamically
load the symbols from libsubid.so, instead of linking to libsubid.so at
build-time and having the dependency noted in the /usr/bin/toolbox
binary. This is done because libsubid.so itself depends on several
other shared libraries, and indirect dependencies can't be influenced
by the RUNPATH [2] embedded in the /usr/bin/toolbox binary [3]. Hence,
when the binary is used inside Toolbx containers (eg., as the entry
point), those indirect dependencies won't be picked from the host's
runtime against which the binary was built. This can render the binary
useless due to ABI compatibility issues. Using dlopen(3) avoids this
problem, especially because libsubid.so is only used when running on the
host.
Care was taken to not load and link libsubid.so twice to separately
validate the subordinate ID ranges for the user and the group. Note
that libsubid_init() must be passed a FILE pointer for logging.
Otherwise, it will create it's own for logging, and there's no way to
close it during dlclose(3).
Version 4 of the libsubid.so API/ABI [4] was released in Shadow 4.10,
which is newer than the versions shipped on RHEL 8 and Debian 10 [5],
and even that newer version had some problems [6]. Therefore, support
for older versions, with the relevant workarounds, is necessary.
Fortunately, the oldest that needs to be support is Shadow 4.9 because
that's when libsubid.so was introduced [7].
Note that SUBID_ABI_VERSION was only introduced with version 4 of the
libsubid.so API/ABI released in Shadow 4.10 [8]. The first release of
libsubid.so in Shadow 4.9 already had an ABI version of 3.0.0 [9], since
it was bumped a few times during development, so that's what's assumed
when SUBID_ABI_VERSION is absent.
This code doesn't set the public variables Prog and shadow_logfd that
older Shadow versions used to expect for logging, because from Shadow
4.9 onwards there's a separate function [4,10] to specify these. This
can be changed if there are libsubid.so versions in the wild that really
do need those public variables to be set.
Finally, ISO C99 is required because of the use of <stdbool.h> in the
libsubid.so API.
Some changes by Debarshi Ray.
[1] https://github.com/containers/storage/blob/main/pkg/idtools/idtools_supported.go
[2] https://man7.org/linux/man-pages/man8/ld.so.8.html
[3] Commit 6063eb27b9https://github.com/containers/toolbox/issues/821
[4] Shadow commit 32f641b207f6ddff
https://github.com/shadow-maint/shadow/commit/32f641b207f6ddffhttps://github.com/shadow-maint/shadow/issues/443
[5] https://packages.debian.org/source/buster/shadow
[6] Shadow commit 79157cbad87f42cd
https://github.com/shadow-maint/shadow/commit/79157cbad87f42cdhttps://github.com/shadow-maint/shadow/issues/465
[7] Shadow commit 0a7888b1fad613a0
https://github.com/shadow-maint/shadow/commit/0a7888b1fad613a0https://github.com/shadow-maint/shadow/issues/154
[8] Shadow commit 0c9f64140852e8d5
https://github.com/shadow-maint/shadow/commit/0c9f64140852e8d5https://github.com/shadow-maint/shadow/pull/449
[9] Shadow commit 3d670ba7ed58f910
https://github.com/shadow-maint/shadow/commit/3d670ba7ed58f910https://github.com/shadow-maint/shadow/issues/339
[10] Shadow commit 2b22a6909dba60d
https://github.com/shadow-maint/shadow/commit/2b22a6909dba60dhttps://github.com/shadow-maint/shadow/issues/325https://github.com/containers/toolbox/issues/1074
Signed-off-by: Martin Jackson <martjack@redhat.com>
Ever since commit bafbbe81c9, the shell completions are generated
using the Toolbx binary, and the 'completion' sub-directory no longer
has any source code, but only the build scripts to invoke the Toolbx
binary to generate them. This is a good opportunity to simplify the
layout of this Git repository by reducing the number of sub-directories.
The file containing the Bash completions had to be renamed to avoid
colliding with the name of the Toolbx binary, since they are both
generated in the same sub-directory.
https://github.com/containers/toolbox/pull/1216
The errors should be propagated up the call chain either verbatim or by
wrapping them with all relevant context when necessary (as long as they
don't violate the API boundaries).
The errors should be logged only when there's a break in the upward
propagation, either because they need to be reformatted before being
shown to the user or because they would expose implementation details
that aren't part of the API contract. Not logging the errors in such
cases might make it difficult to debug problems later on.
https://github.com/containers/toolbox/pull/1202
CentOS Stream 9 has golang-1.19.2, while so far the 'unit tests' were
being run on Fedora 36, which only has golang-1.18.8.
This is a step towards testing on CentOS Stream 9.
https://github.com/containers/toolbox/pull/1199
Currently, if an image was copied with:
$ skopeo copy \
containers-storage:registry.fedoraproject.org/fedora-toolbox:36 \
containers-storage:localhost/fedora-toolbox:36
... or:
$ podman tag \
registry.fedoraproject.org/fedora-toolbox:36 \
localhost/fedora-toolbox:36
... then it would show up twice in 'list' with the same name, and in the
wrong order.
Either as:
$ toolbox list --images
IMAGE ID IMAGE NAME CREATED
2110dbbc33d2 localhost/fedora-toolbox:36 1 day...
e085805ade4a registry.access.redhat.com/ubi8/toolbox:latest 1 day...
2110dbbc33d2 localhost/fedora-toolbox:36 1 day...
70cbe2ce60ca registry.fedoraproject.org/fedora-toolbox:34 1 day...
... or as:
$ toolbox list --images
IMAGE ID IMAGE NAME CREATED
2110dbbc33d2 registry.fedoraproject.org/fedora-toolbox:36 1 day...
e085805ade4a registry.access.redhat.com/ubi8/toolbox:latest 1 day...
2110dbbc33d2 registry.fedoraproject.org/fedora-toolbox:36 1 day...
70cbe2ce60ca registry.fedoraproject.org/fedora-toolbox:34 1 day...
The correct output should be similar to 'podman images', and be sorted
in ascending order of the names:
$ toolbox list --images
IMAGE ID IMAGE NAME CREATED
2110dbbc33d2 localhost/fedora-toolbox:36 1 day...
e085805ade4a registry.access.redhat.com/ubi8/toolbox:latest 1 day...
70cbe2ce60ca registry.fedoraproject.org/fedora-toolbox:34 1 day...
2110dbbc33d2 registry.fedoraproject.org/fedora-toolbox:36 1 day...
The problem is that, in these situations, 'podman images --format json'
returns separate identical JSON collections for each copy of the image,
and all of those copies have multiple names:
[
{
"Id": "2110dbbc33d2",
...
"Names": [
"localhost/fedora-toolbox:36",
"registry.fedoraproject.org/fedora-toolbox:36"
],
...
},
{
"Id": "e085805ade4a",
...
"Names": [
"registry.access.redhat.com/ubi8/toolbox:latest"
],
...
},
{
"Id": "2110dbbc33d2",
...
"Names": [
"localhost/fedora-toolbox:36",
"registry.fedoraproject.org/fedora-toolbox:36"
],
...
}
{
"Id": "70cbe2ce60ca",
...
"Names": [
"registry.fedoraproject.org/fedora-toolbox:34"
],
...
},
]
The image objects need to be flattened to have only one unique name per
copy, but with the same ID, and then sorted to ensure the right order.
Note that the ordering was already broken since commit 2369da5d31,
which started using 'podman images --sort repository'. Podman can sort
by either the image's repository or tag, but not by the unified name,
which is what Toolbx needs. Therefore, even without copied images,
Toolbx really does need to sort the images itself.
Prior to commit 2369da5d31, the ordering was correct, but copied
images would only show up once.
Fallout from 2369da5d31
This reverts parts of commit 67e210378e.
https://github.com/containers/toolbox/issues/1043
This builds on top of commit 0465d78fd9034ce9.
The toolboxImage type has been renamed to Image and moved into the
podman package.
There is nothing Toolbx specific about the type - it represents any
image returned by 'podman images'. The images are only later filtered
for Toolbx images.
Secondly, having the Image type inside the podman package makes it
possible to encapsulate the unmarshalling of the JSON within the package
without exposing the raw JSON to outside consumers. This is desirable
because the unmarshalling involves tracking changes in the JSON output
by different Podman versions, and it's better to limit such details to
the podman package.
https://github.com/containers/toolbox/pull/1190
It's better to avoid single letter variables in general, because they
are so hard to grep for.
This will make the subsequent commit easier to read.
https://github.com/containers/toolbox/pull/1190
This builds on top of commit e772207831.
Currently, the JSON from 'podman images --format json' gets unmarshalled
into a []map[string]interface{} in podman.GetImages, where the maps in
the slice represent images. Each map is then marshalled back into JSON
and then again unmarshalled into a toolboxImage type.
This is wasteful. The toolboxImage type already implements the
json.Unmarshaler interface [1], since commit e772207831. Hence,
the entire JSON from 'podman images --format json' can be directly
unmarshalled into a slice of toolboxImages without involving the
[]map[string]interface{}.
A subsequent commit will move the toolboxImage type into the podman
package to more tightly encapsulate the unmarshalling of the JSON. So,
as an intermediate step in that direction, the podman.GetImages function
has been temporarily changed to return the entire JSON.
[1] https://pkg.go.dev/encoding/json#Unmarshalerhttps://github.com/containers/toolbox/pull/1190
Otherwise, there's so much spew from 'go test', including the successful
tests, that the actual failures don't stand out.
Note that, the different steps involved in building the code base are a
lot more interdependent on each other. Hence, some extra verbosity
can help understand what caused a build failure on non-interactive build
environments. In contrast, the runtime outputs from each test case are
a lot more isolated and independent from one another. The additional
verbosity from successful tests doesn't really help understand why a
particular test failed.
https://github.com/containers/toolbox/pull/1186
Currently, only a so-called high-confidence subset of the default checks
in 'go vet' are being run by 'go test' [1]. Since 'go vet' is part of
the core Go tools, it's worth trying to use more of it. After all,
golangci-lint, which is currently being run through a GitHub Action,
is running the default 'go vet' checks as one of its linters [2].
It's good to have as much of the testing wrapped inside 'meson test', as
possible, because it's easier to run locally and on other non-GitHub CI
environments like those of downstream distributors.
[1] https://pkg.go.dev/cmd/go/internal/test
[2] https://golangci-lint.run/usage/linters/https://golangci-lint.run/usage/linters/#govethttps://github.com/containers/toolbox/pull/1186
In the past, before commit d323143c46, there was either had a
dummy 'return' statement or a self-documenting 'panic' that said that
the code should not be reached. Since neither golangci-lint nor
'go vet' likes those, a comment is the only option left.
Note that the core Go tools like 'go vet' [1], but also 'go lint' [2],
explicitly don't intend to add fine-grained configuration options,
including inline directives or pragmas, to silence specific warnings.
That's something golangci-lint offers [3], to the extent that it's
supported by its linters [4]. However, golangci-lint also uses 'go vet'
as one of those linters, so it's the same problem all over again.
Therefore, between the two extremes of leaving the code difficult to
read and using a very big hammer to disable a needlessly big chuck of
'go vet', a comment is the least worst option.
[1] https://github.com/golang/go/issues/17058https://github.com/golang/go/issues/18432
[2] https://github.com/golang/lint/issues/263
[3] https://golangci-lint.run/usage/false-positives/
[4] https://golangci-lint.run/usage/linters/
Fallout from d323143c46https://github.com/containers/toolbox/pull/1185
