Ever since commit bafbbe81c9, the shell completions are generated
using the Toolbx binary, and the 'completion' sub-directory no longer
has any source code, but only the build scripts to invoke the Toolbx
binary to generate them. This is a good opportunity to simplify the
layout of this Git repository by reducing the number of sub-directories.
The file containing the Bash completions had to be renamed to avoid
colliding with the name of the Toolbx binary, since they are both
generated in the same sub-directory.
https://github.com/containers/toolbox/pull/1216
Skopeo was already listed, so it didn't make sense to leave out the
others. It's useful to give the user a heads-up to make it obvious what
the requirements are.
https://github.com/containers/toolbox/pull/1194
The bash-completion and fish dependencies were already optional - the
shell completions for Bash and fish won't be generated and installed if
they are absent; and there's no dependency required for Z shell. So the
install_completions build option wasn't reducing the dependency burden.
The build option was a way to disable the generation and installation of
the shell completions, regardless of whether the necessary dependencies
are present or not. The only use-case for this is when installing to a
non-system-wide prefix while hacking on Toolbox as a non-root user,
because the locations for the completions advertised by the shells' APIs
might not be accessible. Being able to disable the completions prevents
the installation from failing.
A different way of ensuring a smooth developer experience for a Toolbx
hacker is to offer a way to change the locations where the shell
completions are installed, which is necessary and beneficial for other
use-cases.
Z shell, unlike Bash's bash-completion.pc and fish's fish.pc, doesn't
offer an API to detect the location for the shell completions. This
means that Debian and Fedora use different locations [1, 2]. Namely,
/usr/share/zsh/vendor-completions and /usr/share/zsh/site-functions.
An option to specify the locations for the shell completions can
optimize the build, if there's an alternate API for the location that
doesn't involve using bash-completion.pc and fish.pc as build
dependencies. eg., Fedora provides the _tmpfilesdir RPM macro to
specify the location for vendor-supplied tmpfiles.d(5) files, which
makes it possible to avoid having systemd.pc as a build dependency [3].
Fallout from bafbbe81c9
[1] Debian zsh commit bf0a44a8744469b5
https://salsa.debian.org/debian/zsh/-/commit/bf0a44a8744469b5https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=620452
[2] https://src.fedoraproject.org/rpms/zsh/blob/f37/f/zsh.spec
[3] Fedora toolbox commit 9bebde5bb60f36e3
https://src.fedoraproject.org/rpms/toolbox/c/9bebde5bb60f36e3https://github.com/containers/toolbox/pull/1123https://github.com/containers/toolbox/pull/840
The subsequent commit will touch the POSIX shell implementation, and
hence ShellCheck needs to be run on it.
As long as the POSIX shell implementation is part of the Git repository,
ShellCheck needs to keep running on it, unless it causes some serious
problems. The ShellCheck test is very fast, and the reassurance and
mental peace that it provides is invaluable.
This reverts commit 8c1d441916.
https://github.com/containers/toolbox/pull/1094
The previous commit added a means to generating the completion scripts
and this one plugs that into the build system.
A new build option 'install_completions' has been introduced. Set to
'True' by default.
Completions for bash and fish use pkg-config for getting the preferred
install locations for the completions. If the packages are not
available, fallbacks are in-place.
The 'completion' subdir has been kept to work around the ideology of
Meson that does not allow creating/outputing files in subdirectories nor
using the output of custom_target() in install_data().
https://github.com/containers/toolbox/pull/840
It's only necessary to call 'systemd-tmpfiles --create' when building
and installing from source on the host operating system.
It's not needed when using a pre-built binary downstream package,
because:
* When 'meson install' is called as part of building the package,
that's not when the temporary files need to be created. They need
to be created when the binary package is later downloaded and
installed by the user.
* Downstream tools can sometimes handle it automatically. eg., on
Fedora, the systemd RPM installs a trigger that tells RPM to call
'systemd-tmpfiles --create' automatically when a tmpfiles.d snippet
is installed.
It's also not needed when installing inside a toolbox container because
the files that 'systemd-tmpfiles --create' is supposed to create are
meant to be on the host.
Downstream distributors set the DESTDIR environment variable when
building their packages. Therefore, it's used to detect when a
downstream package is being built.
Unfortunately, environment variables are messy and, generally, Meson
doesn't support accessing them inside its scripts [1]. Therefore, this
adds a spurious build-time dependency on systemd for downstream
distributors. However, that's probably not a big problem because all
supported downstream operating systems are already expected to use
systemd for the tmpfiles.d(5) snippets to work.
[1] https://github.com/mesonbuild/meson/issues/9https://github.com/containers/toolbox/issues/955
Commit 6c86cabbe5 changed the command line interface to behave
a lot similar to that of github.com/coreos/toolbox, which makes things
easier for those switching over from it.
However, it makes things confusing for the vast majority of users who
have never used coreos/toolbox. The Toolbox CLI aims to be friendly to
new users by being self-documenting and offering a smooth onboarding
experience. It's jarring to new users when 'toolbox', without any
commands specified, suggests that it needs to perform a big download.
It's difficult to document two different sets of CLIs, and if the
manuals don't mention the second behaviour, then it just leaves the
users even more confused.
Hence, it will be good to keep the migration path for coreos/toolbox
behind a build-time option, so that only those OS distributors who
truly need it may enable it without impacting others. Fortunately,
coreos/toolbox doesn't have any manuals, which means that there's no
need to conditionalize the documentation.
This commit merely adds the build-time option. Subsequent commits will
use this to actually conditionalize the code.
https://github.com/containers/toolbox/pull/951
Some downstream distributors like RHEL don't have patchelf(1). Relying
on patchelf(1) during the build will make it difficult for such
downstreams to distribute Toolbox.
Fortunately, the path of the dynamic linker (ie., PT_INTERP) is
hardcoded in the ABI specification of each architecture [1]. This means
that Toolbox's build system can keep it's own architecture to dynamic
linker mapping, and specify it during the build through the GNU ld
linker's --dynamic-linker flag, as opposed to using a tool like
patchelf(1) to change the path of the dynamic linker in the built
binary to the one inside /run/host. Currently, the list of
architectures covers the ones that Fedora builds for.
[1] https://sourceware.org/glibc/wiki/ABIListhttps://github.com/containers/toolbox/pull/942
The location for public shared libraries can change from one operating
system distribution to another. eg., while Fedora uses /usr/lib and
/usr/lib64, depending on the hardware architecture, Debian uses paths
like /usr/lib/x86_64-linux-gnu. Therefore, it's best not to assume
anything and ask the toolchain.
https://github.com/containers/toolbox/pull/923
The /usr/bin/toolbox binary is not only used to interact with toolbox
containers and images from the host. It's also used as the entry point
of the containers by bind mounting the binary from the host into the
container. This means that the /usr/bin/toolbox binary on the host must
also work inside the container, even if they have different operating
systems.
In the past, this worked perfectly well with the POSIX shell
implementation because it got intepreted by whichever /bin/sh was
available. However, the Go implementation, can run into ABI
compatibility issues because binaries built on newer toolchains aren't
meant to be run against older runtimes.
The previous approach [1] of restricting the versions of the glibc
symbols that are linked against isn't actually supported by glibc, and
breaks if the early process start-up code changes. This is seen in
glibc-2.34, which is used by Fedora 35 onwards, where a new version of
the __libc_start_main symbol [2] was added as part of some security
hardening:
$ objdump -T ./usr/bin/toolbox | grep GLIBC_2.34
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.34
__libc_start_main
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.34
pthread_detach
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.34
pthread_create
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.34
pthread_attr_getstacksize
This means that /usr/bin/toolbox binaries built against glibc-2.34 on
newer Fedoras fail to run against older glibcs in older Fedoras.
Another option is to make the host's runtime available inside the
toolbox container and ensure that the binary always runs against it.
Luckily, almost all supported containers have the host's /usr available
at /run/host/usr. This is exploited by embedding RPATHs or RUNPATHs to
/run/host/usr/lib and /run/host/usr/lib64 in the binary, and changing
the path of the dynamic linker (ie., PT_INTERP) to the one inside
/run/host.
Unfortunately, there can only be one PT_INTERP entry inside the
binary, so there must be a /run/host on the host too. Therefore, a
/run/host symbolic link is created on the host that points to the
host's /.
Based on ideas from Alexander Larsson and Ray Strode.
[1] Commit 6ad9c63180https://github.com/containers/toolbox/pull/534
[2] glibc commit 035c012e32c11e84
https://sourceware.org/git/?p=glibc.git;a=commit;h=035c012e32c11e84https://sourceware.org/bugzilla/show_bug.cgi?id=23323https://github.com/containers/toolbox/issues/821
When installing to a non-system-wide prefix as a non-root user, the
tmpfilesdir path defined by systemd might not be accessible. Overriding
the path helps to prevent the installation from failing.
https://github.com/containers/toolbox/pull/717
Shell Toolbox has been replaced by the Go implementation a quite while
ago. It is kept in the repository but is no longer actively developed.
There is no need to continue checking it with ShellCheck.
https://github.com/containers/toolbox/pull/733
The /usr/bin/toolbox binary is not only used to interact with toolbox
containers and images from the host. It's also used as the entry point
of the containers by bind mounting the binary from the host into the
container. This means that the /usr/bin/toolbox binary on the host must
also work inside the container, even if they have different operating
systems.
In the past, this worked perfectly well with the POSIX shell
implementation because it got intepreted by whichever /bin/sh was
available.
The Go implementation also mostly worked so far because it's largely
statically linked, with the notable exception of the standard C
library. However, recently glibc-2.32, which is used by Fedora 33
onwards, added a new version of the pthread_sigmask symbol [1] as part
of the libpthread removal project:
$ objdump -T /usr/bin/toolbox | grep GLIBC_2.32
0000000000000000 DO *UND* 0000000000000000 GLIBC_2.32
pthread_sigmask
This means that /usr/bin/toolbox binaries built against glibc-2.32 on
newer Fedoras pick up the latest version of the symbol and fail to run
against older glibcs in older Fedoras.
One way to fix this is to disable the use of any C code from Go by
using the CGO_ENABLED environment variable [2]. However, this can
negatively impact packages like "os/user" [3] and "net" [4], where the
more featureful glibc APIs will be replaced by more limited
equivalents written only in Go.
Instead, since glibc uses symbol versioning, it's better to tell the
Go toolchain to avoid linking against any symbols from glibc-2.32.
This was accomplished by a few linker tricks:
* The GNU ld linker's --wrap flag was used when building the Go code
to divert pthread_sigmask invocations from Go to another function
called __wrap_pthread_sigmask.
* A static library was added to provide this __wrap_pthread_sigmask
function, which forwards calls to the actual pthread_sigmask API in
glibc. This library itself was not linked with --wrap, and
specifies the latest permissible version of the pthread_sigmask
symbol from glibc for each architecture. Currently, the list of
architectures covers the ones that Fedora builds for.
* The Go cmd/link linker was switched to external mode [5]. This
ensures that the final object file containing all the Go code gets
linked to the standard C library and the wrapper static library by
the GNU ld linker for the --wrap flag to kick in.
Based on ideas from Ondřej Míchal.
[1] glibc commit c6663fee4340291c
https://sourceware.org/git/?p=glibc.git;a=commit;h=c6663fee4340291c
[2] https://golang.org/cmd/cgo/
[3] https://golang.org/pkg/os/user/
[4] https://golang.org/pkg/net/
[5] https://golang.org/src/cmd/cgo/doc.gohttps://github.com/containers/toolbox/issues/529
Meson doesn't support Go [1], so this is implemented by a custom target
that invokes 'go build' to generate the binary.
Unfortunately, when using Go modules, 'go build' insists on being
invoked in the same source directory where the go.mod file lives,
while Meson insists on using a build directory separate from the
corresponding source directory. This is addressed by using a build
script that goes into the source directory and then invokes 'go build'.
Currently, the Go code is only built when a Go implementation is found,
and even then, it's not installed. Non-technical end-users are supposed
to continue using the POSIX shell implementation until the Go version
is blessed as stable.
[1] https://github.com/mesonbuild/meson/issues/123https://github.com/containers/toolbox/pull/318
