Commit graph

224 commits

Author SHA1 Message Date
Debarshi Ray
569b4df24d test/system: Test the resource limits
The following caveats must be noted:

  * Podman sets the Toolbx container's soft limit for the maximum number
    of open file descriptors to the host's hard limit, which is often
    greater than the host's soft limit [1].

  * The ulimit(1) options -P, -T, -b, and -k don't work on Fedora 38
    because the corresponding resource arguments for getrlimit(2) are
    absent from the operating system.  These are RLIMIT_NPTS,
    RLIMIT_PTHREAD, RLIMIT_SBSIZE and RLIMIT_KQUEUES respectively.

[1] https://github.com/containers/podman/issues/17681

https://github.com/containers/toolbox/issues/213
2023-07-04 15:34:21 +02:00
Debarshi Ray
ea91335ebb test/system: Limit the scope of temporary files used by a single test
BATS_TMPDIR is the base directory used by Bats for all temporary files
and directories, and BATS_TEST_TMPDIR is unique to each test [1].  It's
better to limit the scope of the tests' temporary files as much as
possible to avoid unexpected collisions with Bats' own internal
temporary files.

[1] https://bats-core.readthedocs.io/en/stable/writing-tests.html

https://github.com/containers/toolbox/pull/1327
2023-06-30 20:45:48 +02:00
Debarshi Ray
c43cf5d763 test/system: Test that interprocess communication works
Note that 'run --keep-empty-lines' counts the trailing newline on the
last line as a separate line.

https://github.com/containers/toolbox/pull/1326
2023-06-30 20:30:48 +02:00
Debarshi Ray
41215cf82e test/system: Test that networking works
Note that 'run --keep-empty-lines' counts the trailing newline on the
last line as a separate line.

https://github.com/containers/toolbox/pull/1325
2023-06-30 19:53:31 +02:00
Debarshi Ray
d7d2fd90cb test/system: Remove stray newline
https://github.com/containers/toolbox/pull/1325
2023-06-30 14:45:21 +02:00
Debarshi Ray
4322824061 test/system: Fix reading the os-release(5) VERSION_ID on Ubuntu
The current approach of extracting the VERSION_ID field from
os-release(5) assumes that the value is not quoted.  There's no
guarantee that this will be the case.  It only happens to be so on
Fedora by chance, and is different on Ubuntu:
  $ cat /etc/os-release
  ...
  VERSION_ID="22.04"
  ...

This means that "22.04", including the double quotes, is read as the
value of VERSION_ID on Ubuntu, not 22.04.  This is wrong because this
value can't be used as is in image and container names.  There's no
image called quay.io/toolbx/ubuntu-toolbox:"22.04" and double quotes are
not allowed in container names.

Instead, use the same approach as profile.d/toolbox.sh and the old POSIX
shell implementation that doesn't rely on the quoting of the
os-release(5) values.

Fallout from b27795a03e

https://github.com/containers/toolbox/pull/1320
2023-06-23 13:58:42 +02:00
Debarshi Ray
62c31ca8ea test/system: Fix reading the os-release(5) ID on Ubuntu
The current approach of selecting all the os-release(5) fields that have
'ID' in their name (eg., ID, VERSION_ID, PLATFORM_ID, VARIANT_ID, etc.)
and then picking the first one, assumes that the ID field will always be
placed above the others in os-release(5).  There's no guarantee that
this will be the case.  It only happens to be so on Fedora by chance,
and is different on Ubuntu:
  $ cat /etc/os-release
  ...
  VERSION_ID="22.04"
  ...
  ID=ubuntu
  ID_LIKE=debian
  ...

This means that "22.04" is read as the value of ID on Ubuntu, which is
clearly wrong.

Instead, use the same approach as profile.d/toolbox.sh and the old POSIX
shell implementation that doesn't rely on the order of the os-release(5)
fields.

Fallout from 54a2ca1ead

https://github.com/containers/toolbox/pull/1320
2023-06-23 13:27:07 +02:00
Matthias Clasen
2c09606603 test/system: Clarify the use of Git submodules
We wasted some time trying to get the tests running locally, when all we
were missing were the 'git submodule ...' commands.

Add some more obvious hints about this possible stumbling block.

Note that Bats cautions against printing outside the @test, setup* or
teardown* functions [1].  In this case, doing so leads to the first line
of the error output going missing, when using the pretty formatter for
human consumption:

  $ bats --formatter pretty ./test/system
   ✗ setup_suite
     Forgot to run 'git submodule init' and 'git submodule update' ?
     bats warning: Executed 1 instead of expected 191 tests

  191 tests, 1 failure, 190 not run

[1] https://bats-core.readthedocs.io/en/stable/writing-tests.html

https://github.com/containers/toolbox/pull/1298

Signed-off-by: Matthias Clasen <mclasen@redhat.com>
2023-06-21 12:34:08 +02:00
Debarshi Ray
7a387dcc8b test/system: Simplify running a subset of the tests with Bats >= 1.7.0
The 000-setup.bats and 999-teardown.bats files were added [1] at a time
when Bats didn't offer any hooks for suite-wide setup and teardown.

That changed in Bats 1.7.0, which introduced the setup_suite and
teardown_suite hooks.  These hooks make it easier to run a subset of the
tests, which is a good thing.

In the past, to run a subset of the tests, one had to do:
  $ bats ./test/system/000-setup.bats ./test/system/002-help.bats \
      ./test/system/999-teardown.bats

Now, one only has to do:
  $ bats ./test/system/002-help.bats

Commit e22a82fec8 already added a dependency on Bats >= 1.7.0.
Therefore, it should be exploited wherever possible to simplify things.

[1] Commit 54a2ca1ead
    https://github.com/containers/toolbox/issues/751

[2] Bats commit fb467ec3f04e322a
    https://github.com/bats-core/bats-core/issues/39
    https://bats-core.readthedocs.io/en/stable/writing-tests.html

https://github.com/containers/toolbox/pull/1317
2023-06-21 09:07:29 +02:00
Debarshi Ray
c37c5238dd test/system: Re-align
All the other Bats files for the system tests are indented by two
spaces, not four.

This will make the subsequent commit easier to read.

https://github.com/containers/toolbox/pull/1317
2023-06-21 09:07:25 +02:00
Nieves Montero
0a1417799a test/system: Fix warnings by specifying the minimum needed Bats version
Bats 1.7.0 emits a warning if a feature that is only available starting
from a certain version of Bats onwards is used without specifying that
version [1]:
  BW02: Using flags on `run` requires at least BATS_VERSION=1.5.0. Use
    `bats_require_minimum_version 1.5.0` to fix this message.
        (from function `bats_warn_minimum_guaranteed_version' in file
           /usr/lib/bats-core/warnings.bash, line 32,
         from function `run' in file
           /usr/lib/bats-core/test_functions.bash, line 227,
         in test file test/system/001-version.bats, line 27)

Note that bats_require_minimum_version itself is only available from
Bats 1.7.0 [2].  Hence, even though the specific feature here (using
flags on 'run') only requires Bats >= 1.5.0, in practice Bats >= 1.7.0
is needed.  Fortunately, commit e22a82fec8 already added a
dependency on Bats >= 1.7.0.  So, there's nothing to worry about.

[1] Bats commit 82002bb6c1a5c418
    https://github.com/bats-core/bats-core/issues/556
    https://bats-core.readthedocs.io/en/stable/warnings/BW02.html

[2] Bats commit 71d6b71cebc3d32b
    https://github.com/bats-core/bats-core/issues/556
    https://bats-core.readthedocs.io/en/stable/warnings/BW02.html

https://github.com/containers/toolbox/pull/1315
2023-06-13 12:48:11 +02:00
Debarshi Ray
bc067f12d6 test/system: Simplify the line count checks by relying on Bats >= 1.7.0
Commit e22a82fec8 already added a dependency on Bats >= 1.7.0,
which is present on Fedora >= 36.  Therefore, it should be exploited
wherever possible to simplify things.

Earlier, when the line counts were checked only with Bats >= 1.7.0,
there was a need to separately check the whole standard error and
output streams with 'assert_output' for the tests to be useful on
Fedora 35, which only had Bats 1.5.0.  Now that the line counts are
being checked unconditionally, there's no need for that anymore.

Note that bats_require_minimum_version itself is only available from
Bats 1.7.0 [1].

[1] Bats commit 71d6b71cebc3d32b
    https://github.com/bats-core/bats-core/issues/556
    https://bats-core.readthedocs.io/en/stable/warnings/BW02.html

https://github.com/containers/toolbox/pull/1314
2023-06-13 12:43:14 +02:00
Debarshi Ray
2ee82affeb pkg/utils: Offer built-in support for Arch Linux
This allows using the 'distro' option to create and enter Arch Linux
containers.  Due to Arch's rolling-release model, the 'release' option
isn't required.  If 'release' is used, the accepted values are 'latest'
and 'rolling'.

https://github.com/containers/toolbox/pull/1311
2023-06-12 22:26:46 +02:00
Debarshi Ray
b6ca18ead1 pkg/utils, test/system: Use the Ubuntu images from quay.io/toolbx/...
https://github.com/containers/toolbox/pull/1306
2023-06-08 15:35:44 +02:00
Ievgen Popovych
a84a358b3b pkg/utils, test/system: Offer built-in support for Ubuntu
This allows using the --distro and --release options to create and enter
Ubuntu containers.

Note that Ubuntu 4.10 was the first ever Ubuntu release [1].  Hence,
values older than that are not permitted for the --release option.

Some changes by Debarshi Ray.

[1] https://wiki.ubuntu.com/Releases

https://github.com/containers/toolbox/pull/483
https://github.com/containers/toolbox/pull/1284

Signed-off-by: Ievgen Popovych <jmennius@gmail.com>
2023-03-28 00:14:19 +02:00
Debarshi Ray
ddd1221d44 test/system: Test 'run' with a RHEL Toolbx container
https://github.com/containers/toolbox/pull/1283
2023-03-27 13:44:38 +02:00
Debarshi Ray
7e9d3918f5 test/system: Test 'run' with a Fedora Toolbx container
It's good to ensure that older well-known Toolbx images continue to work
with newer versions of 'run'.

https://github.com/containers/toolbox/pull/1283
2023-03-27 13:41:51 +02:00
Debarshi Ray
c56b74921a test/system: Test 'create' with a RHEL Toolbx container
https://github.com/containers/toolbox/pull/1282
2023-03-26 10:49:11 +02:00
Debarshi Ray
23e636bb2a test/system: Remove redundant quotes
https://github.com/containers/toolbox/pull/1282
2023-03-26 10:37:10 +02:00
Debarshi Ray
9f4d906e39 test/system: Remove redundant comment
It's quite obvious what the corresponding code is doing, and it isn't
any harder to understand than the rest of the code that's not commented.

https://github.com/containers/toolbox/pull/1282
2023-03-26 10:36:59 +02:00
Debarshi Ray
06245dbf52 test/system: Synchronize with the parseRelease() unit tests
https://github.com/containers/toolbox/pull/1280
2023-03-24 19:57:33 +01:00
Debarshi Ray
26b031f686 test/system: Split the tests for parsing releases
This will make it easier to find out exactly which test failed.

https://github.com/containers/toolbox/pull/1280
2023-03-23 18:09:45 +01:00
Debarshi Ray
44cbdb1f18 test/system: Ensure that both non-Toolbx containers & images are skipped
https://github.com/containers/toolbox/pull/1273
2023-03-22 12:05:06 +01:00
Debarshi Ray
11f5cde45e test/system: Check the number of images after pulling them
This is a quick sanity check with 'podman images' to ensure that all the
images are in place before running 'list'.  Other tests already do this,
so this change makes these two tests consistent with the rest.

https://github.com/containers/toolbox/pull/1273
2023-03-22 10:40:57 +01:00
Debarshi Ray
63aeabe86b test/system: Factor out some repeated code
https://github.com/containers/toolbox/pull/1273
2023-03-22 10:08:47 +01:00
Debarshi Ray
55bf3d06a8 test/system: Use fully qualified image names to check the 'list' output
https://github.com/containers/toolbox/pull/1273
2023-03-22 09:45:23 +01:00
Debarshi Ray
164c9eff36 test/system: Test 'list' with a RHEL Toolbx image
This restores parts of commit e09de9f3e5.

https://github.com/containers/toolbox/pull/1273
2023-03-22 09:18:04 +01:00
Debarshi Ray
aa9da193c6 test/system: Test 'list' with an old Fedora Toolbx image
It's good to ensure that older well-known Toolbx images continue to work
with newer versions of 'list'.

https://github.com/containers/toolbox/pull/1278
2023-03-22 09:16:58 +01:00
Debarshi Ray
571cfc1296 test/system: Use the same convenience function as elsewhere
Fallout from b27795a03e

https://github.com/containers/toolbox/pull/1278
2023-03-22 09:16:52 +01:00
Debarshi Ray
96d629c21f test/system: Test 'list' with the default Toolbx image
This is the 'simple' case of having a well-known Toolbx image (ie.,
not a copy, not an image without a name, not a non-Toolbx image).  It's
good to ensure that the default image works as expected with 'list'
before moving on to more complex scenarios.

https://github.com/containers/toolbox/pull/1278
2023-03-22 09:16:44 +01:00
Debarshi Ray
20fa122820 test/system: Test 'list' both with and without --images
https://github.com/containers/toolbox/pull/1278
2023-03-22 09:16:37 +01:00
Debarshi Ray
6890f5dc8d test/system: Rename a 'list' test for clarity
A subsequent commit will add a variant of this test that uses 'list'
without the '--images' option.

https://github.com/containers/toolbox/pull/1278
2023-03-22 09:16:29 +01:00
Debarshi Ray
9cae66ddd8 test/system: Check the standard error & output streams separately
https://github.com/containers/toolbox/pull/1278
2023-03-22 09:16:20 +01:00
Debarshi Ray
e22a82fec8 test/system: Simplify the line count checks by relying on Bats >= 1.7.0
Fedora 35 was stuck with Bats 1.5.0.  However, it reached End of Life on
13th December 2022 and was dropped from the CI [1].  Fedora 36 is the
oldest supported Fedora and it has Bats 1.8.2.

Therefore, there's no need to retain compatibility with Bats < 1.7.0.

Note that bats_require_minimum_version itself is only available from
Bats 1.7.0 [2].

[1] Commit 419e4e8cd9
    https://github.com/containers/toolbox/pull/1237

[2] Bats commit 71d6b71cebc3d32b
    https://github.com/bats-core/bats-core/issues/556
    https://bats-core.readthedocs.io/en/stable/warnings/BW02.html

https://github.com/containers/toolbox/pull/1273
2023-03-21 19:20:21 +01:00
Debarshi Ray
08e40e666e test/system: Shorten the names of the tests and use consistent wording
Currently, some of the names of the tests were too long, and had
inconsistent and verbose wording.  This made it difficult to look at
them and get a gist of all the scenarios being tested.  The names are
like headings.  They shouldn't be too long, should capture the primary
objective of the test and be consistent in their wording.

https://github.com/containers/toolbox/pull/1276
2023-03-21 19:11:20 +01:00
Debarshi Ray
5b8a7a00c0 test/system: Use long options, instead of their shorter aliases
The long options are easier to grep(1) for in the sources than their
shorter aliases.

https://github.com/containers/toolbox/pull/1276
2023-03-21 19:11:13 +01:00
Debarshi Ray
f364812781 test/system: Shorten the names of the tests and use consistent wording
Currently, some of the names of the tests were too long, and had
inconsistent and verbose wording.  This made it difficult to look at
them and get a gist of all the scenarios being tested.  The names are
like headings.  They shouldn't be too long, should capture the primary
objective of the test and be consistent in their wording.

https://github.com/containers/toolbox/pull/1271
2023-03-20 18:28:10 +01:00
Debarshi Ray
edd82d7e6e test/system: Style fixes
Fallout from e5bab51bca

https://github.com/containers/toolbox/pull/1271
2023-03-20 18:28:07 +01:00
Debarshi Ray
c6760299d2 pkg/utils, test/system: Add some more tests for parsing RHEL releases
https://github.com/containers/toolbox/pull/1262
2023-03-16 22:01:45 +01:00
Debarshi Ray
e01927bfdc pkg/utils, test/system: Add some more tests for parsing Fedora releases
https://github.com/containers/toolbox/pull/1262
2023-03-16 22:01:45 +01:00
Debarshi Ray
a4cc791f39 test/system: Synchronize with the parseRelease() unit tests
https://github.com/containers/toolbox/pull/1266
2023-03-16 18:41:24 +01:00
Debarshi Ray
1bd6cf17ce test/system: Split the tests for parsing releases
This will make it easier to find out exactly which test failed.

https://github.com/containers/toolbox/pull/1266
2023-03-16 18:31:12 +01:00
Debarshi Ray
e5bab51bca test/system: Shorten the names of the tests and use consistent wording
Currently, some of the names of the tests were too long, and had
inconsistent and verbose wording.  This made it difficult to look at
them and get a gist of all the scenarios being tested.  The names are
like headings.  They shouldn't be too long, should capture the primary
objective of the test and be consistent in their wording.

https://github.com/containers/toolbox/pull/1265
2023-03-16 17:57:46 +01:00
Debarshi Ray
068d22a199 test/system: Use long options, instead of their shorter aliases
The long options are easier to grep(1) for in the sources than their
shorter aliases.

https://github.com/containers/toolbox/pull/1265
2023-03-16 17:37:16 +01:00
Debarshi Ray
052666840a Fix spelling mistakes
The noun is 'setup' and the verb is 'set up'.  Similarly 'lookup' and
'look up'.

Original patch from Erik Sjölund for Podman [1,2].

[1] Podman commit aa4279ae151fa9df
    https://github.com/containers/podman/commit/aa4279ae151fa9df
    https://github.com/containers/podman/pull/14658

[2] Podman commit 2827140907255ed0
    https://github.com/containers/podman/commit/2827140907255ed0
    https://github.com/containers/podman/pull/14659

https://github.com/containers/toolbox/pull/1223
2023-01-31 15:53:15 +01:00
Debarshi Ray
936a157117 doc, test/system: Fix the titles of the manuals
Currently, the titles of the manuals are rendered with a pair of empty
parentheses and no section title:
  toolbox(1)()                                              toolbox(1)()

  NAME
         toolbox - Tool for containerized command line environments...

However, they should be:
  toolbox(1)             General Commands Manual              toolbox(1)

  NAME
         toolbox - Tool for containerized command line environments...

This is because the troff generated by go-md2man from Markdown has a
faulty invocation of the .TH macro [1]:
  .nh
  .TH toolbox(1)
  .SH NAME
  .PP
  toolbox - Tool for containerized command line environments on Linux

It should be:
  .nh
  .TH toolbox 1
  .SH NAME
  .PP
  toolbox - Tool for containerized command line environments on Linux

Original patch from Andrew Denton for Podman [2].

[1] https://www.gnu.org/software/groff/manual/groff.html

[2] Podman commit 63c779a857b55b00
    https://github.com/containers/podman/pull/15621

https://github.com/containers/toolbox/pull/1210
2023-01-12 18:45:00 +01:00
Debarshi Ray
cac6b25f19 test/system: Tighten the regular expression used to detect the image ID
An OCI image's identifier (or ID) is a SHA-256 hash [1] whose length is
64 hexadecimal digits.  The ID can be abbreviated to a smaller number of
digits but the minimum is 6.  The regular expression being used to parse
the image ID wasn't checking the number of digits, and would even accept
an empty string as a valid ID, which was clearly wrong.

The format of a image reference is defined in:
https://github.com/moby/moby/blob/master/image/spec/v1.1.md
https://github.com/distribution/distribution/blob/main/reference/reference.go
https://github.com/distribution/distribution/blob/main/reference/regexp.go

Fallout from c6c2e426e0

[1] https://en.wikipedia.org/wiki/SHA-2

https://github.com/containers/toolbox/pull/1209
2023-01-12 14:44:05 +01:00
Debarshi Ray
65194bd45e test/system: Avoid false positives from codespell
Otherwise codespell would complain:
  : @test "create: Try to create a container with invalid custom name...
  >   run $TOOLBOX -y create "ßpeci@l.Nam€"
  :
  ./test/system/101-create.bats:57: Nam ==> Name

CentOS Stream 9 has codespell-2.2.1, while so far the 'unit tests' were
being run on Fedora 36, which only has codespell-2.1.0.

This is a step towards testing on CentOS Stream 9.

https://github.com/containers/toolbox/pull/1200
2022-12-14 23:09:21 +01:00
Debarshi Ray
5f5f7b6aab test/system: Fix spelling mistake using codespell
CentOS Stream 9 has codespell-2.2.1, while so far the 'unit tests' were
being run on Fedora 36, which only has codespell-2.1.0.

This is a step towards testing on CentOS Stream 9.

Fallout from ecd1ced719

https://github.com/containers/toolbox/pull/1200
2022-12-14 23:09:21 +01:00
Martin Krajnak
05a062f8c9 test/system: Ensure that copied images are clearly identified
Note that 'run --keep-empty-lines' counts the trailing newline on the
last line as a separate line.

Until Bats 1.7.0, 'run --keep-empty-lines' had a bug where even when a
command produced no output, it would report a line count of one [1] due
to a stray line feed character.  This needs to be conditionalized, since
Fedora 35 has Bats 1.5.0.

[1] https://github.com/bats-core/bats-core/issues/573

https://github.com/containers/toolbox/issues/1043
2022-12-12 21:49:21 +01:00
Debarshi Ray
6aab0a6175 Unbreak sorting and clearly identify copied images in 'list'
Currently, if an image was copied with:
  $ skopeo copy \
      containers-storage:registry.fedoraproject.org/fedora-toolbox:36 \
      containers-storage:localhost/fedora-toolbox:36

... or:
  $ podman tag \
      registry.fedoraproject.org/fedora-toolbox:36 \
      localhost/fedora-toolbox:36

... then it would show up twice in 'list' with the same name, and in the
wrong order.

Either as:
  $ toolbox list --images
  IMAGE ID      IMAGE NAME                                      CREATED
  2110dbbc33d2  localhost/fedora-toolbox:36                     1 day...
  e085805ade4a  registry.access.redhat.com/ubi8/toolbox:latest  1 day...
  2110dbbc33d2  localhost/fedora-toolbox:36                     1 day...
  70cbe2ce60ca  registry.fedoraproject.org/fedora-toolbox:34    1 day...

... or as:
  $ toolbox list --images
  IMAGE ID      IMAGE NAME                                      CREATED
  2110dbbc33d2  registry.fedoraproject.org/fedora-toolbox:36    1 day...
  e085805ade4a  registry.access.redhat.com/ubi8/toolbox:latest  1 day...
  2110dbbc33d2  registry.fedoraproject.org/fedora-toolbox:36    1 day...
  70cbe2ce60ca  registry.fedoraproject.org/fedora-toolbox:34    1 day...

The correct output should be similar to 'podman images', and be sorted
in ascending order of the names:
  $ toolbox list --images
  IMAGE ID      IMAGE NAME                                      CREATED
  2110dbbc33d2  localhost/fedora-toolbox:36                     1 day...
  e085805ade4a  registry.access.redhat.com/ubi8/toolbox:latest  1 day...
  70cbe2ce60ca  registry.fedoraproject.org/fedora-toolbox:34    1 day...
  2110dbbc33d2  registry.fedoraproject.org/fedora-toolbox:36    1 day...

The problem is that, in these situations, 'podman images --format json'
returns separate identical JSON collections for each copy of the image,
and all of those copies have multiple names:
  [
    {
      "Id": "2110dbbc33d2",
      ...
      "Names": [
        "localhost/fedora-toolbox:36",
        "registry.fedoraproject.org/fedora-toolbox:36"
      ],
      ...
    },
    {
      "Id": "e085805ade4a",
      ...
      "Names": [
        "registry.access.redhat.com/ubi8/toolbox:latest"
      ],
      ...
    },
    {
      "Id": "2110dbbc33d2",
      ...
      "Names": [
        "localhost/fedora-toolbox:36",
        "registry.fedoraproject.org/fedora-toolbox:36"
      ],
      ...
    }
    {
      "Id": "70cbe2ce60ca",
      ...
      "Names": [
        "registry.fedoraproject.org/fedora-toolbox:34"
      ],
      ...
    },
  ]

The image objects need to be flattened to have only one unique name per
copy, but with the same ID, and then sorted to ensure the right order.

Note that the ordering was already broken since commit 2369da5d31,
which started using 'podman images --sort repository'.  Podman can sort
by either the image's repository or tag, but not by the unified name,
which is what Toolbx needs.  Therefore, even without copied images,
Toolbx really does need to sort the images itself.

Prior to commit 2369da5d31, the ordering was correct, but copied
images would only show up once.

Fallout from 2369da5d31

This reverts parts of commit 67e210378e.

https://github.com/containers/toolbox/issues/1043
2022-12-12 21:49:21 +01:00
Debarshi Ray
51eccd3da5 test/system: Test 'rmi' with an image
https://github.com/containers/toolbox/pull/1195
2022-12-12 20:59:52 +01:00
Debarshi Ray
bbdf4ddb63 test/system: Test 'rmi' with an image and its copy
https://github.com/containers/toolbox/issues/1043
2022-12-12 20:59:52 +01:00
Martin Krajnak
d5daa7167e test/system: Add a helper to pull the default image and copy it
This will be used in subsequent commits to test the handling of such
copied images in 'toolbox list' and 'toolbox rmi'.

https://github.com/containers/toolbox/issues/1043
2022-12-12 20:59:47 +01:00
Debarshi Ray
a0d4c957b3 test/system: Test 'rmi' with an image without a name
https://github.com/containers/toolbox/pull/1195
2022-12-12 15:13:26 +01:00
Debarshi Ray
e25ab310fa test/system: Test 'rmi --all' with an image without a name
https://github.com/containers/toolbox/pull/1195
2022-12-12 15:13:26 +01:00
Debarshi Ray
8a37c08780 test/system: Test 'rmi --all' without any images
https://github.com/containers/toolbox/pull/1195
2022-12-12 15:13:26 +01:00
Debarshi Ray
210985ecd1 test/system: Ensure that error messages go to the standard error stream
Currently, there's no way to get assert_line to use the stderr_lines
array [1].  This is worked around by assigning stderr_lines to the
'lines' array.

[1] https://github.com/bats-core/bats-assert/issues/42

https://github.com/containers/toolbox/pull/1195
2022-12-12 15:13:26 +01:00
Debarshi Ray
26ed682cd1 test/system: Keep empty lines to prevent missing and spurious newlines
https://github.com/containers/toolbox/pull/1195
2022-12-12 15:13:26 +01:00
Debarshi Ray
303c7ae99a test/system: Don't ignore copies when counting images
If an image was copied with:
  $ skopeo copy \
      containers-storage:registry.fedoraproject.org/fedora-toolbox:36 \
      containers-storage:localhost/fedora-toolbox:36

... or:
  $ podman tag \
      registry.fedoraproject.org/fedora-toolbox:36 \
      localhost/fedora-toolbox:36

... then the image ID is only showed once in 'podman images --quiet',
not twice.

A subsequent commit will use this to write tests to ensure that copied
images are correctly handled.

https://github.com/containers/toolbox/issues/1043
2022-12-12 15:04:42 +01:00
Debarshi Ray
5e8446971c test/system: Use long options, instead of their shorter aliases
The long options are easier to grep(1) for in the sources than their
shorter aliases.

https://github.com/containers/toolbox/pull/1197
2022-12-12 15:00:46 +01:00
Debarshi Ray
f17a632f9a test/system: Check the line count in the standard error & output streams
Note that 'run --keep-empty-lines' counts the trailing newline on the
last line as a separate line.

Until Bats 1.7.0, 'run --keep-empty-lines' had a bug where even when a
command produced no output, it would report a line count of one [1] due
to a stray line feed character.  This needs to be conditionalized, since
Fedora 35 has Bats 1.5.0.

[1] https://github.com/bats-core/bats-core/issues/573

https://github.com/containers/toolbox/pull/1192
2022-12-08 23:08:47 +01:00
Debarshi Ray
89385e12b5 test/system: Ensure that non-error messages go to the standard output
https://github.com/containers/toolbox/pull/1192
2022-12-08 22:33:35 +01:00
Debarshi Ray
4d1cc5b39b test/system: Test the order in 'list' for images with & without names
Note that 'run --keep-empty-lines' counts the trailing newline on the
last line as a separate line.

Until Bats 1.7.0, 'run --keep-empty-lines' had a bug where even when a
command produced no output, it would report a line count of one [1] due
to a stray line feed character.  This needs to be conditionalized, since
Fedora 35 has Bats 1.5.0.

[1] https://github.com/bats-core/bats-core/issues/573

https://github.com/containers/toolbox/pull/1192
2022-12-08 22:33:35 +01:00
Debarshi Ray
cc60bc6893 test/system: Split out the code to build an image without a name
This will be used by a subsequent commit to test the order in which
images with and without names are listed.

https://github.com/containers/toolbox/pull/1192
2022-12-08 22:33:35 +01:00
Debarshi Ray
54f09ae8a6 test/system: Group the test cases somewhat logically
A subsequent commit will test the order in which images with and without
names are listed.  It's logical for that test to come after the one
about the basic support for images without names.

https://github.com/containers/toolbox/pull/1192
2022-12-08 22:33:35 +01:00
Debarshi Ray
f0a805af84 test/system: Fix indentation
https://github.com/containers/toolbox/pull/1192
2022-12-08 22:33:35 +01:00
Debarshi Ray
0fde202d82 test/system: Keep empty lines to prevent missing and spurious newlines
The tests are intended for Toolbx, not Podman or other commands.  Hence,
it's only necessary to keep the empty lines for Toolbx invocations.
Being too sensitive about the exact output of other commands can lead to
spurious failures [1].

[1] Commit 259afdf815
    https://github.com/containers/toolbox/pull/846

https://github.com/containers/toolbox/pull/1192
2022-12-08 22:33:35 +01:00
Debarshi Ray
da1724c896 test/system: Test the order in 'list' for images and containers
https://github.com/containers/toolbox/pull/1192
2022-12-08 22:33:35 +01:00
Debarshi Ray
7375be82d0 test/system: Remove stray (possibly for debugging) 'podman images'
This was making it difficult to read the Bats assertions on test
failures, by polluting it with unexpected and irrelevant output from
'podman images'.  For example [1]:
  not ok 39 list: Images with and without names in 12332ms
  # (from function `assert' in file test/system/libs/bats-assert/src/assert.bash, line 46,
  #  in test file test/system/102-list.bats, line 126)
  #   `assert [ ${#stderr_lines[@]} -eq 0 ]' failed
  # REPOSITORY                                 TAG         IMAGE ID      CREATED      SIZE
  # registry.fedoraproject.org/fedora-toolbox  35          862705390e8b  4 weeks ago  332 MB
  # REPOSITORY                                 TAG         IMAGE ID      CREATED       SIZE
  # registry.fedoraproject.org/fedora-toolbox  35          862705390e8b  4 weeks ago   332 MB
  # registry.fedoraproject.org/fedora-toolbox  34          70cbe2ce60ca  7 months ago  354 MB
  #
  # -- assertion failed --
  # expression : [ 1 -eq 0 ]
  # --
  #

Fallout from 7973181136

[1] https://github.com/containers/toolbox/pull/1192

https://github.com/containers/toolbox/pull/1193
2022-12-08 00:32:12 +01:00
Debarshi Ray
aeb5d8ec1a test/system: Test a container with an old forward incompatible runtime
Commit ae43560d45 had added a test with a similar intention.  When
the test suite is run on a Fedora Rawhide host, it tests whether the
containers for the two previous stable Fedora releases start or not.
Fedora N-2 reaches End of Life 4 weeks after Fedora N is released [1].
So, testing the containers for Fedora Rawhide and the two previous
stable releases on a Fedora Rawhide host is a decent test of general
backwards compatibility.

However, as seen recently [2], this isn't enough to catch some known
ABI compatibility issues [3,4].  These involve toolbox binaries built
on hosts with newer toolchains that aren't meant to be run against
containers with older runtimes.  A targeted test is needed to defend
against these scenarios.

The fedora-toolbox:34 image has glibc-2.33, which is old enough to be
unable to run binaries compiled on Fedora 35 with glibc-2.34 and newer.

[1] https://docs.fedoraproject.org/en-US/releases/

[2] https://github.com/containers/toolbox/pull/1180

[3] Commit 6063eb27b9
    https://github.com/containers/toolbox/issues/821

[4] Commit 6ad9c63180
    https://github.com/containers/toolbox/issues/529

https://github.com/containers/toolbox/pull/1187
2022-12-07 13:03:09 +01:00
Debarshi Ray
7ab2f9b866 test/system: Replace fedora-toolbox:32 with fedora-toolbox:34
Fedora 32 reached End of Life on 25th May 2021:
https://docs.fedoraproject.org/en-US/releases/eol/

That's quite old because right now Fedora 35 is nearing its End of Life.

Since the tests are intended for Toolbx, not the Fedora infrastructure,
it will be better to use a newer image, because images that are too old
can get lost from registry.fedoraproject.org.  The fedora-toolbox:34
image can be a drop-in replacement for the fedora-toolbox:32 image for
the purposes of this test suite, and has the advantage of being newer.

Note that fedora-toolbox:34 is also old enough to test that the toolbox
binary runs against it's build-time ABI from the host, and not the
Toolbx container's ABI, when it's invoked as the entry point of the
container [1,2].  This is important because the subsequent commit will
add a test to ensure that.

[1] Commit 6063eb27b9
    https://github.com/containers/toolbox/issues/821

[2] Commit 6ad9c63180
    https://github.com/containers/toolbox/issues/529

https://github.com/containers/toolbox/pull/1187
2022-12-07 12:56:41 +01:00
Debarshi Ray
021053716e test/system: Add copyright and license notices
https://github.com/containers/toolbox/pull/1179
2022-11-29 13:54:13 +01:00
Nieves Montero
f2b7e440e1 Fix spelling mistakes using codespell
https://github.com/containers/toolbox/pull/1166
https://github.com/containers/toolbox/pull/1149

Signed-off-by: Nieves Montero <nmontero@redhat.com>
2022-11-17 11:56:58 +01:00
Debarshi Ray
1b85f711e4 test/system: Test 'completion'
https://github.com/containers/toolbox/pull/1165
2022-11-17 11:34:42 +01:00
Debarshi Ray
685f1f794d test/system: Be more strict when checking the version
https://github.com/containers/toolbox/pull/1165
2022-11-17 11:28:00 +01:00
Debarshi Ray
68d63bf09e test/system: Ensure that the right containers are run
https://github.com/containers/toolbox/pull/1163
2022-11-16 20:46:47 +01:00
Debarshi Ray
a61f88d7b2 test/system: Ensure that /run/.containerenv and /run/.toolboxenv exist
This is a precursor to verifying the names of the containers and
ensuring that the right ones are getting used.

https://github.com/containers/toolbox/pull/1163
2022-11-16 20:46:47 +01:00
Debarshi Ray
0000c68ee6 test/system: Ensure that 'toolbox run false' has exit code 1
This is a precursor to checking that higher valued exit codes from the
command running inside the container are retained, and commands like
test(1) can be used with 'toolbox run ...' in subsequent test cases.

https://github.com/containers/toolbox/pull/1163
2022-11-16 20:46:40 +01:00
Debarshi Ray
78683b38ae test/system: Use long options, instead of their shorter aliases
The long options are easier to grep(1) for in the sources than their
shorter aliases.

https://github.com/containers/toolbox/pull/1162
2022-11-16 19:56:24 +01:00
Debarshi Ray
220164b396 test/system: Shorten the names of the tests and use consistent wording
Currently, some of the names of the tests were too long, and had
inconsistent and verbose wording.  This made it difficult to look at
them and get a gist of all the scenarios being tested.  The names are
like headings.  They shouldn't be too long, should capture the primary
objective of the test and be consistent in their wording.

https://github.com/containers/toolbox/pull/1161
2022-11-16 19:49:56 +01:00
Debarshi Ray
8ec03ab85f test/system: Order the tests by increasing order of the exit code
This is another step towards making it easy to look at the file and get
a gist of all the scenarios being tested.

https://github.com/containers/toolbox/pull/1161
2022-11-16 13:41:43 +01:00
Debarshi Ray
485489867b test/system: Simplify the exit code checks by relying on Bats >= 1.5.0
Commit 978bb524e4 already added a dependency on Bats >= 1.5.0,
which is present in Fedora >= 35.  Therefore, it should be exploited
wherever possible to simplify things.

However, bats_require_minimum_version can't be used, because it's
only available from Bats 1.7.0 [1], which is new enough that it's absent
from Fedora 35.

[1] Bats commit 71d6b71cebc3d32b
    https://github.com/bats-core/bats-core/issues/556
    https://bats-core.readthedocs.io/en/stable/warnings/BW02.html

https://github.com/containers/toolbox/pull/1161
2022-11-16 13:41:39 +01:00
Debarshi Ray
80f9ac754d test/system: Remove redundant assertion
Fallout from 978bb524e4

https://github.com/containers/toolbox/pull/1161
2022-11-16 13:40:36 +01:00
Debarshi Ray
04e94868e0 test/system: Check the line count in the standard error & output streams
https://github.com/containers/toolbox/pull/1160
2022-11-16 12:51:35 +01:00
Debarshi Ray
e8ad1eaad0 test/system: Ensure that error messages go to the standard error stream
Currently, there's no way to get assert_line to use the stderr_lines
array [1].  This is worked around by assigning stderr_lines to the
'lines' array.

[1] https://github.com/bats-core/bats-assert/issues/42

https://github.com/containers/toolbox/pull/1160
2022-11-16 12:36:46 +01:00
Debarshi Ray
9bf9f97e2c test/system: Ensure that 'toolbox run --preserve-fds ...' works
Note that file descriptors 3 and 4 are reserved by Bats.  The former is
used for adding custom text to the Test Anything Protocol (or TAP)
stream [1] and the latter for tracing [2].

[1] https://bats-core.readthedocs.io/en/stable/writing-tests.html#file-descriptor-3-read-this-if-bats-hangs
    https://bats-core.readthedocs.io/en/stable/writing-tests.html#printing-to-the-terminal

[2] Bats commit 635700cd2282b754
    https://github.com/bats-core/bats-core/pull/467
    https://github.com/bats-core/bats-core/pull/488

https://github.com/containers/toolbox/issues/1066
2022-11-14 23:10:29 +01:00
Debarshi Ray
f779c798f0 test/system: Remove workaround for carriage return without a terminal
Commit a22d7821cb ensured that a nested pseudo-terminal device is
only created for the process running inside the container, if the Toolbx
binary's standard input and output streams are connected to a terminal.

Therefore, 'echo ...' no longer ends with an unwanted extra carriage
return when terminal devices are absent - there's only a line feed for
the trailing newline.  Hence, there's no need to use the -n flag to skip
the trailing newline.

This reverts parts of commit 16b0c5d88f.

https://github.com/containers/toolbox/issues/157
2022-11-10 19:28:54 +01:00
Debarshi Ray
190a76ac0a test/system: Group the test cases somewhat logically
It seems that as new test cases got developed they got appended towards
the end of the file.  Now that there are a non-trivial number of test
cases, it's difficult to look at the file and get a gist of all the
scenarios being tested.

It will be better to have some logical grouping -- starting with the
most basic functionality, then moving on to more advanced features,
and then finally the errors.

This is a step towards that.

https://github.com/containers/toolbox/pull/1155
2022-11-10 18:09:19 +01:00
Debarshi Ray
741603c64e test/system: Ensure that $HOME is used as a fallback working directory
This needs a directory that's going to be present on the host operating
system across various configurations of all supported distributions,
such as the hosts running the CI, but not inside the Toolbx containers.

It looks like /etc/kernel is present on both Debian and Fedora, but
absent from the fedora-toolbox images.  On a Debian 10 server, it's
owned by several packages:
  $ dpkg-query --search /etc/kernel
  dkms, systemd, grub2-common, initramfs-tools, apt: /etc/kernel

... while on Fedora 36 Workstation:
  $ rpm --file --query /etc/kernel
  systemd-udev-250.8-1.fc36.x86_64

Currently, there's no way to get assert_line to use the stderr_lines
array [1].  This is worked around by assigning stderr_lines to the
'lines' array.

[1] https://github.com/bats-core/bats-assert/issues/42

https://github.com/containers/toolbox/pull/1153
2022-11-10 12:28:14 +01:00
Debarshi Ray
3326dda259 test/system: Group the test cases somewhat logically
It seems that as new test cases got developed they got appended towards
the end of the file.  Now that there are a non-trivial number of test
cases, it's difficult to look at the file and get a gist of all the
scenarios being tested.

It will be better to have some logical grouping -- starting with the
most basic functionality, then moving on to more advanced features,
and then finally the errors.

This a step towards that.

https://github.com/containers/toolbox/pull/1152
2022-11-08 20:06:11 +01:00
Debarshi Ray
1ce59a6a2d cmd/run: Ensure that 'run' has the same container environment as 'enter'
Currently, commands invoked using 'toolbox run' have a different
environment than the interactive environment offered by 'toolbox enter'.
This is because 'toolbox run' was invoking the commands using something
like this:
  $ bash -c 'exec "$@"' bash [COMMAND]

... whereas, 'toolbox enter' was using something like this:
  $ bash -c 'exec "$@"' bash bash --login

In the first case, the helper Bash shell is a non-interactive non-login
shell.  This means that it doesn't read any of the usual start-up files,
and, hence, it doesn't pick up anything that's specified in them.  It
runs with the default environment variables set up by Podman and the
Toolbx image, plus the environment variables set by Toolbx itself.

In the second case, even though the helper Bash shell is still the same
as the first, it eventually invokes a login shell, which runs the usual
set of start-up files and picks up everything that's specified in them.

Therefore, to ensure parity, 'toolbox run' should always have a login
shell in the call chain inside the Toolbx container.

The easiest option is to always use a helper shell that's a login shell
with 'toolbox run', but not 'toolbox enter' so as to avoid reading the
same start-up files twice, due to two login shells in the call chain.
It will still end up reading the same start-up files twice, if someone
tried to invoke a login shell through 'toolbox run', which is fine.
It's very difficult to be sure that the user is invoking a login shell
through 'toolbox run', and it's not what most users will be doing.

https://github.com/containers/toolbox/issues/1076
2022-10-25 16:56:20 +02:00
Debarshi Ray
5284f875d3 cmd/run: Fix the name of the shell for running commands in containers
For the most part, this fixes a minor cosmetic issue for users, but it
does make the code less misleading to read for those hacking on Toolbx.
Further details below.

Commands are invoked inside a Toolbx from a helper shell invoked by
capsh(1).  Unless capsh(1) is built with custom options, the helper
shell is always bash, not /bin/sh:
  $ capsh --caps="" -- -c 'echo "$(readlink /proc/$$/exe)"'
  /usr/bin/bash

( The possibility of capsh(1) using a different shell, other than Bash,
  through a custom build option is ignored for the time being.  If there
  really are downstream distributors who do that, then this can be
  addressed one way or another. )

Secondly, the name assigned to the embedded command string's '$0' should
only be the basename of the helper shell's binary, not the full path, to
match the usual behaviour:
  $ bash -c 'exec foo'
  bash: line 1: exec: foo: not found

With 'toolbox run' it was:
  $ toolbox run foo
  /bin/sh: line 1: exec: foo: not found
  Error: command foo not found in container fedora-toolbox-36

https://github.com/containers/toolbox/pull/1147
2022-10-24 18:04:19 +02:00
Debarshi Ray
8a1d81df28 test/system: Tweak
Using 'true' is likely going to be quicker than launching the entire
shell (ie., /bin/sh).

Note that 'toolbox run' already invokes a wrapper shell via capsh(1)
before invoking the user-specified command.  So, this was the second
instance of a shell.

https://github.com/containers/toolbox/pull/1145
2022-10-23 21:02:11 +02:00
Ondřej Míchal
8f6deadaef test/system: Drop extra pull when caching images
In 54a2ca1 image caching has been done by first pulling using Podman and
then moving the image from the local container store to a directory. The
pull to the local container store can be skipped and instead we can use
Skopeo to directly save the pulled image into a directory.

On my machine this reduced the time of the system test setup "test" by
about 50 seconds. This speed-up largely depends on the available network
connection, though.
2022-09-21 14:49:21 +02:00
Debarshi Ray
32b147b9ff cmd/create: Improve the error messages for mutually exclusive options
https://github.com/containers/toolbox/pull/1109
2022-09-07 19:20:17 +02:00
Debarshi Ray
947e582c0f cmd/create: Improve the error message for the --authfile option
https://github.com/containers/toolbox/pull/1109
2022-09-07 19:20:05 +02:00
Debarshi Ray
8454b31a82 cmd/utils, pkg/utils: Improve the error messages for the distro option
https://github.com/containers/toolbox/issues/937
https://github.com/containers/toolbox/pull/1103
2022-09-02 14:52:44 +02:00
Debarshi Ray
8ca5611942 Increase the validation coverage for the container & release options
Currently, the container name and release are only validated if they
were specified as command line options.  Neither the value of release
in the configuration file nor the container name generated from an
image are validated.

There's also a lot of repeated code in the command front-ends to
validate the container name and release.  This opens the door for
mistakes.  Any adjustment to the code must be repeated elsewhere, and
there are subtle interactions and overlaps between the validation code
and the code to resolve container and image names.

It's worth noting that the container and image name resolution happens
for both the command line and configuration file options, and generates
the container name from the image when necessary.

Therefore, validating everything while resolving cleans up the command
front-ends and increases the coverage of the validation.

This introduces the use of sentinel error values and custom error
implementations to identify the different errors that can occur while
resolving the container and images, so that they can be appropriately
shown to the user.

https://github.com/containers/toolbox/pull/1101
2022-09-02 13:11:32 +02:00
Debarshi Ray
b5474bff84 cmd, pkg/utils: Clarify the error message if the release is invalid
Currently, if --release has an invalid argument, the error message
doesn't give any hints as to what's an acceptable value.  This can be
confusing.  eg., is 36 a valid argument for Fedora?  Or is it f36?  Or
is it F36?  Is 'rawhide' accepted?

https://github.com/containers/toolbox/issues/937
https://github.com/containers/toolbox/pull/1100
2022-09-01 17:57:39 +02:00