Commit graph

155 commits

Author SHA1 Message Date
Debarshi Ray
9f4d906e39 test/system: Remove redundant comment
It's quite obvious what the corresponding code is doing, and it isn't
any harder to understand than the rest of the code that's not commented.

https://github.com/containers/toolbox/pull/1282
2023-03-26 10:36:59 +02:00
Debarshi Ray
06245dbf52 test/system: Synchronize with the parseRelease() unit tests
https://github.com/containers/toolbox/pull/1280
2023-03-24 19:57:33 +01:00
Debarshi Ray
26b031f686 test/system: Split the tests for parsing releases
This will make it easier to find out exactly which test failed.

https://github.com/containers/toolbox/pull/1280
2023-03-23 18:09:45 +01:00
Debarshi Ray
44cbdb1f18 test/system: Ensure that both non-Toolbx containers & images are skipped
https://github.com/containers/toolbox/pull/1273
2023-03-22 12:05:06 +01:00
Debarshi Ray
11f5cde45e test/system: Check the number of images after pulling them
This is a quick sanity check with 'podman images' to ensure that all the
images are in place before running 'list'.  Other tests already do this,
so this change makes these two tests consistent with the rest.

https://github.com/containers/toolbox/pull/1273
2023-03-22 10:40:57 +01:00
Debarshi Ray
63aeabe86b test/system: Factor out some repeated code
https://github.com/containers/toolbox/pull/1273
2023-03-22 10:08:47 +01:00
Debarshi Ray
55bf3d06a8 test/system: Use fully qualified image names to check the 'list' output
https://github.com/containers/toolbox/pull/1273
2023-03-22 09:45:23 +01:00
Debarshi Ray
164c9eff36 test/system: Test 'list' with a RHEL Toolbx image
This restores parts of commit e09de9f3e5.

https://github.com/containers/toolbox/pull/1273
2023-03-22 09:18:04 +01:00
Debarshi Ray
aa9da193c6 test/system: Test 'list' with an old Fedora Toolbx image
It's good to ensure that older well-known Toolbx images continue to work
with newer versions of 'list'.

https://github.com/containers/toolbox/pull/1278
2023-03-22 09:16:58 +01:00
Debarshi Ray
571cfc1296 test/system: Use the same convenience function as elsewhere
Fallout from b27795a03e

https://github.com/containers/toolbox/pull/1278
2023-03-22 09:16:52 +01:00
Debarshi Ray
96d629c21f test/system: Test 'list' with the default Toolbx image
This is the 'simple' case of having a well-known Toolbx image (ie.,
not a copy, not an image without a name, not a non-Toolbx image).  It's
good to ensure that the default image works as expected with 'list'
before moving on to more complex scenarios.

https://github.com/containers/toolbox/pull/1278
2023-03-22 09:16:44 +01:00
Debarshi Ray
20fa122820 test/system: Test 'list' both with and without --images
https://github.com/containers/toolbox/pull/1278
2023-03-22 09:16:37 +01:00
Debarshi Ray
6890f5dc8d test/system: Rename a 'list' test for clarity
A subsequent commit will add a variant of this test that uses 'list'
without the '--images' option.

https://github.com/containers/toolbox/pull/1278
2023-03-22 09:16:29 +01:00
Debarshi Ray
9cae66ddd8 test/system: Check the standard error & output streams separately
https://github.com/containers/toolbox/pull/1278
2023-03-22 09:16:20 +01:00
Debarshi Ray
e22a82fec8 test/system: Simplify the line count checks by relying on Bats >= 1.7.0
Fedora 35 was stuck with Bats 1.5.0.  However, it reached End of Life on
13th December 2022 and was dropped from the CI [1].  Fedora 36 is the
oldest supported Fedora and it has Bats 1.8.2.

Therefore, there's no need to retain compatibility with Bats < 1.7.0.

Note that bats_require_minimum_version itself is only available from
Bats 1.7.0 [2].

[1] Commit 419e4e8cd9
    https://github.com/containers/toolbox/pull/1237

[2] Bats commit 71d6b71cebc3d32b
    https://github.com/bats-core/bats-core/issues/556
    https://bats-core.readthedocs.io/en/stable/warnings/BW02.html

https://github.com/containers/toolbox/pull/1273
2023-03-21 19:20:21 +01:00
Debarshi Ray
08e40e666e test/system: Shorten the names of the tests and use consistent wording
Currently, some of the names of the tests were too long, and had
inconsistent and verbose wording.  This made it difficult to look at
them and get a gist of all the scenarios being tested.  The names are
like headings.  They shouldn't be too long, should capture the primary
objective of the test and be consistent in their wording.

https://github.com/containers/toolbox/pull/1276
2023-03-21 19:11:20 +01:00
Debarshi Ray
5b8a7a00c0 test/system: Use long options, instead of their shorter aliases
The long options are easier to grep(1) for in the sources than their
shorter aliases.

https://github.com/containers/toolbox/pull/1276
2023-03-21 19:11:13 +01:00
Debarshi Ray
f364812781 test/system: Shorten the names of the tests and use consistent wording
Currently, some of the names of the tests were too long, and had
inconsistent and verbose wording.  This made it difficult to look at
them and get a gist of all the scenarios being tested.  The names are
like headings.  They shouldn't be too long, should capture the primary
objective of the test and be consistent in their wording.

https://github.com/containers/toolbox/pull/1271
2023-03-20 18:28:10 +01:00
Debarshi Ray
edd82d7e6e test/system: Style fixes
Fallout from e5bab51bca

https://github.com/containers/toolbox/pull/1271
2023-03-20 18:28:07 +01:00
Debarshi Ray
c6760299d2 pkg/utils, test/system: Add some more tests for parsing RHEL releases
https://github.com/containers/toolbox/pull/1262
2023-03-16 22:01:45 +01:00
Debarshi Ray
e01927bfdc pkg/utils, test/system: Add some more tests for parsing Fedora releases
https://github.com/containers/toolbox/pull/1262
2023-03-16 22:01:45 +01:00
Debarshi Ray
a4cc791f39 test/system: Synchronize with the parseRelease() unit tests
https://github.com/containers/toolbox/pull/1266
2023-03-16 18:41:24 +01:00
Debarshi Ray
1bd6cf17ce test/system: Split the tests for parsing releases
This will make it easier to find out exactly which test failed.

https://github.com/containers/toolbox/pull/1266
2023-03-16 18:31:12 +01:00
Debarshi Ray
e5bab51bca test/system: Shorten the names of the tests and use consistent wording
Currently, some of the names of the tests were too long, and had
inconsistent and verbose wording.  This made it difficult to look at
them and get a gist of all the scenarios being tested.  The names are
like headings.  They shouldn't be too long, should capture the primary
objective of the test and be consistent in their wording.

https://github.com/containers/toolbox/pull/1265
2023-03-16 17:57:46 +01:00
Debarshi Ray
068d22a199 test/system: Use long options, instead of their shorter aliases
The long options are easier to grep(1) for in the sources than their
shorter aliases.

https://github.com/containers/toolbox/pull/1265
2023-03-16 17:37:16 +01:00
Debarshi Ray
052666840a Fix spelling mistakes
The noun is 'setup' and the verb is 'set up'.  Similarly 'lookup' and
'look up'.

Original patch from Erik Sjölund for Podman [1,2].

[1] Podman commit aa4279ae151fa9df
    https://github.com/containers/podman/commit/aa4279ae151fa9df
    https://github.com/containers/podman/pull/14658

[2] Podman commit 2827140907255ed0
    https://github.com/containers/podman/commit/2827140907255ed0
    https://github.com/containers/podman/pull/14659

https://github.com/containers/toolbox/pull/1223
2023-01-31 15:53:15 +01:00
Debarshi Ray
936a157117 doc, test/system: Fix the titles of the manuals
Currently, the titles of the manuals are rendered with a pair of empty
parentheses and no section title:
  toolbox(1)()                                              toolbox(1)()

  NAME
         toolbox - Tool for containerized command line environments...

However, they should be:
  toolbox(1)             General Commands Manual              toolbox(1)

  NAME
         toolbox - Tool for containerized command line environments...

This is because the troff generated by go-md2man from Markdown has a
faulty invocation of the .TH macro [1]:
  .nh
  .TH toolbox(1)
  .SH NAME
  .PP
  toolbox - Tool for containerized command line environments on Linux

It should be:
  .nh
  .TH toolbox 1
  .SH NAME
  .PP
  toolbox - Tool for containerized command line environments on Linux

Original patch from Andrew Denton for Podman [2].

[1] https://www.gnu.org/software/groff/manual/groff.html

[2] Podman commit 63c779a857b55b00
    https://github.com/containers/podman/pull/15621

https://github.com/containers/toolbox/pull/1210
2023-01-12 18:45:00 +01:00
Debarshi Ray
cac6b25f19 test/system: Tighten the regular expression used to detect the image ID
An OCI image's identifier (or ID) is a SHA-256 hash [1] whose length is
64 hexadecimal digits.  The ID can be abbreviated to a smaller number of
digits but the minimum is 6.  The regular expression being used to parse
the image ID wasn't checking the number of digits, and would even accept
an empty string as a valid ID, which was clearly wrong.

The format of a image reference is defined in:
https://github.com/moby/moby/blob/master/image/spec/v1.1.md
https://github.com/distribution/distribution/blob/main/reference/reference.go
https://github.com/distribution/distribution/blob/main/reference/regexp.go

Fallout from c6c2e426e0

[1] https://en.wikipedia.org/wiki/SHA-2

https://github.com/containers/toolbox/pull/1209
2023-01-12 14:44:05 +01:00
Debarshi Ray
65194bd45e test/system: Avoid false positives from codespell
Otherwise codespell would complain:
  : @test "create: Try to create a container with invalid custom name...
  >   run $TOOLBOX -y create "ßpeci@l.Nam€"
  :
  ./test/system/101-create.bats:57: Nam ==> Name

CentOS Stream 9 has codespell-2.2.1, while so far the 'unit tests' were
being run on Fedora 36, which only has codespell-2.1.0.

This is a step towards testing on CentOS Stream 9.

https://github.com/containers/toolbox/pull/1200
2022-12-14 23:09:21 +01:00
Debarshi Ray
5f5f7b6aab test/system: Fix spelling mistake using codespell
CentOS Stream 9 has codespell-2.2.1, while so far the 'unit tests' were
being run on Fedora 36, which only has codespell-2.1.0.

This is a step towards testing on CentOS Stream 9.

Fallout from ecd1ced719

https://github.com/containers/toolbox/pull/1200
2022-12-14 23:09:21 +01:00
Martin Krajnak
05a062f8c9 test/system: Ensure that copied images are clearly identified
Note that 'run --keep-empty-lines' counts the trailing newline on the
last line as a separate line.

Until Bats 1.7.0, 'run --keep-empty-lines' had a bug where even when a
command produced no output, it would report a line count of one [1] due
to a stray line feed character.  This needs to be conditionalized, since
Fedora 35 has Bats 1.5.0.

[1] https://github.com/bats-core/bats-core/issues/573

https://github.com/containers/toolbox/issues/1043
2022-12-12 21:49:21 +01:00
Debarshi Ray
6aab0a6175 Unbreak sorting and clearly identify copied images in 'list'
Currently, if an image was copied with:
  $ skopeo copy \
      containers-storage:registry.fedoraproject.org/fedora-toolbox:36 \
      containers-storage:localhost/fedora-toolbox:36

... or:
  $ podman tag \
      registry.fedoraproject.org/fedora-toolbox:36 \
      localhost/fedora-toolbox:36

... then it would show up twice in 'list' with the same name, and in the
wrong order.

Either as:
  $ toolbox list --images
  IMAGE ID      IMAGE NAME                                      CREATED
  2110dbbc33d2  localhost/fedora-toolbox:36                     1 day...
  e085805ade4a  registry.access.redhat.com/ubi8/toolbox:latest  1 day...
  2110dbbc33d2  localhost/fedora-toolbox:36                     1 day...
  70cbe2ce60ca  registry.fedoraproject.org/fedora-toolbox:34    1 day...

... or as:
  $ toolbox list --images
  IMAGE ID      IMAGE NAME                                      CREATED
  2110dbbc33d2  registry.fedoraproject.org/fedora-toolbox:36    1 day...
  e085805ade4a  registry.access.redhat.com/ubi8/toolbox:latest  1 day...
  2110dbbc33d2  registry.fedoraproject.org/fedora-toolbox:36    1 day...
  70cbe2ce60ca  registry.fedoraproject.org/fedora-toolbox:34    1 day...

The correct output should be similar to 'podman images', and be sorted
in ascending order of the names:
  $ toolbox list --images
  IMAGE ID      IMAGE NAME                                      CREATED
  2110dbbc33d2  localhost/fedora-toolbox:36                     1 day...
  e085805ade4a  registry.access.redhat.com/ubi8/toolbox:latest  1 day...
  70cbe2ce60ca  registry.fedoraproject.org/fedora-toolbox:34    1 day...
  2110dbbc33d2  registry.fedoraproject.org/fedora-toolbox:36    1 day...

The problem is that, in these situations, 'podman images --format json'
returns separate identical JSON collections for each copy of the image,
and all of those copies have multiple names:
  [
    {
      "Id": "2110dbbc33d2",
      ...
      "Names": [
        "localhost/fedora-toolbox:36",
        "registry.fedoraproject.org/fedora-toolbox:36"
      ],
      ...
    },
    {
      "Id": "e085805ade4a",
      ...
      "Names": [
        "registry.access.redhat.com/ubi8/toolbox:latest"
      ],
      ...
    },
    {
      "Id": "2110dbbc33d2",
      ...
      "Names": [
        "localhost/fedora-toolbox:36",
        "registry.fedoraproject.org/fedora-toolbox:36"
      ],
      ...
    }
    {
      "Id": "70cbe2ce60ca",
      ...
      "Names": [
        "registry.fedoraproject.org/fedora-toolbox:34"
      ],
      ...
    },
  ]

The image objects need to be flattened to have only one unique name per
copy, but with the same ID, and then sorted to ensure the right order.

Note that the ordering was already broken since commit 2369da5d31,
which started using 'podman images --sort repository'.  Podman can sort
by either the image's repository or tag, but not by the unified name,
which is what Toolbx needs.  Therefore, even without copied images,
Toolbx really does need to sort the images itself.

Prior to commit 2369da5d31, the ordering was correct, but copied
images would only show up once.

Fallout from 2369da5d31

This reverts parts of commit 67e210378e.

https://github.com/containers/toolbox/issues/1043
2022-12-12 21:49:21 +01:00
Debarshi Ray
51eccd3da5 test/system: Test 'rmi' with an image
https://github.com/containers/toolbox/pull/1195
2022-12-12 20:59:52 +01:00
Debarshi Ray
bbdf4ddb63 test/system: Test 'rmi' with an image and its copy
https://github.com/containers/toolbox/issues/1043
2022-12-12 20:59:52 +01:00
Martin Krajnak
d5daa7167e test/system: Add a helper to pull the default image and copy it
This will be used in subsequent commits to test the handling of such
copied images in 'toolbox list' and 'toolbox rmi'.

https://github.com/containers/toolbox/issues/1043
2022-12-12 20:59:47 +01:00
Debarshi Ray
a0d4c957b3 test/system: Test 'rmi' with an image without a name
https://github.com/containers/toolbox/pull/1195
2022-12-12 15:13:26 +01:00
Debarshi Ray
e25ab310fa test/system: Test 'rmi --all' with an image without a name
https://github.com/containers/toolbox/pull/1195
2022-12-12 15:13:26 +01:00
Debarshi Ray
8a37c08780 test/system: Test 'rmi --all' without any images
https://github.com/containers/toolbox/pull/1195
2022-12-12 15:13:26 +01:00
Debarshi Ray
210985ecd1 test/system: Ensure that error messages go to the standard error stream
Currently, there's no way to get assert_line to use the stderr_lines
array [1].  This is worked around by assigning stderr_lines to the
'lines' array.

[1] https://github.com/bats-core/bats-assert/issues/42

https://github.com/containers/toolbox/pull/1195
2022-12-12 15:13:26 +01:00
Debarshi Ray
26ed682cd1 test/system: Keep empty lines to prevent missing and spurious newlines
https://github.com/containers/toolbox/pull/1195
2022-12-12 15:13:26 +01:00
Debarshi Ray
303c7ae99a test/system: Don't ignore copies when counting images
If an image was copied with:
  $ skopeo copy \
      containers-storage:registry.fedoraproject.org/fedora-toolbox:36 \
      containers-storage:localhost/fedora-toolbox:36

... or:
  $ podman tag \
      registry.fedoraproject.org/fedora-toolbox:36 \
      localhost/fedora-toolbox:36

... then the image ID is only showed once in 'podman images --quiet',
not twice.

A subsequent commit will use this to write tests to ensure that copied
images are correctly handled.

https://github.com/containers/toolbox/issues/1043
2022-12-12 15:04:42 +01:00
Debarshi Ray
5e8446971c test/system: Use long options, instead of their shorter aliases
The long options are easier to grep(1) for in the sources than their
shorter aliases.

https://github.com/containers/toolbox/pull/1197
2022-12-12 15:00:46 +01:00
Debarshi Ray
f17a632f9a test/system: Check the line count in the standard error & output streams
Note that 'run --keep-empty-lines' counts the trailing newline on the
last line as a separate line.

Until Bats 1.7.0, 'run --keep-empty-lines' had a bug where even when a
command produced no output, it would report a line count of one [1] due
to a stray line feed character.  This needs to be conditionalized, since
Fedora 35 has Bats 1.5.0.

[1] https://github.com/bats-core/bats-core/issues/573

https://github.com/containers/toolbox/pull/1192
2022-12-08 23:08:47 +01:00
Debarshi Ray
89385e12b5 test/system: Ensure that non-error messages go to the standard output
https://github.com/containers/toolbox/pull/1192
2022-12-08 22:33:35 +01:00
Debarshi Ray
4d1cc5b39b test/system: Test the order in 'list' for images with & without names
Note that 'run --keep-empty-lines' counts the trailing newline on the
last line as a separate line.

Until Bats 1.7.0, 'run --keep-empty-lines' had a bug where even when a
command produced no output, it would report a line count of one [1] due
to a stray line feed character.  This needs to be conditionalized, since
Fedora 35 has Bats 1.5.0.

[1] https://github.com/bats-core/bats-core/issues/573

https://github.com/containers/toolbox/pull/1192
2022-12-08 22:33:35 +01:00
Debarshi Ray
cc60bc6893 test/system: Split out the code to build an image without a name
This will be used by a subsequent commit to test the order in which
images with and without names are listed.

https://github.com/containers/toolbox/pull/1192
2022-12-08 22:33:35 +01:00
Debarshi Ray
54f09ae8a6 test/system: Group the test cases somewhat logically
A subsequent commit will test the order in which images with and without
names are listed.  It's logical for that test to come after the one
about the basic support for images without names.

https://github.com/containers/toolbox/pull/1192
2022-12-08 22:33:35 +01:00
Debarshi Ray
f0a805af84 test/system: Fix indentation
https://github.com/containers/toolbox/pull/1192
2022-12-08 22:33:35 +01:00
Debarshi Ray
0fde202d82 test/system: Keep empty lines to prevent missing and spurious newlines
The tests are intended for Toolbx, not Podman or other commands.  Hence,
it's only necessary to keep the empty lines for Toolbx invocations.
Being too sensitive about the exact output of other commands can lead to
spurious failures [1].

[1] Commit 259afdf815
    https://github.com/containers/toolbox/pull/846

https://github.com/containers/toolbox/pull/1192
2022-12-08 22:33:35 +01:00
Debarshi Ray
da1724c896 test/system: Test the order in 'list' for images and containers
https://github.com/containers/toolbox/pull/1192
2022-12-08 22:33:35 +01:00