46f4e1bec5
Around 138 distinct errors found and fixed; thanks! Reviewed-by: Kurt Roeckx <kurt@roeckx.be> Reviewed-by: Tim Hudson <tjh@openssl.org> Reviewed-by: Rich Salz <rsalz@openssl.org> (Merged from https://github.com/openssl/openssl/pull/3459)
164 lines
4.6 KiB
Text
164 lines
4.6 KiB
Text
Running external test suites with OpenSSL
|
|
=========================================
|
|
|
|
It is possible to integrate external test suites into OpenSSL's "make test".
|
|
This capability is considered a developer option and does not work on all
|
|
platforms.
|
|
|
|
|
|
|
|
The BoringSSL test suite
|
|
========================
|
|
|
|
In order to run the BoringSSL tests with OpenSSL, first checkout the BoringSSL
|
|
source code into an appropriate directory. This can be done in two ways:
|
|
|
|
1) Separately from the OpenSSL checkout using:
|
|
|
|
$ git clone https://boringssl.googlesource.com/boringssl boringssl
|
|
|
|
The BoringSSL tests are only confirmed to work at a specific commit in the
|
|
BoringSSL repository. Later commits may or may not pass the test suite:
|
|
|
|
$ cd boringssl
|
|
$ git checkout 490469f850e
|
|
|
|
2) Using the already configured submodule settings in OpenSSL:
|
|
|
|
$ git submodule update --init
|
|
|
|
Configure the OpenSSL source code to enable the external tests:
|
|
|
|
$ cd ../openssl
|
|
$ ./config enable-ssl3 enable-ssl3-method enable-weak-ssl-ciphers \
|
|
enable-external-tests
|
|
|
|
Note that using other config options than those given above may cause the tests
|
|
to fail.
|
|
|
|
Run the OpenSSL tests by providing the path to the BoringSSL test runner in the
|
|
BORING_RUNNER_DIR environment variable:
|
|
|
|
$ BORING_RUNNER_DIR=/path/to/boringssl/ssl/test/runner make test
|
|
|
|
Note that the test suite may change directory while running so the path provided
|
|
should be absolute and not relative to the current working directory.
|
|
|
|
To see more detailed output you can run just the BoringSSL tests with the
|
|
verbose option:
|
|
|
|
$ VERBOSE=1 BORING_RUNNER_DIR=/path/to/boringssl/ssl/test/runner make \
|
|
TESTS="test_external_boringssl" test
|
|
|
|
|
|
Test failures and suppressions
|
|
------------------------------
|
|
|
|
A large number of the BoringSSL tests are known to fail. A test could fail
|
|
because of many possible reasons. For example:
|
|
|
|
- A bug in OpenSSL
|
|
- Different interpretations of standards
|
|
- Assumptions about the way BoringSSL works that do not apply to OpenSSL
|
|
- The test uses APIs added to BoringSSL that are not present in OpenSSL
|
|
- etc
|
|
|
|
In order to provide a "clean" baseline run with all the tests passing a config
|
|
file has been provided that suppresses the running of tests that are known to
|
|
fail. These suppressions are held in the file "test/ossl_shim/ossl_config.json"
|
|
within the OpenSSL source code.
|
|
|
|
The community is encouraged to contribute patches which reduce the number of
|
|
suppressions that are currently present.
|
|
|
|
|
|
Python PYCA/Cryptography test suite
|
|
===================================
|
|
|
|
This python test suite runs cryptographic tests with a local OpenSSL build as
|
|
the implementation.
|
|
|
|
First checkout the PYCA/Cryptography module into ./pyca-cryptography using:
|
|
|
|
$ git submodule update --init
|
|
|
|
Then configure/build OpenSSL compatible with the python module:
|
|
|
|
$ ./config shared enable-external-tests
|
|
$ make
|
|
|
|
The tests will run in a python virtual environment which requires virtualenv
|
|
to be installed.
|
|
|
|
$ make test VERBOSE=1 TESTS=test_external_pyca
|
|
|
|
Test failures and suppressions
|
|
------------------------------
|
|
|
|
Some tests target older (<=1.0.2) versions so will not run. Other tests target
|
|
other crypto implementations so are not relevant. Currently no tests fail.
|
|
|
|
|
|
krb5 test suite
|
|
===============
|
|
|
|
Much like the PYCA/Cryptography test suite, this builds and runs the krb5
|
|
tests against the local OpenSSL build.
|
|
|
|
You will need a git checkout of krb5 at the top level:
|
|
|
|
$ git clone https://github.com/krb5/krb5
|
|
|
|
krb5's master has to pass this same CI, but a known-good version is
|
|
krb5-1.15.1-final if you want to be sure.
|
|
|
|
$ cd krb5
|
|
$ git checkout krb5-1.15.1-final
|
|
$ cd ..
|
|
|
|
OpenSSL must be built with external tests enabled:
|
|
|
|
$ ./config enable-external-tests
|
|
$ make
|
|
|
|
krb5's tests will then be run as part of the rest of the suite, or can be
|
|
explicitly run (with more debugging):
|
|
|
|
$ VERBOSE=1 make TESTS=test_external_krb5 test
|
|
|
|
Test-failures suppressions
|
|
--------------------------
|
|
|
|
krb5 will automatically adapt its test suite to account for the configuration
|
|
of your system. Certain tests may require more installed packages to run. No
|
|
tests are expected to fail.
|
|
|
|
|
|
Updating test suites
|
|
====================
|
|
|
|
To update the commit for any of the above test suites:
|
|
|
|
- Make sure the submodules are cloned locally:
|
|
|
|
$ git submodule update --init --recursive
|
|
|
|
- Enter subdirectory and pull from the repository (use a specific branch/tag if required):
|
|
|
|
$ cd <submodule-dir>
|
|
$ git pull origin master
|
|
|
|
- Go to root directory, there should be a new git status:
|
|
|
|
$ cd ../
|
|
$ git status
|
|
...
|
|
# modified: <submodule-dir> (new commits)
|
|
...
|
|
|
|
- Add/commit/push the update
|
|
|
|
git add <submodule-dir>
|
|
git commit -m "Updated <submodule> to latest commit"
|
|
git push
|
|
|