openssl/test
Richard Levitte 89cb61de2a Don't test the collected system errors when configured to not have them
Config options 'no-err' and 'no-autoerrinit'

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Tim Hudson <tjh@openssl.org>
Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
(Merged from https://github.com/openssl/openssl/pull/7711)

(cherry picked from commit f1d49ed947)
2018-11-27 09:03:59 +00:00
..
certs Add some test brainpool certificates 2018-11-12 11:19:58 +00:00
ct
d2i-tests add test for CVE-2016-7053 2016-11-10 13:04:11 +00:00
ocsp-tests Fix OCSP_basic_verify() cert chain construction in case bs->certs is NULL 2017-08-16 14:32:38 -04:00
ossl_shim Fix BoringSSL external test failures 2018-08-22 15:15:19 +01:00
recipes Don't test the collected system errors when configured to not have them 2018-11-27 09:03:59 +00:00
smime-certs Add alternative CMS P-256 cert 2017-08-10 16:48:18 +01:00
ssl-tests Update copyright year 2018-11-20 13:27:36 +00:00
testutil Update copyright year 2018-09-11 13:45:17 +01:00
aborttest.c
afalgtest.c Revert "Modify test/afalgtest to fail if the afalg engine couldn't be loaded" 2018-02-07 22:18:44 +01:00
asn1_decode_test.c TESTS: add test of decoding of invalid zero length ASN.1 INTEGER zero 2018-09-09 03:35:26 +02:00
asn1_encode_test.c Update copyright year 2018-02-27 13:59:42 +00:00
asn1_internal_test.c Update copyright year 2018-09-11 13:45:17 +01:00
asn1_string_table_test.c [Win] Fix some test method signatures ... 2017-08-16 10:36:34 -04:00
asn1_time_test.c test/asn1_time_test.c: make it work on 64-bit HP-UX. 2018-04-08 11:17:44 +02:00
asynciotest.c Update code for the final RFC version of TLSv1.3 (RFC8446) 2018-08-15 12:33:30 +01:00
asynctest.c Update copyright year 2018-05-29 13:16:04 +01:00
bad_dtls_test.c Remove unicode characters from source 2017-12-08 11:56:37 +01:00
bftest.c Consistent formatting for sizeof(foo) 2017-12-07 19:11:49 -05:00
bio_callback_test.c Extend the BIO callback tests to check the return value semantics 2018-10-04 14:20:27 +01:00
bio_enc_test.c Fix no-chacha and no-poly1305 2017-08-25 11:34:08 +01:00
bioprinttest.c Update the test framework so that the need for test_main is removed. Everything 2017-07-27 07:53:08 +10:00
bntest.c Fixed range of random produced in BN_is_prime_fasttest_ex() to be 1 < rand < w-1. It was using 1<= rand < w (which is wrong by 1 on both ends) 2018-06-22 07:07:20 +10:00
bntests.pl Make bntest be (mostly) file-based. 2016-11-28 12:26:05 -05:00
build.info Test: link drbgtest statically against libcrypto 2018-11-08 16:32:30 +01:00
CAss.cnf
CAssdh.cnf
CAssdsa.cnf
CAssrsa.cnf
casttest.c Use "" not <> for internal/ includes 2017-08-22 09:54:20 -04:00
CAtsa.cnf Added support for ESSCertIDv2 2017-05-03 09:04:23 +02:00
chacha_internal_test.c Update the test framework so that the need for test_main is removed. Everything 2017-07-27 07:53:08 +10:00
cipher_overhead_test.c Use "" not <> for internal/ includes 2017-08-22 09:54:20 -04:00
cipherbytes_test.c Use "" not <> for internal/ includes 2017-08-22 09:54:20 -04:00
cipherlist_test.c Use void in all function definitions that do not take any arguments 2018-05-11 14:37:48 +02:00
ciphername_test.c Use "" not <> for internal/ includes 2017-08-22 09:54:20 -04:00
clienthellotest.c Enhance ssltestlib's create_ssl_ctx_pair to take min and max proto version 2018-03-19 18:24:30 +01:00
cms-examples.pl
cmsapitest.c Add a CMS API test 2018-05-08 08:43:39 +01:00
conf_include_test.c NCONF_get_number refix. 2018-07-11 09:03:22 +10:00
constant_time_test.c Update copyright year 2018-02-27 13:59:42 +00:00
crltest.c Factorise duplicated code. 2017-11-13 07:52:35 -05:00
ct_test.c Add an explicit cast to time_t 2018-09-13 09:07:51 +01:00
ctype_internal_test.c Update copyright year 2018-03-20 13:08:46 +00:00
curve448_internal_test.c Update copyright year 2018-02-27 13:59:42 +00:00
d2i_test.c Consistent formatting for sizeof(foo) 2017-12-07 19:11:49 -05:00
danetest.c Use "" not <> for internal/ includes 2017-08-22 09:54:20 -04:00
danetest.in
danetest.pem
destest.c Update the test framework so that the need for test_main is removed. Everything 2017-07-27 07:53:08 +10:00
dhtest.c test/dhtest.c: fix resource leak 2018-09-10 10:32:33 +01:00
drbg_cavs_data.c Update copyright year 2018-04-03 13:57:12 +01:00
drbg_cavs_data.h Update copyright year 2018-04-03 13:57:12 +01:00
drbg_cavs_test.c Update copyright year 2018-04-03 13:57:12 +01:00
drbgtest.c Test: link drbgtest statically against libcrypto 2018-11-08 16:32:30 +01:00
drbgtest.h Add DRBG random method 2017-07-19 03:25:16 -04:00
dsa_no_digest_size_test.c Add test for DSA signatures of raw digests of various sizes 2018-07-29 21:27:36 +02:00
dsatest.c Use "" not <> for internal/ includes 2017-08-22 09:54:20 -04:00
dtls_mtu_test.c Update copyright year 2018-05-29 13:16:04 +01:00
dtlstest.c Add a test for duplicated DTLS records 2018-10-26 14:28:18 +01:00
dtlsv1listentest.c Update copyright year 2018-05-29 13:16:04 +01:00
ecdsatest.c Fix no-engine 2018-10-16 09:36:47 +10:00
ecstresstest.c Update copyright year 2018-09-11 13:45:17 +01:00
ectest.c Use the new non-curve type specific EC functions internally 2018-07-31 09:08:38 +01:00
enginetest.c Add EVP_PKEY_METHOD redirection test 2017-10-12 00:03:32 +01:00
errtest.c Save and restore the Windows error around TlsGetValue. 2018-05-23 17:34:54 -04:00
evp_extra_test.c Add a test where we reuse the EVP_PKEY_CTX for two HKDF test runs 2018-10-29 14:11:40 +00:00
evp_test.c test/evp_test.c: don't misuse pkey_test_ctrl() in mac_test_run() 2018-10-29 17:32:27 +01:00
evp_test.h Add support for multiple update calls in evp_test 2017-05-19 21:02:24 +01:00
exdatatest.c Update copyright year 2018-05-29 13:16:04 +01:00
exptest.c Use "" not <> for internal/ includes 2017-08-22 09:54:20 -04:00
fatalerrtest.c Update copyright year 2018-03-20 13:08:46 +00:00
generate_buildtest.pl Update copyright year 2018-05-29 13:16:04 +01:00
generate_ssl_tests.pl Consolidate the locations where we have our internal perl modules 2017-08-15 11:30:47 +02:00
gmdifftest.c Update the test framework so that the need for test_main is removed. Everything 2017-07-27 07:53:08 +10:00
gosttest.c Add a GOST test 2018-07-13 18:14:43 +01:00
handshake_helper.c Added missing signature algorithm reflection functions 2018-11-12 16:53:32 -05:00
handshake_helper.h Update copyright year 2018-03-20 13:08:46 +00:00
hmactest.c Use "" not <> for internal/ includes 2017-08-22 09:54:20 -04:00
ideatest.c Use "" not <> for internal/ includes 2017-08-22 09:54:20 -04:00
igetest.c Consistent formatting for sizeof(foo) 2017-12-07 19:11:49 -05:00
lhash_test.c Use "" not <> for internal/ includes 2017-08-22 09:54:20 -04:00
md2test.c Use "" not <> for internal/ includes 2017-08-22 09:54:20 -04:00
mdc2_internal_test.c Update copyright year 2018-05-29 13:16:04 +01:00
mdc2test.c Use "" not <> for internal/ includes 2017-08-22 09:54:20 -04:00
memleaktest.c Update secmemtest and memeleaktest to use the test infrastructure. 2017-04-12 10:59:53 +01:00
modes_internal_test.c Use void in all function definitions that do not take any arguments 2018-05-11 14:37:48 +02:00
ocspapitest.c Wrap more of ocspapitest.c in OPENSSL_NO_OCSP 2017-12-08 09:16:36 -06:00
P1ss.cnf
P2ss.cnf
packettest.c Update copyright year 2018-03-20 13:08:46 +00:00
pbelutest.c Update the test framework so that the need for test_main is removed. Everything 2017-07-27 07:53:08 +10:00
pemtest.c Update copyright year 2018-09-11 13:45:17 +01:00
pkcs7-1.pem
pkcs7.pem
pkey_meth_kdf_test.c Update copyright year 2018-05-29 13:16:04 +01:00
pkey_meth_test.c Update copyright year 2018-05-29 13:16:04 +01:00
pkits-test.pl Many spelling fixes/typo's corrected. 2017-11-11 19:03:10 -05:00
poly1305_internal_test.c Update copyright year 2018-02-13 13:59:25 +00:00
rc2test.c Use "" not <> for internal/ includes 2017-08-22 09:54:20 -04:00
rc4test.c Use "" not <> for internal/ includes 2017-08-22 09:54:20 -04:00
rc5test.c Use "" not <> for internal/ includes 2017-08-22 09:54:20 -04:00
rdrand_sanitytest.c Use void in all function definitions that do not take any arguments 2018-05-11 14:37:48 +02:00
README Fix test documentation. 2017-09-08 13:58:59 -05:00
README.external Many spelling fixes/typo's corrected. 2017-11-11 19:03:10 -05:00
README.ssltest.md Session resume broken switching contexts 2017-10-04 10:21:08 +10:00
recordlentest.c Update copyright year 2018-03-20 13:08:46 +00:00
rsa_complex.c Add a compile time test to verify that openssl/rsa.h and complex.h can 2018-09-17 09:44:45 +10:00
rsa_mp_test.c rsa/rsa_gen.c: harmonize keygen's ability with RSA_security_bits. 2017-11-28 20:05:48 +01:00
rsa_test.c Use "" not <> for internal/ includes 2017-08-22 09:54:20 -04:00
run_tests.pl Update copyright year 2018-05-01 13:34:30 +01:00
sanitytest.c Update copyright year 2018-09-11 13:45:17 +01:00
secmemtest.c test/secmemtest: test secure memory only if it is implemented 2018-10-05 12:23:34 +02:00
serverinfo.pem
serverinfo2.pem Add a SERVERINFOV2 format test file 2017-05-03 14:37:42 +01:00
servername_test.c Update copyright year 2018-09-11 13:45:17 +01:00
session.pem Don't store the ticket nonce in the session 2018-06-07 10:58:35 +01:00
shibboleth.pfx
shlibloadtest.c Extend dladdr() for AIX, consequence from changes for openssl#6368. 2018-08-22 21:50:33 +02:00
siphash_internal_test.c test/siphash_internal_test.c: ensure the SIPHASH structure is zeroed 2018-11-15 15:35:49 +01:00
sm2_internal_test.c Make SM2 ID stick to specification 2018-09-07 18:12:26 +08:00
sm4_internal_test.c SM4: Add SM4 block cipher to EVP 2017-10-31 15:19:14 +10:00
smcont.txt
srptest.c Update the test framework so that the need for test_main is removed. Everything 2017-07-27 07:53:08 +10:00
ssl_cert_table_internal_test.c Use void in all function definitions that do not take any arguments 2018-05-11 14:37:48 +02:00
ssl_test.c Add a config option to disable automatic config loading 2018-04-17 16:33:15 +02:00
ssl_test.tmpl
ssl_test_ctx.c Change Post Handshake auth so that it is opt-in 2018-08-20 15:14:01 +01:00
ssl_test_ctx.h Change Post Handshake auth so that it is opt-in 2018-08-20 15:14:01 +01:00
ssl_test_ctx_test.c Update copyright year 2018-03-20 13:08:46 +00:00
ssl_test_ctx_test.conf Implement Maximum Fragment Length TLS extension. 2017-11-05 17:46:48 +01:00
sslapitest.c Add a test for SSL_CTX_set0_CA_list()/SSL_CTX_set_client_CA_list() 2018-11-12 14:38:47 +00:00
sslbuffertest.c Update copyright year 2018-03-20 13:08:46 +00:00
sslcorrupttest.c Use void in all function definitions that do not take any arguments 2018-05-11 14:37:48 +02:00
ssltest_old.c Added missing signature algorithm reflection functions 2018-11-12 16:53:32 -05:00
ssltestlib.c Add a test for duplicated DTLS records 2018-10-26 14:28:18 +01:00
ssltestlib.h Add a test for duplicated DTLS records 2018-10-26 14:28:18 +01:00
Sssdsa.cnf
Sssrsa.cnf
stack_test.c Add a reserve call to the stack data structure. 2017-09-28 06:53:40 +10:00
sysdefault.cnf Apply system_default configuration on SSL_CTX_new(). 2018-03-19 10:22:49 -04:00
sysdefaulttest.c Update copyright year 2018-03-20 13:08:46 +00:00
test.cnf
test_test.c Relocate memcmp test. 2018-08-07 10:51:01 +10:00
testcrl.pem
testdsa.pem
testdsapub.pem
testec-p256.pem
testecpub-p256.pem
testp7.pem
testreq2.pem
testrsa.pem
testrsapub.pem
testsid.pem
testutil.h Test support for time_t comparisons. 2017-11-28 08:56:45 +10:00
testx509.pem
threadstest.c Update the test framework so that the need for test_main is removed. Everything 2017-07-27 07:53:08 +10:00
time_offset_test.c Update copyright year 2018-05-29 13:16:04 +01:00
tls13ccstest.c Use void in all function definitions that do not take any arguments 2018-05-11 14:37:48 +02:00
tls13encryptiontest.c Update the TLSv1.3 test vectors 2018-07-20 10:45:41 +01:00
tls13secretstest.c Update the TLSv1.3 test vectors 2018-07-20 10:45:41 +01:00
uitest.c [Win] Fix some test method signatures ... 2017-08-16 10:36:34 -04:00
Uss.cnf
v3-cert1.pem
v3-cert2.pem
v3ext.c Update the test framework so that the need for test_main is removed. Everything 2017-07-27 07:53:08 +10:00
v3nametest.c Use void in all function definitions that do not take any arguments 2018-05-11 14:37:48 +02:00
verify_extra_test.c Update copyright year 2018-05-01 13:34:30 +01:00
versions.c Refuse to run test_cipherlist unless shared library matches build 2018-03-31 16:40:07 +02:00
wpackettest.c Update the test framework so that the need for test_main is removed. Everything 2017-07-27 07:53:08 +10:00
x509_check_cert_pkey_test.c Update the test framework so that the need for test_main is removed. Everything 2017-07-27 07:53:08 +10:00
x509_dup_cert_test.c Update the test framework so that the need for test_main is removed. Everything 2017-07-27 07:53:08 +10:00
x509_internal_test.c Update copyright year 2018-05-29 13:16:04 +01:00
x509_time_test.c Update copyright year 2018-05-29 13:16:04 +01:00
x509aux.c Update copyright year 2018-06-20 15:29:23 +01:00

How to add recipes
==================

For any test that you want to perform, you write a script located in
test/recipes/, named {nn}-test_{name}.t, where {nn} is a two digit number and
{name} is a unique name of your choice.

Please note that if a test involves a new testing executable, you will need to
do some additions in test/Makefile.  More on this later.


Naming conventions
=================

A test executable is named test/{name}test.c

A test recipe is named test/recipes/{nn}-test_{name}.t, where {nn} is a two
digit number and {name} is a unique name of your choice.

The number {nn} is (somewhat loosely) grouped as follows:

00-04  sanity, internal and essential API tests
05-09  individual symmetric cipher algorithms
10-14  math (bignum)
15-19  individual asymmetric cipher algorithms
20-24  openssl commands (some otherwise not tested)
25-29  certificate forms, generation and verification
30-35  engine and evp
60-79  APIs
   70  PACKET layer
80-89  "larger" protocols (CA, CMS, OCSP, SSL, TSA)
90-98  misc
99     most time consuming tests [such as test_fuzz]


A recipe that just runs a test executable
=========================================

A script that just runs a program looks like this:

    #! /usr/bin/perl

    use OpenSSL::Test::Simple;

    simple_test("test_{name}", "{name}test", "{name}");

{name} is the unique name you have chosen for your test.

The second argument to `simple_test' is the test executable, and `simple_test'
expects it to be located in test/

For documentation on OpenSSL::Test::Simple, do
`perldoc util/perl/OpenSSL/Test/Simple.pm'.


A recipe that runs a more complex test
======================================

For more complex tests, you will need to read up on Test::More and
OpenSSL::Test.  Test::More is normally preinstalled, do `man Test::More' for
documentation.  For OpenSSL::Test, do `perldoc util/perl/OpenSSL/Test.pm'.

A script to start from could be this:

    #! /usr/bin/perl

    use strict;
    use warnings;
    use OpenSSL::Test;

    setup("test_{name}");

    plan tests => 2;                # The number of tests being performed

    ok(test1, "test1");
    ok(test2, "test1");

    sub test1
    {
        # test feature 1
    }

    sub test2
    {
        # test feature 2
    }


Changes to test/build.info
==========================

Whenever a new test involves a new test executable you need to do the
following (at all times, replace {NAME} and {name} with the name of your
test):

* add {name} to the list of programs under PROGRAMS_NO_INST

* create a three line description of how to build the test, you will have
to modify the include paths and source files if you don't want to use the
basic test framework:

    SOURCE[{name}]={name}.c
    INCLUDE[{name}]=.. ../include
    DEPEND[{name}]=../libcrypto libtestutil.a

Generic form of C test executables
==================================

    #include "testutil.h"

    static int my_test(void)
    {
        int testresult = 0;                 /* Assume the test will fail    */
        int observed;

        observed = function();              /* Call the code under test     */
        if (!TEST_int_equal(observed, 2))   /* Check the result is correct  */
            goto end;                       /* Exit on failure - optional   */

        testresult = 1;                     /* Mark the test case a success */
    end:
        cleanup();                          /* Any cleanup you require      */
        return testresult;
    }

    int setup_tests(void)
    {
        ADD_TEST(my_test);                  /* Add each test separately     */
        return 1;                           /* Indicate success             */
    }

You should use the TEST_xxx macros provided by testutil.h to test all failure
conditions.  These macros produce an error message in a standard format if the
condition is not met (and nothing if the condition is met).  Additional
information can be presented with the TEST_info macro that takes a printf
format string and arguments.  TEST_error is useful for complicated conditions,
it also takes a printf format string and argument.  In all cases the TEST_xxx
macros are guaranteed to evaluate their arguments exactly once.  This means
that expressions with side effects are allowed as parameters.  Thus,

    if (!TEST_ptr(ptr = OPENSSL_malloc(..)))

works fine and can be used in place of:

    ptr = OPENSSL_malloc(..);
    if (!TEST_ptr(ptr))

The former produces a more meaningful message on failure than the latter.