With the perl test framework comes the output format TAP
(Test Anything Protocol, see http://testanything.org/) with
extra extension for subtests. This change extends that same
output format to any test program using testutil.
In this implementation, each test program is seen as a full test that
can be used as a subtest. The perl framework passes on the subtest
level to the test programs with the environment variable
HARNESS_OSSL_LEVEL. Furthermore, and series of tests added with
ADD_ALL_TESTS is regarded as another subtest level.
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3296)
If $_ is not private, it can wipe caller's one, which proved to be
problematic...
Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
95-test_external_boringssl.t had a specialised run() variant to prefix
the command output so it wouldn't disturb Test::Harness. This
functionality if now moved to the run() command, using the added
option 'prefix' that can be set to the string to prefix the output
with.
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3201)
Fix some comments too
[skip ci]
Reviewed-by: Tim Hudson <tjh@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3069)
When using run() with capture => 1, there was no way to find out if
the command was successful or not. This change adds a statusvar
option, that must refer to a scalar variable, for example:
my $status = undef;
my @line = run(["whatever"], capture => 1, statusvar => \$status);
$status will be 1 if the command "whatever" was successful, 0
otherwise.
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3004)
This data directory is formed automatically by taking the recipe name
and changing '.t' to '_data'. Files in there can be reached with the
new function data_file()
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2027)
So far, apps and test programs, were a bit rigidely accessible as
executables or perl scripts. But what about scripts in some other
language? Or what about running entirely external programs? The
answer is certainly not to add new functions to access scripts for
each language or wrapping all the external program calls in our magic!
Instead, this adds a new functions, cmd(), which is useful to access
executables and scripts in a more generalised manner. app(), test(),
fuzz(), perlapp() and perltest() are rewritten in terms of cmd(), and
serve as examples how to do something similar for other scripting
languages, or constrain the programs to certain directories.
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1686)
The previous fix contained a mistake, where any absolute path in
%directories would be cleared away instead of just being left alone.
Reviewed-by: Rich Salz <rsalz@openssl.org>
We recalculate the location of the directories we keep track of.
However, we did so after having moved to the new directory already, so
the data we did the calculations from were possibly not quite correct.
This change moves the calculations to happen before moving to the new
directory.
This issue is sporadic, and possibly dependent on the platform.
Reviewed-by: Matt Caswell <matt@openssl.org>
If we're going to redirect STDERR and STDOUT, it's better to do it by
the book. This fix is a straight copy of examples in the perlfunc
manual.
Reviewed-by: Matt Caswell <matt@openssl.org>
In rare cases, the shell we run test programs in may have complaints.
Shut those up unless testing verbosely.
Reviewed-by: Viktor Dukhovni <viktor@openssl.org>
The environment variables TOP, SRCTOP, BLDTOP, ... are used to affect
the testing framework. However, subprocesses may want to use them as
well, and therefore need their values corrected when we move to a
different directory.
Reviewed-by: Emilia Käsper <emilia@openssl.org>
Within OpenSSL::Test, all commands end up existing in two variants,
one that has redirections that are needed internally to work well
together with the test harness, and one without those redirections.
Depending on what the result is going to be used for, the caller may
want one for or the other, so we give them the possibility.
Reviewed-by: Rich Salz <rsalz@openssl.org>
Because some operating systems have executable extensions, typically
".exe", we need to append it when looking for files in test() and
app() (or rather, their subroutines).
Reviewed-by: Rich Salz <rsalz@openssl.org>
If the command file that app(), test(), perlapp(9 and perltest() are
looking for doesn't exist in the build tree, look for it in the source
tree as well.
Reviewed-by: Emilia Käsper <emilia@openssl.org>
$? in perl gets the status value from wait(2), which is a word with
the exit code in the upper half and the number of a raised signal in
the lower half. OpenSSL::Test::run() ignored the signal half up until
now.
With this change, we recalculate an exit code the same way the Unix
shells do, using this formula:
($? & 0x7f) ? ($? & 0x7f)|0x80 : ($? >> 8);
Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Emilia Käsper <emilia@openssl.org>
It was unexpected that OpenSSL::Test::setup() should be called twice
by the same recipe. However, that may happen if a recipe combines
OpenSSL::Test and OpenSSL::Test::Simple, which can be a sensible thing
to do. Therefore, we now allow it.
Reviewed-by: Rich Salz <rsalz@openssl.org>
The logging that was performed in OpenSSL::Test was initially set up
as a means not to let messages that test programs write to STDERR get
displayed when a test isn't running in verbose mode. However, the way
it was implemented, it meant that those messages were never displayed,
and you had to look in a test log. This also meant that output to
STDERR and output to STDOUT got broken apart, which isn't optimal.
So, we remove the whole test log file implementation, and instead,
we're sending STDERR to the null device unless one of these conditions
apply:
- the test recipe already redirects stderr. Just let it.
- the environment variable HARNESS_ACTIVE is undefined, meaning the
recipe is run directly as a perl script instead of being harnessed
by Test::Harness
- the environment variable HARNESS_VERBOSE is set.
Getting a full log of the tests now becomes as simple as this:
HARNESS_VERBOSE=yes make test 2>&1 | tee tests.log
Reviewed-by: Rich Salz <rsalz@openssl.org>
To be able to run tests when we've built in a directory other than
the source tree, the testing framework needs a few adjustments.
test/testlib/OpenSSL/Test.pm needs to know where it can find
shlib_wrap.sh, and a number of other tests need to be told a different
place to find engines than what they may be able to figure out on
their own. Relying to $TOP is not enough, $SRCTOP and $BLDTOP can be
used as an alternative.
As part of this change, top_file and top_dir are removed and
srctop_file, bldtop_file, srctop_dir and bldtop_dir take their place.
Reviewed-by: Ben Laurie <ben@openssl.org>
Just like File::Path::make_path, File::Path::remove_tree didn't show
up before File::Path 2.06 / perl v5.10.1, so we prefer the legacy
function here as well.
Reviewed-by: Andy Polyakov <appro@openssl.org>
This uilds on the same way of checking for availability as we do in
TLSProxy. We use all IP factories we know of, starting with those who
know both IPv6 and IPv4 and ending with the one that only knows IPv4
and cache their possible success as foundation for checking the
available of each IP domain.
80-test_ssl.t has bigger chances of working on platforms that do not
run both IP domains.
Reviewed-by: Rich Salz <rsalz@openssl.org>
$EXE_SHELL should only be used with out own programs, not with
surrounding programs such as the perl interpreter.
Reviewed-by: Matt Caswell <matt@openssl.org>
Since we're building some of our perl scripts and the result might not
end up in apps/ (*), we may need to treat them like the compile
programs we use for testing.
This introduces perlapp() and perltest(), which behave like app() and
test(), but will add the perl executable in the command line.
-----
(*) For example, with a mk1mf build, the result will end up in $(BIN_D)
Reviewed-by: Matt Caswell <matt@openssl.org>
Enhances the routines in OpenSSL::Test::Utils for checking disabled
stuff to get their information directly from Configure instead of
'openssl list -disabled'.
Reviewed-by: Viktor Dukhovni <viktor@openssl.org>
On VMS, the command MCR will assume SYS$SYSTEM: when the first
argument lacks a directory spec. So for programs in the current
directory, we add [] to tell MCR it is in the current directory.
It's the same as having ./ at the start of a program on Unix so the
shell doesn't start looking along $PATH.
Reviewed-by: Rich Salz <rsalz@openssl.org>
Add Utils.pm for test utilities. This currently just contains one function:
disabled which checks if a feature is disabled based on the output of
openssl list -disabled
Reviewed-by: Richard Levitte <levitte@openssl.org>
If a test recipe does something like this:
indir "foo.$$" => sub {
chmod 0500, File::Spec->curdir();
ok(run(app(["something"])));
}
we get a problem, because we were storing the temporary stderr file in
the current directory at all times (so while inside the 'indir', we
would attemp to store it in "foo.$$").
So, change our ways to always store that temporary file in the exact
same location, defined by the environment variable RESULT_D, or
failing that TEST_D, or failing that $TOP/test.
Reviewed-by: Rich Salz <rsalz@openssl.org>
Be careful when shifting in a function argument, you end up changing
the caller's value. Instead, when it is an array, make a shallow copy
and shift in that instead.
Reviewed-by: Rich Salz <rsalz@openssl.org>
