I3Test Reference¶
Overview¶
The best way to verify the correctness of an implementation is to fully exercise all functionality with an independent implementation. “Thorough” means: a run of the test suite to completion without error should be enough to convince an aggressive skeptic that the implementation is correct. The independent implementation itself should be so simple that it is plausibly correct. (This specification from the Library of Practical Abstractions manual, see References at the bottom of this document.)
One should provide test suites for all code. Test suites should exercise the interface of a class in normal and borderline cases. They should demonstrate what it is possible to do with a class, and that a class behaves reasonably (correctly signals error conditions) when misused.
A test suite should run to completion in a reasonable amount of time. The complexity of a test suite will of necessity be proportional to the complexity of the behavior of what it is testing. One of the main guidelines of object-oriented design is “one class, one responsibility”. A class’ tests verify its ability to assume this responsibility: if it is hard to write a set of tests for a class because it does so many things, this indicates that the class should probably be broken up into smaller pieces, each with a clearly defined purpose and interface.
I3Test allows anyone capable of running the command make
to
identify at a glance exactly what failed, including file, line number,
the expression that failed, and optional user-defined
messages. Intimate knowledge of the software under testing is
unnecessary. It provides a uniform interface to testing. This allows
developers to verify that they have not broken one anothers’ code when
modify their own, and allows you to do the same. It also reduces the
amount of “boilerplate” required to write tests.
Test suites as documentation, and insurance¶
Test suites serve several additional purposes. They allow people other than the original author to fix bugs in modules with confidence that they have not introduced additional bugs. They also serve as excellent documentation for how to use modules: if a testsuite is thorough, it demonstrates normal uses, borderline cases and errors, and can be an excellent reference if you are trying to figure out how to use the class yourself.
They have the additional advantage over documentation that they will be verified to work before each release. Unlike documentation, they cannot get out of sync with the actual code.
Time invested in writing test suites is well invested.
Regression tests: submit bugfixes with a test for the bug.¶
This isn’t regression as in ‘least squares curve fitting’, this is regression as the opposite of progression. Bugs reappear. Regression tests provide a way to automatically know what bugs have done so. Often the easiest way to track down a bug is to write a little program that produces it, and then hack at the code exercised by the little program, until the error condition is fixed.
This little program is precisely a regression test. If you stop using it when you have fixed the bug, you have wasted your time and created conditions where you will likely have to track down the bug again in the future. Formulate this little program as a test, however, and it will continue to work for you for the rest of the life of your code. As you encounter and fix bugs in your code, your base of testsuites will grow, and the amount of time you spend debugging will shrink.
Testing Interface¶
Example¶
Here is a file containing test group named example_group containing four unit tests named it_works, pinkness_is_4, pinkness_is_almost_pi, and this_one_fails
#include <I3Test.h>
TEST_GROUP(example_group);
TEST(it_works)
{
bool it_works = true;
ENSURE(it_works);
}
TEST(pinkness_is_4)
{
double pinkness = 4.0;
double redness = 4.0;
ENSURE_EQUAL(pinkness, redness, "pinkness and redness not equal");
}
TEST(pinkness_is_almost_pi)
{
double pinkness = 4;
ENSURE_DISTANCE(pinkness, M_PI, 1.0, "pinkness not within 1 of pi");
}
TEST(this_one_fails)
{
FAIL("this will fail unconditionally");
}
The unit tests signal success or failure using the statement
ENSURE()
, ENSURE_DISTANCE()
, or FAIL()
.
Build configuration¶
A test group is contained in one file, which contains any number of
uniquely named unit tests. A directory containing one or more test
groups, plus a main()
routine, is called a test suite. One informs
the build system which directories contain test groups via
i3_test_executable() (in the project’s
CMakeCache.txt
).
One of these files must contain the main routine for the test
driver. To do so, place one file (main.cxx
is a reasonable name) in
the test suite directory containing only the line:
#include <I3TestMain.ixx>
This I3TestMain.ixx
contains what is required.
Each test group (file) must include the statement:
#include <I3Test.h>
which pulls in definitions for TEST_GROUP
, ENSURE
etc. (see below).
-
TEST_GROUP
Valid context: toplevel scope of implementation files in test suite directory.
This signals to the build system that this file contains a TEST_GROUP with the name GROUPNAME. GROUPNAME must be a valid C++ identifier name (alphanumeric plus underscore, starting with letter or underscore.)
The individual unit tests found in this file will be organized under this test group.
-
TEST
Valid context: toplevel scope of implementation files in test suite directory.
TEST(testname)
defines a unit test named testname. As withTEST_GROUP()
, testname must be a valid C++ identifier name.TEST(t)
looks much like a function definition (which it becomes afterTEST(t)
is expanded by the preprocessor). It is immediately followed by a scope (open curly-brace, statements, close curly-brace), containing testing statements.
-
ENSURE
ENSURE
is analogous toassert()
. It takes one argument, a predicate, and an optional comment.ENSURE
checks whether the predicate is true or false. If it is false, it will return a test failure. If the optional comment is included as a string, it will return it along with the failure.Note
A predicate is simply an expression that evaluates to true or false. Comparisons of two numbers, e.g. numhits>50 or pinkness=4, a call of a function that returns bool or something convertible to bool, e.g.
hits_are_ok();
, and evaluation of pointers for nullness are all predicates.
-
ENSURE_EQUAL
ENSURE_EQUAL
ensures that left-value and right-value are, well, equal. If they aren’t, it will throw a test failure. If the optional comment is specified, the failure will come with that message.Why not just:
ENSURE(something == something_else);
you ask. If this ENSURE fails, you will get only the error message:
File: private/test/sample_error.cxx Line: 17 Predicate: something == something_else
Whereas ENSURE_EQUAL will show:
File: private/test/SampleClassTest.cxx Line: 24 Predicate: ensure_distance: expected [0.16665;0.16667] actual 0.3333333333
Which gives one a better idea what the problem is.
-
ENSURE_DISTANCE
ENSURE_DISTANCE
verifies that left-value is within distance of right-value. If it is not, it throws a test failure. If the optional comment is specified, the failure will come withthat message.
-
FAIL
This fails unconditionally. If the optional comment is specified, the failure will come withthat message.
FAIL
is useful when verifying that an operation should throw, or that a certain point in the code is never reached:try { operation_that_throws(); FAIL("that should have thrown"); } catch (const std::exception& e) { // NB: DO NOT catch-all, meaning NEVER, EVER, EVER 'catch(...)' // In that case your test will always pass and test nothing. // OK. That operation threw and therefore we didn't hit FAIL() }
Even better:
EXPECT_THROW(operation_that_throws);
Building and running I3Tests¶
See Targets for how to build and run. Once built the test driver binaries
will be in the bin/
directory of the build.
The test driver command line interface¶
One may also run tests manually. The test driver programs have command-line help and a robust set of arguments. With no arguments, a test driver will show the following help:
% icetray-test --help
I3 Test Suite Options Summary:
-h [ --help ] : this message
-f [ --fork ] : fork() before each test (all tests will run
even if one dumps core)
-a [ --all ] : run all tests
-x [ --xml ] arg : run all tests, output xml to FILE (implies --fork
and --timeout 600)
-s [ --show-spew ] : pass along any thing the test spews
to cout/cerr (you want to see it)
-l [ --list ] : list tests and groups in this suite
--run-tests arg : list of tests to run
-t [ --timeout ] arg : timeout unit tests after this many seconds
One may specify short or long arguments in typical UNIX command-line utility style. Test driver arguments in detail (default: off)
References¶
Eric Sven Ristad and Peter N. Yianilos
Library of Practical Abstractions
http://www.pnylab.com/pny/software/libpa/main.html
February 1998