certify [-c db] [-aACdDefiGIklmMnpPQqrRsuUvWVZ] [-H dir] [-w msg] [-x path] [-X results] [testid ...]
A few directories and naming conventions are standard. Each test has an identifying short string, the testid.
Stdin input for a particular test (if required at all) is kept in input/testid or input/testid/testid (if multiple input files are needed). Other data may be kept in the data directory, where it might be shared by multiple tests. The files in the directories input and data should be read-only, and test programs should not alter the files or write to the directory.
Output from test programs is kept in the directory results, either as results/testid if there is only stdout and stderr output, or in the directory results/testid if multiple output files are required. In this case, the stdout output is put in results/testid/testid, and the stderr output is put into results/testid/testid.stderr. This separation is important because otherwise the two streams can be intermingled in fairly random ways.
The correct results from a test program execution are kept in the directory correct, either as correct/testid if only the stdout/stderr output must be tested, or in the directory correct/testid if there are multiple output files.
There are variants on the correct directory, for tests where the result is quite different from one machine to another. certify recognizes the following variations, in this order:
where:
If the test program must write to its input files, these files should be copied to results/testid directory before the program is run. Any such preparation of the directory results/testid should be done by the pre command specified in the database.
The certify table of the database specifies the tests to perform, and is most conveniently created with dbe. Each record in the certify database specifies a testid, a description, a program, program arguments, a pre command (mentioned above), an expected return value and a diff command. In the minimal case, only a testid and command need be specified in the database, though a description is handy later. The pre command is null by default, and the default diff is to run diff to compare the contents of results/testid with the content of correct/testid.
In addition, there are require and avoid fields for restricting the
situations where certain tests are run. certify creates a characterization
of the current environment, consisting of the hostname, user, os (Linux or Darwin),
the machine architecture (x86 or arm), adds f77 if
a fortran compiler named f77 or gfortran is present.
adds maps if the mapdata database is present,
adds regions if the regions database is present,
adds iasp91 if the iasp91 tau-p travel time tables are present,
and X if the environment variable DISPLAY is defined.
If the requires field has f77 in it, then the corresponding test is
run only if a fortran compiler is present.
If the the avoid field has arm in it, then this test is not run on the
marmot.
Defined Characters | |
name | meaning |
f77 | has fortran |
matlab | has matlab |
-t | stdin is a tty (interactive mode) |
maps | mapdata file is present |
regions | grnames.fer is present |
iasp91 | iasp91 travel time table is present |
X | X DISPLAY is defined |
To perform each test, certify:
In each of the pre, diff, cleanup and the actual command, you may use shorthand percent codes as shown in the table below. Note, however, that the path to data is modified when cd is specified in the certify record, but the pre, diff and cleanup commands are run from the current directory.
%r | result directory ("results/testid") |
%R | results output file ("results/testid/testid") |
%c | correct directory ("correct/testid") |
%C | correct output file ("correct/testid/testid") |
%d | relative path to ./data (modified appropriately when cd is true) |
%D | path to testdata ("/opt/antelope/testdata") |
%I | unique id |
%t | testid |
The correct result directory or file may optionally be modified by a suffix of the OS (e.g., SunOS, Linux), the OS.Release, (e.g., SunOS.5.8), or the processor (e.g., sun4u, i86pc, i686). This helps to handle cases where the results differ among different machines.
At each step, return codes are tested, and the test fails and is halted at any step with an unexpected return code. The pre and diff commands should return 0 for success. The successful return value for the test comand is specified in the database, but is zero by default.
Failures are noted on stdout. Results are recorded in the history table, which contains the time of the most recent success and most recent failure, a count of successes and failures, and the most recent execution time.
An entire suite of tests in a directory can be suppressed (for a recursive certify execution) by adding an empty file certify.disable in the directory.
% cronrun -v `which certify` -H $cwd simple
certify database tables: certify, history, log, and updates.
The empty results directories keep a dummy file .keep_for_git so that git preserves the directories.
The '**' indicates the test failed last time it was run for this variety.% certify -l **tbl exhaustive test of lists stbl exhaustive test of sorted lists glob simple test of glob zopen simple test of zopen shell simple test of shell sort search simple test of binary search patsub simple test of pattern substitution regex simple test of regular expressions pfcheck try parameter file routine arr exhaustive test of arrays isort test insertion sort getopt try getopt with long options quicksort test quick sort error test error routines rb test red-black tree routines
% certify -v tbl : ttbl 25 stbl : tstbl glob : tryglob results differ: diff -r zopen : zopen results/zopen 2c t1.Z t2 t3.gz t4 2c.Z shell : tryshell search : trysearch patsub : try_patsub regex : try_regex pfcheck : pfcheck results/pfcheck short.out arr : tarr isort : tisort getopt : getopt1 -add add -append -delete delete -verbose -create -file file -abc c -d d -e e -0123456789 extra quicksort : tquicksort error : terror rb : rb input/rb 1 failures
xdiff(1) cmplcnt(1) cntlines(1)
It's better to do a lot of short tests which each focus on a single operation, than to make a gigantic test. With a large test, the first problem is figuring out what exactly failed; with a more focused test, the problem should at least be obvious.
Deciding when to run a test is done by using make(1) to determine when a program is out of date. This isn't quite right.
It's important to avoid conflicts between the program's normal output files, and the stdout/stderr output file created by certify
The history database should indicate the times when the correct results are updated, perhaps with some message indicating why.