• Antelope Release 5.10 Linux CentOS release 7.6.1810 (Core) 3.10.0 2020-05-12

 

NAME

certify - run validity tests and record results

SYNOPSIS

certify [-c db] [-aACdDefiGIklmMnpPQqrRsuUvWVZ] [-H dir] [-w
                msg] [-x path] [-X results] [testid ...]

DESCRIPTION

certify manages small databases of regression tests which can be run repeatedly to verify to some degree at least that software is running properly. The basic idea is to specify a command line with some specified input which generates some output. The output data is compared with output data saved from an earlier run; any differences constitute a failure, while an exact comparison is success. The most recent successes and failures are recorded in the database.

A few directories and naming conventions are standard. Each test has an identifying short string, the testid.

Stdin input for a particular test (if required at all) is kept in input/testid or input/testid/testid (if multiple input files are needed). Other data may be kept in the data directory, where it might be shared by multiple tests. The files in the directories input and data should be read-only, and test programs should not alter the files or write to the directory.

Output from test programs is kept in the directory results, either as results/testid if there is only stdout and stderr output, or in the directory results/testid if multiple output files are required. In this case, the stdout output is put in results/testid/testid, and the stderr output is put into results/testid/testid.stderr. This separation is important because otherwise the two streams can be intermingled in fairly random ways.

The correct results from a test program execution are kept in the directory correct, either as correct/testid if only the stdout/stderr output must be tested, or in the directory correct/testid if there are multiple output files.

There are variants on the correct directory, for tests where the result is quite different from one machine to another. certify recognizes the following variations, in this order:

where:

If the test program must write to its input files, these files should be copied to results/testid directory before the program is run. Any such preparation of the directory results/testid should be done by the pre command specified in the database.

The certify table of the database specifies the tests to perform, and is most conveniently created with dbe. Each record in the certify database specifies a testid, a description, a program, program arguments, a pre command (mentioned above), an expected return value and a diff command. In the minimal case, only a testid and command need be specified in the database, though a description is handy later. The pre command is null by default, and the default diff is to run diff to compare the contents of results/testid with the content of correct/testid.

In addition, there are require and avoid fields for restricting the situations where certain tests are run. certify creates a characterization of the current environment, consisting of the hostname, user, os (Linux or Darwin), the machine architecture (x86 or arm), adds f77 if a fortran compiler named f77 or gfortran is present. adds maps if the mapdata database is present, adds regions if the regions database is present, adds iasp91 if the iasp91 tau-p travel time tables are present, and X if the environment variable DISPLAY is defined. If the requires field has f77 in it, then the corresponding test is run only if a fortran compiler is present. If the the avoid field has arm in it, then this test is not run on the marmot.

Defined Characters
name meaning
f77 has fortran
matlab has matlab
-t stdin is a tty (interactive mode)
maps mapdata file is present
regions grnames.fer is present
iasp91 iasp91 travel time table is present
X X DISPLAY is defined

To perform each test, certify:

In each of the pre, diff, cleanup and the actual command, you may use shorthand percent codes as shown in the table below. Note, however, that the path to data is modified when cd is specified in the certify record, but the pre, diff and cleanup commands are run from the current directory.

%r result directory ("results/testid")
%R results output file ("results/testid/testid")
%c correct directory ("correct/testid")
%C correct output file ("correct/testid/testid")
%d relative path to ./data (modified appropriately when cd is true)
%D path to testdata ("/opt/antelope/testdata")
%I unique id
%t testid

The correct result directory or file may optionally be modified by a suffix of the OS (e.g., SunOS, Linux), the OS.Release, (e.g., SunOS.5.8), or the processor (e.g., sun4u, i86pc, i686). This helps to handle cases where the results differ among different machines.

At each step, return codes are tested, and the test fails and is halted at any step with an unexpected return code. The pre and diff commands should return 0 for success. The successful return value for the test comand is specified in the database, but is zero by default.

Failures are noted on stdout. Results are recorded in the history table, which contains the time of the most recent success and most recent failure, a count of successes and failures, and the most recent execution time.

An entire suite of tests in a directory can be suppressed (for a recursive certify execution) by adding an empty file certify.disable in the directory.

OPTIONS

FILES

certify tests for the existence of a file ~/.out; if this file exists and certify is not being run interactively, it exits immediately. This is to suppress crontab jobs when out of town.

certify database tables: certify, history, log, and updates.

The empty results directories keep a dummy file .keep_for_git so that git preserves the directories.

ENVIRONMENT

see antelopeenv(5)

PARAMETER FILE

The certify parameter file specifies the environment in which certify runs, after the fashion of rtexec(1), using the Defines, Env and Limit arrays.

EXAMPLE


% certify -l
**tbl   exhaustive test of lists
  stbl  exhaustive test of sorted lists
  glob  simple test of glob
  zopen simple test of zopen
  shell simple test of shell sort
  search        simple test of binary search
  patsub        simple test of pattern substitution
  regex simple test of regular expressions
  pfcheck       try parameter file routine
  arr   exhaustive test of arrays
  isort test insertion sort
  getopt        try getopt with long options
  quicksort     test quick sort
  error test error routines
  rb    test red-black tree routines

The '**' indicates the test failed last time it was run for this variety.


% certify -v
tbl : ttbl 25
stbl : tstbl
glob : tryglob
        results differ: diff -r
zopen : zopen results/zopen 2c t1.Z t2 t3.gz t4 2c.Z
shell : tryshell
search : trysearch
patsub : try_patsub
regex : try_regex
pfcheck : pfcheck results/pfcheck short.out
arr : tarr
isort : tisort
getopt : getopt1 -add add -append -delete delete -verbose -create -file file -abc c -d d -e e -0123456789 extra
quicksort : tquicksort
error : terror
rb : rb input/rb
1 failures

RETURN VALUES

certify returns the number of failures.

DIAGNOSTICS

SEE ALSO

xdiff(1)
cmplcnt(1)
cntlines(1)

BUGS AND CAVEATS

It's better to do a lot of short tests which each focus on a single operation, than to make a gigantic test. With a large test, the first problem is figuring out what exactly failed; with a more focused test, the problem should at least be obvious.

Deciding when to run a test is done by using make(1) to determine when a program is out of date. This isn't quite right.

It's important to avoid conflicts between the program's normal output files, and the stdout/stderr output file created by certify

The history database should indicate the times when the correct results are updated, perhaps with some message indicating why.

AUTHOR

Daniel Quinlan
Printer icon