Re: michael/wjr merge

[Michael McClary <michael>]

>       - Though in a file whose name implies it is a regression test, the
> 	test is actually a test-prod test, which tests only one aspect of
> 	the module.
> 
> What's a "test-prod test"?

A test which tests only one aspect of the module.

It is more a pattern of use than anything peculiar to the test itself.

If you:

 - write a program that tests the first thing you want to test, then

 - add code to test the second thing you want to test (leaving the
   first test in place, and {if the test has output} let a "diff"
   against a reference file suppress the output from the first test
   unless it changes), then

 - add code to test the third thing you want to test (leaving the
   first two tests in place...)

you have either a regression or a pass-fail test.  (The distinction:
A regression test creates a "signature" which is a function of the
desired behavior of the module, and compares it to a reference.  A
pass-fail test "understands" the desired behavior and just says yes
or no.)

 - A regression test will inform you at any stage if a change has broken
   something you've already tested.

 - Each of its subtests will have been written when the programmer had
   a clear idea of what needed to be tested.

 - It forms a permanent record of the correct behavior of the module,
   which can be used to insure that ports to another environment have been
   successful, and that the module itself has not accidentially been
   replaced by an earlier version which does NOT work correctly.

 - It can be automatically administered, with no human attention required
   unless something fails.

 - Though the output of the test proper may be large, the programmer will
   see only the output of the new subtest he's adding, unless one of the
   other subtests begins to fail.

 - Regression tests are not interactive.

If you:

 - write a program that tests the first thing you want to test, then

 - change it to test the second thing you want to test, then

 - change it to test the third thing you want to test, then...

you have a test-prod test.

It is like testing a piece of electronic cirtuitry by using a (set of)
test-prods to connect a test instrument to one part of the circuit and
measuring what it happening there, then moving the test-prod to a
different part of the circuit.  And it has all the advantages and
disadvantages of such a test procedure:

 - The "test prods" may be applied to the "connectors", or they may
   violate the normal boundaries of the module and test internal
   state and activity.

    - "Opening" the "covers" (lowering protection barriers) may
      have side-effects which must be taken into account.

    - Temporary changes may be made to the module to allow
      access, without adding permanent overhead to the module.
      (When making only one test, you can probably remember to
      unmake all the changes.)

 - Once the "test-prod" is moved to another "test point", the test
   it was performing is lost.

    - It is no longer being performed.  If the thing it was testing breaks
      later, this will not be detected.

    - Before it will be performed again, a skilled programmer must:
       - Suspect it SHOULD be performed.
       - Determine how to perform it.
       - Write a replacement test or recover the original from manuals,
	 commented-out code, or whatnot.
       - Determine what the correct output should be.

 - Test-prod tests can be interactive.  (adb and dbx are test-prod programs)

Both test-prod tests and regression tests have a place in a software QA
program.  Pass-fail tests do not.

A regression test assures a module is working correctly (and has not
reverted to a previous version that works correctly in most cases).
It MAY tell you what is broken, but that is not its primary purpose.

(The regression test is your software agent for testing future releases:
It should look at everything >you< had to look at to assure yourself the
first implementation of the module was operating correctly.  If you
construct it as you write the module it tests, and use it as your only
pass/fail test on the original module, this happens automatically.)

Once errors are detected, a test-prod test may be an appropriate tool to
trace down what's wrong.  It should never be used to verify correctness,
or to detect that errors exist.

A test-prod test may be standalone, but it is usually convenient to use
the regression test as a platform for it, for two reasons:

 - The regression test already has most of the necessary hooks for getting
   into the module and talking about it on the I/O.  Mutating it is usually
   much easier than writing a test-prod test from scratch.

 - The fact that you had to look at something with a test-prod test is a
   hint that the regression test should look at it, too.  (The test-prod
   test may have been the regression subtest you were about to write when
   you finished with THIS one - but you had to write it now to find out
   why THIS test wasn't working.)  When this happens, you can add it to
   the regression test by updating the reference file.

A pass-fail test is appropriate to a manufacturing plant, where it flags
defective instances of the product.  Software replicates perfectly, and
is not subject to such failures.  Software errors occur as a result of
upgrades (model changeover) or porting (building a new plant).  In these
situations, a pass-fail test suffers from the "who tests the testers"
infinite regress, and requires nearly as much attention as writing a new
test from scratch.  (This is one reason Detroit is so reluctant to make
any major changes in design.)

Though a pass-fail test may test exactly the same things as a regression
test, it does it in a different way, and is more labor-intensive and
error prone for several reasons:

 - The operation of a pass-fail test is more closely analogous to the
   operation of the thing it is testing than the operation of a regression
   test.  This makes canceling pairs of errors much more likely.

 - When porting, the pass-fail test and the module it tests are also more
   subject to canceling pairs of errors, introduced either by a human or
   a translating program, than the module plus a regression test.  (The
   signature is unchanged.)

 - A pass-fail test gives no external indication of the revision level of
   the module under test, the tester itself, how much was tested, whether
   anything was tested at all, or whether the tester and the tested module
   reverted to a previous version together.   Two program modules are more
   likely to regress together than two program modules and a text file, and
   the latter case is easier to detect.

 - With a regression test, you examine the signature to see if the module
   under test (and the test) behave(s) correctly.  With a pass-fail test,
   you must examine the test module's code to see if it does the right
   thing.  (If you could do that, why didn't you just do it to the original
   code?  Why do you need a test at all?  B-) )
   
   By examining both code and data, rather than two pieces of code, you
   are thinking about the problem in two categories rather than one, and
   are much more likely to catch any errors.

 - With a regression test, the signature becomes the "true specification",
   rigorously enforced by automatic processes.  With a pass-fail test,
   the "true specification" is encoded in the structure of the test program.
   This is not much better than an untested module, where the "true
   specification" is the module itself.

Some random notes on testing:

Regression-test signature output only needs to be machine-independent
when the test is succeeding.

A no-load test jack (in C++) is:

	friend class FooJig;

which allows you to write a subclass which can reach into Foo's private
variables, yet doesn't add any code to the production module.

	michael