Overeasy: Racket Language Test Engine
License: LGPL 3 Web: http://www.neilvandyke.org/overeasy/
| (require (planet neil/overeasy:1:=0)) |
1 Introduction
Overeasy is a software test engine for the Racket programming language. It designed for all of:
rapid interactive testing of expressions in the REPL;
unit testing of individual modules; and
running hierarchical sets of individual module unit tests at once.
An individual test case, or test, is specified by the programmer with the test syntax, and evaluation of that syntax causes the test to be run. Properties that are checked by tests are:
values of expressions (single value, or multiple value);
exceptions raised; and
output to current-output-port and current-error-port.
Some checking is also done to help protect test suites from crashing due to errors in the setup of the test itself, such as errors in evaluating an expression that provides an expected value for a test.
A future version of Overeasy might permit the properties that are tested to be extensible, such as for testing network state or other resources.
For the properties checked by tests, in most cases, the programmer can specify both an expected value and a predicate, or checker, for comparing expected and actual values. Note that, if the predicate is not an equality predicate of some kind, then the “expected” would be a misnomer, and “argument to the predicate” would be more accurate. The actual test syntax does not include the word “expected.” Specification of expected exceptions is diferent from values and output ports, in that only the predicate is specified, with no separate expected or argument value. All these have have reasonable defaults whenever possible.
1.1 Examples
Here’s a simple test, with the first argument the expression under test, and the other argument the expected value.
How the results of tests are reported varies depending on how the tests are run. For purposes of these examples, we will pretend we are running tests in the simplest way. In this way, tests that fail produce one-line error-messages to current-error-port, which in DrRacket show up as red italic text by default. Tests that pass in this way do not produce any message at all. So, our first example above, does not produce any message.
Now, for a test that fails:
Test FAIL : Value 6 did not match expected value 7 by equal?. |
That’s a quick way to do a test in a REPL or when you’re otherwise in a hurry, but if you’re reviewing a report of failed tests for one or more modules, you’d probably like a more descriptive way of seeing which tests failed. That’s what the test ID is for, and to specify it, we have to give keyword arguments in our test:
(test (+ 1 2 3) 7 #:id 'simple-addition)
Test FAIL simple-addition : Value 6 did not match expected value 7 by equal?. |
Quick note on syntax. The above is actually shorthand syntax. In the non-shorthand syntax, every argument to test has a keyword, so the above is actually shorthand for:
(test #:code (+ 1 2 3) #:val 7 #:id 'simple-addition)
#:code and #:val are used so often that the keywords can be left off, so long as the values are positionally the first and second arguments of test. One reason you might want to use the non-shorthand is if you like to have an ID for each test and you like to have #:id as the first thing, like a label or heading:
(test #:id 'simple-addition #:code (+ 1 2 3) #:val 7)
In the rest of these examples, we’ll use the shorthand syntax, because it’s quicker to type, and getting rid of the #:code and #:val keywords also makes less-common keyword arguments stand out.
So far, we’ve been checking the values of code, and we haven’t yet dealt in exceptions. Exceptions, such as due to programming errors in the code being tested, can also be reported:
(test (+ 1 (error "help!") 3) 3)
Test FAIL : Got exception #(struct:exn:fail "help!"), but expected value 3. |
And if an exception is the correct behavior, instead of specifying an expected value, we can use #:exn to specify predicate just like for with-handlers:
(test (+ 1 (error "help!") 3) #:exn exn:fail?)
That test passed. But if our code under test doesn’t throw an exception matched by our #:exn predicate, that’s a test failure:
(test (+ 1 2 3) #:exn exn:fail?)
Test FAIL : Got value 6, but expected exception matched by predicate exn:fail?. |
Of course, when you want finer discrimination of exceptions than, say, exn:fail? or exn:fail:filesystem?, you can write a custom predicate that uses exn-message or other information, and supply it to test’s #:exn.
Multiple values are supported:
(test (begin 1 2 3) (values 1 2 3))
Test FAIL : Value 3 did not match expected values (1 2 3) by equal?. |
You might have noticed that a lot of the test failure messages say “by equal?”. That’s referring to the default predicate, so, the following test passes:
(test (string-append "a" "b" "c") "abc")
But we let’s say we wanted the expected and actual values to not only be equal? but to be eq? as well:
(test (string-append "a" "b" "c") "abc" #:val-check eq?)
Test FAIL : Value "abc" did not match expected value "abc" by eq?. |
As mentioned earlier, the checker does not have to be an equality predicate, and it can use whatever reasoning you like in rendering is verdict on whether the actual value should be considered OK.
In addition to values an exceptions, test also intercepts and permits checking of current-output-port and current-error-port. By default, it assumes no output to either of those ports, which is especially good for catching programming errors like neglecting to specify an output port to a procedure for which the port is optional:
(test (let ((o (open-output-string))) (display 'a o) (display 'b) (display 'c o) (get-output-string o)) "abc")
Test FAIL : Value "ac" did not match expected value "abc" by equal?. Out bytes #"b" did not match expected #"" by equal?. |
Likewise, messages to current-error-port, such as warnings and errors from legacy code, are also caught by default:
(test (begin (fprintf (current-error-port) "%W%SYS$FROBINATOR_OVERHEAT\n") 0) 42)
Test FAIL : Value 0 did not match expected value 42 by equal?. Err bytes #"%W%SYS$FROBINATOR_OVERHEAT\n" did not match expected #"" by equal?. |
Now we know why we’ve started getting 0, which information might have gone unnoticed had our test engine not captured error port output: the frobinator is failing, after all these years of valiant service.
With the #:out-check and #:err-check keyword arguments to test, you can specify predicates other than equal?. Also, by setting one of these predicates to #f, you can cause the output to be consumed but not stored and checked. This is useful if, for example, the code produces large amounts of debugging message output.
(test (begin (display "blah") (display "blah") (display "blah") (* 44 2)) 88 #:out-check #f)
There are some more tricks you can do with test. Most notably, you’ll sometimes want to set up state in the system – Racket parameters, test input files, whatever. Because the test syntax can appear anywhere normal Racket code can, you can set up this state using normal Racket code. No special forms for setup and tear-down are required, nor are they provided.
1.2 Report Backends
The architecture of Overeasy is designed to permit different backends for reporting test results to be plugged in. Currently implemented backends are for:
quick one-line error messages for tests that fail; and
more verbose textual report of all test cases run.
In the future, Web front-end and GUI backends might also be implemented. The backend is dynamic context, so no changes to the files containing test code is required to run the tests with a different backend.
1.3 Test Contexts and Test Sections
The architectural notion that permits the backends to be plugged in is called the test context. Test context are nested dynamically, with each introduced context having the previous context as a parent. The same test context notion that permits backends for reporting to be introduced also permits test sections for grouping tests to be nested dynamically. The dynamic nesting of test sections facilitates reporting of test results when running unit tests for multiple modules together. Plugging in a backend for reporting simply means establishing it as the first or topmost test context.
By default, if a test is run without a test context, then the one-line error messages are used. If a test section context is introduced without a parent context, such as would usually be the case for an individual module’s unit tests, then the text report backend is plugged in by default.
One place you’ll want to use a section is for the unit tests for a particular module. This groups the tests together if the module’s unit tests are run in the context of a larger test suite, and it also provides a default report context when the unit tests are run by themselves. You might want to package the module’s unit tests in a procedure, for ease of use as part of a test suite. (Unless you have rigged up something different, like by having require or dynamic-require simply run the tests, without needing to then invoke a provided procedure. For illustration in this document, we’ll use procedures.) For example, if you have a fruits module, in file fruits.rkt, then you might want to put its unit tests in a procedure in file test-fruits.rkt, like so:
(define (test-fruits) (with-test-section #:id 'fruits (test #:id 'apple #:code (+ 1 2 3) #:val 6) (test #:id 'banana #:code (+ 4 5 6) #:val 6) (test #:id 'cherry #:code (+ 7 8 9) #:val 24)))
Notice that we put all the tests for module in fruits in with-test-section here, and gave it an ID. The ID didn’t have to be fruits like the module name; we could have called it fruity-unit-tests, fructose, or any other symbol.
Then let’s say we have a cars module, so in file some-cars-tests.rkt, we put this procedure:
(define (test-drive-cars) (with-test-section #:id 'cars (test (+ 77 11) 88 #:id 'delorean) (test (or (and #f 'i-cant-drive) 55) 55 #:id 'ferrari) (test (+ 300 8) 308 #:id 'magnum)))
Those unit test suites are used independently. Later, those modules are integrated into a larger system, COLOSSUS. For running all the unit tests for the modules of COLOSSUS, we add another module, which requires the other test modules, and invokes the each unit test procedure within its own test section:
(with-test-section #:id 'colossus-components (test-fruits) (test-drive-cars))
Unless this is done within another test context, the result will be to execute the tests in the default text report context. This produces a report like:
;; START-TESTS |
;; |
;; START-TEST-SECTION colossus-components |
;; |
;; START-TEST-SECTION fruits |
;; |
;; TEST apple |
;; (+ 1 2 3) |
;; OK |
;; |
;; TEST banana |
;; (+ 4 5 6) |
;; *FAIL* Value 15 did not match expected value 6 by equal?. |
;; |
;; TEST cherry |
;; (+ 7 8 9) |
;; OK |
;; |
;; END-TEST-SECTION fruits |
;; |
;; START-TEST-SECTION cars |
;; |
;; TEST delorean |
;; (+ 77 11) |
;; OK |
;; |
;; TEST ferrari |
;; (or (and #f (quote i-cant-drive)) 55) |
;; OK |
;; |
;; TEST magnum |
;; (+ 300 8) |
;; OK |
;; |
;; END-TEST-SECTION cars |
;; |
;; END-TEST-SECTION colossus-components |
;; |
;; END-TESTS |
;; OK: 5 FAIL: 1 BROKEN: 0 |
;; SOME TESTS *FAILED*! |
The test sections here are nested only two deep, but test sections may be nested to arbitrary depth. You can use test sections at each nested subsystem, to organize the unit tests for a module into groups, to group variations of generated test cases (e.g., if evaluating the same test form multiple times, with different values or state each time), or other purposes.
1.4 Project Status and History
Work is ongoing, but Overeasy should be useful already. It is being developed both as a useful tool, and as input to discussion in the Racket developer community about unifying the various test engines.
This package does not yet provide an interface so that additional reporting backends can be added. This is intentional, so that we can be comfortable that the interface won’t be changing soon before others start developing to it.
As a historical note, Overeasy is much superior to the author’s 2005 lightweight unit testing library, Testeez.
2 Interface
| (with-test-section #:id id body ...) |
See above.
| (test ...) |
See above.
3 History
Version 0.1 —
2011-08-26 – PLaneT (1 0) Initial release.
4 Legal
Copyright (c) 2011 Neil Van Dyke. This program is Free Software; Software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 3 of the License (LGPL 3), or (at your option) any later version. This program is distributed in the hope that it will be useful, but without any warranty; without even the implied warranty of merchantability or fitness for a particular purpose. See http://www.gnu.org/licenses/ for details. For other licenses and consulting, please contact the author.
Standard Documentation Format Note: The API signatures in this documentation are likely incorrect in some regards, such as indicating type any/c for things that are not, and not indicating when arguments are optional. This is due to a transitioning from the Texinfo documentation format to Scribble, which the author intends to finish someday.