Overeasy: Racket Language Test Engine

Version: 3:1

Overeasy: Racket Language Test Engine

Neil Van Dyke

License: LGPLv3 Web: http://www.neilvandyke.org/overeasy/

(require (planet neil/overeasy:3:1))

1 Introduction

Overeasy is a software test engine for the Racket programming language. It designed for all of:

interspersing unit tests in-line with implementation code, using the Racket test submodule, or in separate files;
unit testing of individual modules, either from within DrRacket, or from the command line or Emacs;
rapid interactive testing of expressions in the REPL; and
running hierarchical sets of individual module unit tests at once.

An individual test case, or test, is specified by the programmer with the test syntax, and evaluation of that syntax causes the test to be run. Properties that are checked by tests are:

values of expressions (single value, or multiple value);
exceptions raised; and
output to current-output-port and current-error-port.

Tests can also be marked as expected to fail. Tests can be organized into hierarchically-nested sections.

Some checking is also done to help protect test suites from crashing due to errors in the setup of the test itself, such as errors in evaluating an expression that provides an expected value for a test.

1.1 Simple Examples

Here’s a simple test, with the first argument the expression under test, and the other argument the expected value.

> (test (+ 1 2 3) 6)

How the results of tests are reported varies depending on how the tests are run. For purposes of these examples, we will pretend we are running tests in the simplest way. In this way, tests that fail produce one-line error-messages to current-error-port, which in DrRacket show up as red italic text by default. Tests that pass in this way do not produce any message at all. So, our first example above, does not produce any message.

Now, for a test that fails:

> (test (+ 1 2 3) 7)

TEST FAILED [???] Value 6 did not match expected value 7 by equal?.

That’s a quick way to do a test in a REPL or when you’re otherwise in a hurry, but if you’re reviewing a report of failed tests for one or more modules, you’d probably like a more descriptive way of seeing which tests failed. That’s what the test ID is for, and to specify it, we use the #:id keyword arguments in our test:

> (test #:id 'simple-addition
(+ 1 2 3)
7)

TEST FAILED [simple-addition] Value 6 did not match expected value 7 by equal?.

Quick note on syntax. The above is actually shorthand syntax. In the non-shorthand syntax, every argument to test has a keyword, so the above is actually shorthand for:

(test #:id   'simple-addition
      #:code (+ 1 2 3)
      #:val  7)

#:code and #:val are used so often that the keywords can be left off, so long as there are no other keyword arguments before them, other than #:id.

You can even leave off the #:id keyword, so long as you have both code and val expressions, also without keywords. So, the above example has equivalent shorthand:

(test 'simple-addition
(+ 1 2 3)
7)

In the rest of these examples, we’ll use the shorthand syntax, because it’s quicker to type, and getting rid of the #:code and #:val keywords also makes less-common keyword arguments stand out.

1.2 Exceptions

So far, we’ve been checking the values of code, and we haven’t yet dealt in exceptions. Exceptions, such as due to programming errors in the code being tested, can also be reported:

> (test (+ 1 (error "help!") 3)
3)

TEST FAILED [???] Got exception #(struct:exn:fail "help!"), but expected value 3.

And if an exception is the correct behavior, instead of specifying an expected value, we can use #:exn to specify predicate just like for with-handlers:

> (test (+ 1 (error "help!") 3)
#:exn exn:fail?)

That test passed. But if our code under test doesn’t throw an exception matched by our #:exn predicate, that’s a test failure:

> (test (+ 1 2 3)
#:exn exn:fail?)

TEST FAILED [???] Got value 6, but expected exception matched by predicate exn:fail?.

Of course, when you want finer discrimination of exceptions than, say,exn:fail? or exn:fail:filesystem?, you can write a custom predicate that uses exn-message or other information, and supply it to test’s #:exn.

1.3 Multiple Values

Multiple values are supported:

> (test (begin 1 2 3)
(values 1 2 3))

TEST FAILED [???] Value 3 did not match expected values (1 2 3) by equal?.

1.4 Custom Value Checks

You might have noticed that a lot of the test failure messages say “by equal?”. That’s referring to the default predicate, so, the following test passes:

> (test (string-append "a" "b" "c")
"abc")

But we let’s say we wanted the expected and actual values to not necessarily be equal? but to be numbers within 3 decimal places of being equal:

> (define (close-enough-val-check a-values b-values)
    (and (null? (cdr a-values))
         (null? (cdr b-values))
         (let ((a (car a-values))
               (b (car b-values)))
           (and (number? a)
                (number? b)
                (equal? (round (* 1000 a))
                        (round (* 1000 b)))))))

> (test 3.142
3.14159
#:val-check close-enough-val-check)

Note that, since expressions can produce multiple values, the #:val-check predicate receives lists of values instead of single values.

As mentioned earlier, the checker does not have to be an equality predicate, and it can use whatever reasoning you like in rendering its verdict on whether the actual value should be considered OK.

1.5 Output Ports

In addition to values and exceptions, test also intercepts and permits checking of current-output-port and current-error-port. By default, it assumes no output to either of those ports, which is especially good for catching programming errors like neglecting to specify an output port to a procedure for which the port is optional:

> (test (let ((o (open-output-string)))
          (display 'a o) (display 'b) (display 'c o)
          (get-output-string o))
        "abc")

TEST FAILED [???] Value "ac" did not match expected value "abc" by equal?. Out bytes #"b" did not match expected #"" by equal?.

Likewise, messages to current-error-port, such as warnings and errors from legacy code, are also caught by default:

> (test (begin (fprintf (current-error-port)
                        "%W%SYS$FROBINATOR_OVERHEAT\n")
               0)
        42)

TEST FAILED [???] Value 0 did not match expected value 42 by equal?. Err bytes #"%W%SYS$FROBINATOR_OVERHEAT\n" did not match expected #"" by equal?.

Now we know why we’ve started getting 0, which information might have gone unnoticed had our test engine not captured error port output: the frobinator is failing, after all these years of valiant service.

With the #:out-check and #:err-check keyword arguments to test, you can specify predicates other than equal?. Also, by setting one of these predicates to #f, you can cause the output to be consumed but not stored and checked. This is useful if, for example, the code produces large amounts of debugging message output.

> (test (begin (display "blah")
               (display "blah")
               (display "blah")
               (* 44 2))
        88
        #:out-check #f)

1.6 Test Sections

Sequences of tests can be nested in a test section, and the test section given an ID. Test sections can be nested within each other.

For example:

> (test-section 'fundamentals

    (test-section 'writing

      (test 'abcs
            (string-append "a" "b" "c")
            "abc"))

    (test-section 'arithmetic

      (test 'one-two-three
            (+ 1 2 3)
            6)

      (test 'for-fife-sax
            (+ 4 5 6)
            666)))

TEST FAILED [fundamentals arithmetic for-fife-sax] Value 15 did not match expected value 666 by equal?.

Note that the reference to test ID for-fife-sax in the error message is qualified with the path through the test sections: section fundamentals and its child section, arithmetic. In large test suites, this can help to locate the test.

Note that a given instance test-section syntax appear inside procedures and loops. This can be very useful for testing code with different arguments or context, when the behavior is the same or similar for many of the combinations. When doing this, note that a test-section ID need not be a constant symbol like 'for-fife-sax, but can also be a Racket expression, so it could be used to indicate one or more of the arguments. For example, suppose that there is a case for three-argument procedure foo in which, if the third argument is 0, the answer should be 42:

> (test-section 'foo-constant-with-z-arg-zero

    (for ((bar? (in-list '(#true #false))))

      (test-section bar?

        (for ((power (in-range 1 9)))

          (test #:id   power
                #:code (foo bar? power 0)
                #:val  42)))))

When this test code is run, Racket logger entries starting with the following should be made (and can be viewed in the DrRacket Log window, and elsewhere):

overeasy: Start Test Section [foo-constant-with-z-arg-zero]
overeasy: Start Test Section [foo-constant-with-z-arg-zero #t]
overeasy: Test Passed [foo-constant-with-z-arg-zero #t 1]
overeasy: Test Passed [foo-constant-with-z-arg-zero #t 2]
overeasy: Test Passed [foo-constant-with-z-arg-zero #t 3]

By the way, in the examples above, we have left out the optional #:id keyword from the test-section syntax. The following two forms are equivalent:

(test-section #:id 'math (test (+ 1 2) 3) ...)

(test-section 'math (test (+ 1 2) 3) ...)

1.7 Expected Failures

Sometimes, you’ll have a test case that is known to fail, but that you are deferring fixing, and that you don’t want distracting you from other test cases at this time. Rather than commenting-out the test case code, which might result in being lost or forgotten, you can instead mark the test case with #:fail. For example:

> (test 'basic-arithmetic
        (plussy 1 2 3)
        6
        #:fail "bug til move to new library")

In this example, the string "bug til move to new library" gives the rationale for expecting the test to fail but deferring corrective action on it. When this test syntax is evaluated, instead of an exception being raised, instead a warning level message is sent to the Racket logger:

overeasy: Test Failed Expectedly [basic-arithmetic] Value 5.9 did not match expected value 6 by equal?. (#:fail "bug til move to new library")

Note that if (plussy 1 2 3) does produce the correct 6 value, but the #:fail argument is still present, then the test will actually be considered to fail:

TEST FAILED [basic-arithmetic] Passed unexpectedly. (#:fail "bug til move to new library")

1.8 Intermixed Racket Code

There are some more tricks you can do with test. Most notably, you’ll sometimes want to set up state in the system – Racket parameters, test input files, whatever. Because the test syntax can appear anywhere normal Racket code can, you can set up this state using normal Racket code. No special forms for setup and tear-down are required, nor are they provided.

2 Interface

syntax
(test-section maybe-id-kw id body ...+)

maybe-id-kw =
| #:id

See above.

syntax
(test !!!)

See above.

3 Deprecated

syntax
(with-test-usection maybe-id-kw id body ...+)

maybe-id-kw =
| #:id

Deprecated. Alias for test-section.

4 Known Issues

Document the support for #rx with #:exn.
Document the support for #:notes, but rename it to #:note and support the old name (which some package might have already used, even though undocumented) as an alias.
Document how to use with submodules and DrRacket. Just examples of test submodule, and tests run when hitting the Run button with particular DrRacket option set.
In documentation, fill out the defproc for test form.
For #:out and #:err, force expected value to bytes (convert using UTF-8 if string), and also ensure that port encoding is UTF-8 (or whatever encoding makes sense).
Add conveniences for defining #:val-check predicates. For example, a convenient way to specify eq? of a single value (not multiple values).
Try to mess with the continuation marks for test failues due to unexpected exceptions, so that user can navigate first to the test form, and then to the origin of the exception.
Provide convenience for running tests from multiple files, in test submodules, and possibly entire files/modules.
This package does not yet expose an interface so that alternative means of reporting (e.g., GUI) can be added easily. This is intentional, until we can be comfortable that the existing internal interface won’t be changing soon.

5 History

PLaneT 3:1 — 2012-12-26
- Documentation tweaks.
PLaneT 3:0 — 2012-12-25
- Now requires Racket 5.3.1.
- Converted to use the Racket logger facility, and format of the messages changed. This is reason for the change to the major version number of the PLaneT package.
- #:fail feature added.
- Messages regarding tests now qualify the test ID with the IDs of the parent test sections.
- Documentation changes.
- Internal simplifications to test contexts. There are no longer parent and child test contexts, but a single test context. Test contexts no longer have handlers for the start and end of tests. When there is not yet any test context, previously a test-section would create one kind of transient context, and test would create a different kind of transient context; now, both forms create the same kind of context, and it is not transient.
PLaneT 2:1 — 2012-11-15
- Fixed bug regarding “fprintf: format string requires 1 arguments, given 2; arguments were: #<output-port> "TEST BROKEN!~A\ " ...
PLaneT 2:0 — 2012-06-11
- Converted to McFly.
- The default test context now raises an exception with syntax location info for failed test cases, rather than only writing a message to current-error-port.
- The test syntax now preserves syntax location info better.
- Added shorthand syntax (test ID CODE VAL RESTn ...).
- The new name test-section is now preferred to the old name with-test-section.
- In test-section syntax, the #:id keyword itself is now optional.
- test-section may have no body forms.
Version 0.1 — PLaneT 1:0 — 2011-08-26
Initial release.

6 Legal

Copyright 2011–2012 Neil Van Dyke. This program is Free Software; Software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but without any warranty; without even the implied warranty of merchantability or fitness for a particular purpose. See http://www.gnu.org/licenses/ for details. For other licenses and consulting, please contact the author.

1	Introduction
2	Interface
3	Deprecated
4	Known Issues
5	History
6	Legal