SchemeUnit: Unit Testing in Scheme
_SchemeUnit: Unit Testing in Scheme_
====================================
By Noel Welsh (noelwelsh at yahoo dot com)
and Ryan Culpepper (ryan_sml at yahoo dot com)
This manual documents SchemeUnit version 2.0
Time-stamp: <05/02/13 22:52:57 ryanc>
Keywords: _schemeunit_, _test_, _testing_, _unit testing_,
_unit test_
Introduction
============
Unit testing is the testing in isolation of individual
elements of a program. SchemeUnit is a framework for
defining, organizing, and executing unit tests written in
the PLT dialect of Scheme (http://www.plt-scheme.org/).
SchemeUnit draws inspiration from two strands of work:
existing practice in interactive environments and the
development of unit testing frameworks following the growth
of Extreme Programming (http://www.extremeprogramming.org/).
In an interactive environment it is natural to write in a
``code a little, test a little'' cycle: evaluating
definitions and then immediately testing them in the
read-eval-print loop (REPL). We take the simplicity and
immediacy of this cycle as our model. By codifying these
practices we preserve the test cases beyond the running time
of the interpreter allowing the tests to be run again when
code changes.
Unit testing is one of the core practices of the Extreme
Programming software development methodology. Unit testing
is not new to Extreme Programming but Extreme Programming's
emphasis on unit testing has spurred the development of
software frameworks for unit tests. SchemeUnit draws
inspiration from, and significantly extends the
expressiveness of, these frameworks.
Quick Start
===========
[The example in this section assumes you are using the PLaneT
distribution of SchemeUnit. If you have downloaded and installed
SchemeUnit from the Schematics site, see the next section, titled
"PLaneT vs Schematics Bundles."]
Suppose you want to test the code contained in "file.scm".
Create a file called "file-test.scm". This file will
contain the test cases.
At the top of file-test.scm import file.scm and the
SchemeUnit library:
(require "file.scm"
(planet "test.ss" ("schematics" "schemeunit.plt" 1)))
Now were are going to create a test suite to hold all the
tests for file.scm.
(define file-tests
(make-test-suite
"Tests for file.scm"
...))
Now we define test cases within the file-tests test suite.
Let's assume that file.scm implements versions of + and -
called my-+ and my--. We are going to test my-+ and my--
for integer arthimetic.
(define file-tests
(make-test-suite
"Tests for file.scm"
(make-test-case
"Simple addition"
(assert = 2 (my-+ 1 1)))
(make-test-case
"Simple subtraction"
(assert = 0 (my-- 1 1)))))
Above we implemented two simple test cases. Test cases fail
if any one of the assertions they contain fail. There are
many predefined assertions and you can define your own using
the define-assertion macro, but that's a bit complicated for
a quick start guide!
Finally to run our tests we can either use the graphical or
textual user interfaces for SchemeUnit. We'll use the
textual interface as it is the simplest to use. The
graphical user interface gives much more information and may
be preferred in some cases. In file-test.scm we import the
textual user interface, and then run our tests using it.
(require (planet "text-ui.ss" ("schematics" "schemeunit.plt" 1)))
(test/text-ui file-tests)
Now we can execute this file in DrScheme and the textual
user interface will report the success or failure of tests!
PLaneT vs Schematics Bundles
----------------------------
SchemeUnit is distributed in two forms: as traditional .plt bundles
and via PLaneT. The traditional bundles are available from the
Schematics website (http://schematics.sourceforge.net), and they can
be installed using the "setup-plt" program. PLaneT is a network
service that automatically downloads and installs PLT Scheme
packages. For more information on PLaneT, search in the Help Desk or
visit http://planet.plt-scheme.org.
The syntax you use to require SchemeUnit modules depends on which
distribution of SchemeUnit you are using.
PLaneT:
(require (planet "test.ss" ("schematics" "schemeunit.plt" 1)))
Traditional:
(require (lib "test.ss" "schemeunit"))
In this manual, both forms are always given. However, to use the
example files, you may need to manually translate from one form to the
other.
SchemeUnit API
==============
Core SchemeUnit Types
---------------------
The _test_ is the core type in SchemeUnit. A test is either
a _test case_, which is a single action to test, or a _test
suite_, which is a collection of tests. Both test-cases and
test-suites may have optional setup and teardown actions.
> test := test-case | test-suite
> test-case := name action [setup] [teardown]
> test-suite := name (list-of test) [setup] [teardown]
An _assertion_ is a predicate that checks a condition. The
assertion fails if it's condition is false. Otherwise it
succeeds. In SchemeUnit assertions fail by raising an
exception of type _exn:test:assertion_. Otherwise
they return #t.
A test-case _action_ is a function of no arguments. A
test-case succeeds if the action evaluates without raising
an exception; otherwise it fails. Test-case failures are
divided into two cases: those thrown as the result of an
assertion failure, which we call _failures_ and those thrown
due to other reasons, which we call _errors_. We define a
type _test-result_ to encapsulate these concepts:
> test-result := test-failure | test-error | test-success
> test-failure := test-case failure-exn
> test-error := test-case error-exn
> test-success := test-case result
Constructing Test Cases and Test Suites
---------------------------------------
To use the following definitions:
Traditional:
(require (lib "test.ss" "schemeunit"))
PLaneT:
(require (planet "test.ss" ("schematics" "schemeunit.plt" 1))
Test cases are constructed using the _make-test-case_ macro.
> (make-test-case name expr ... [setup expr] [teardown expr])
The name is a string that is reported as the name of the
test case.
The exprs are one or more expressions that are run as the
test case action.
The setup expr is an expression that is executed before the
action. The setup expr is signalled by placing the symbol
setup (unquoted because make-test-case is a macro) before
the expression.
The teardown expr is an expression that is executed after
the action. The teardown expr is signalled by placing the
symbol teardown (unquoted because make-test-case is a macro)
before the expression.
For example, the test below checks that the file "test.dat"
contains the string "foo". The setup action writes to this
file. The teardown action deletes it.
(make-test-case
"The name"
(with-input-from-file "test.dat"
(lambda ()
(assert-equal? "foo" (read))))
setup
(with-output-to-file "test.dat"
(lambda ()
(write "foo")))
teardown
(delete-file "test.dat"))
Test suites are constructed using the _make-test-suite_
function.
> (make-test-suite name test ... [setup thunk] [teardown thunk])
The name is a string that is reported as the name of the
test suite.
The tests are one or more test cases or test suites.
The setup thunk is an expression that is executed before the
tests are run. The setup thunk is signalled by placing the
symbol setup before the thunk.
The teardown thunk is an expression that is executed after
the tests are run. The teardown thunk is signalled by
placing the symbol teardown before the thunk.
For example:
(make-test-case
"The name"
(make-test-case
"Foo"
(assert = 1 1))
'setup
(lambda () (display "setup"))
'teardown
(lambda () (display "teardown")))
Predefined Assertions
---------------------
To use the following definitions:
Traditional:
(require (lib "test.ss" "schemeunit"))
PLaneT:
(require (planet "test.ss" ("schematics" "schemeunit.plt" 1)))
SchemeUnit provides a rich library of predefined assertions.
Every assertion takes an optional _message_, which is a
string that is displayed if the assertion fails. Each
assertion comes in two variants: a macro with the given
name, and a function with a * appended to the name. The
macro version grabs source location information, which aids
debugging, and so should be used wherever possible.
The predefined assertions are:
> (assert binary-predicate actual expected [message])
> (assert-equal? actual expected [message])
> (assert-eqv? actual expected [message])
> (assert-eq? actual expected [message])
> (assert-true actual [message])
> (assert-false actual [message])
> (assert-not-false actual [message])
> (assert-pred unary-predicate actual [message])
> (assert-exn exn-predicate thunk [message])
> (assert-not-exn thunk [message])
> (fail [message])
Providing Additional Information With Assertions
------------------------------------------------
To use the following definitions:
Traditional:
(require (lib "test.ss" "schemeunit"))
PLaneT:
(require (planet "test.ss" ("schematics" "schemeunit.plt" 1)))
When an assertion fails is stores information including the
name of the assertion, the location and message (if
available), the expression the assertion is called with, and
the parameters to the assertion. Additional information can
be stored by using the _with-assertion-info*_ function, and
the _with-assertion-info_ macro.
> (with-assertion-info* info thunk)
The with-assertion-info* function stores the given info in
the assertion information stack for the duration of the
execution of thunk. Info is a list of assertion-info
structures (see below).
> (with-assertion-info ((name val) ...) body ...)
The with-assertion-info macro stores the given information
in the assertion information stack for the duration of the
execution of the body expressions. Name is a symbol and val
is any value.
> (make-assertion-info name value)
An assertion-info structure stores information associated
with the context of execution of an assertion. Name is a
symbol. Value is any value.
The are several predefined functions that create assertion
information structures with predefined names. This avoids
misspelling errors.
> (make-assertion-name name)
> (make-assertion-params params)
> (make-assertion-location loc)
> (make-assertion-expression msg)
> (make-assertion-message msg)
User Defined Assertions
-----------------------
To use the following definitions:
Traditional:
(require (lib "test.ss" "schemeunit"))
PLaneT:
(require (planet "test.ss" ("schematics" "schemeunit.plt" 1)))
SchemeUnit provides a way for user's to extend its builtin
collection of assertions using the _define-simple-assertion_
and _define-assertion_ macros. To understand these macros
it is useful to understand a few details about an assertions
evaluation model.
Firstly, an assertion should be considered a function, even
though most uses are actually macros. In particular,
assertions always evaluate their arguments exactly once
before executing any expressions in the body of the
assertions. Hence if you wish to write assertions that
evalute user defined code that code must be wrapped in a
thunk (a function of no arguments) by the user. The
predefined assert-exn is an example of this type of
assertion.
It is also useful to understand how the assertion
information stack operates. The stack is stored in a
parameter and the with-assertion-info forms evaluate to
calls to parameterize. Hence assertion information has
lexical scope. For this reason simple assertions (see
below) cannot report additional information. All assertions
created using define-simple-assertion or define-assertion
grab some information by default: the name of the assertions
and the values of the parameters. Additionally the macro
forms of assertions grab location information and the
expressions passed as parameters.
> (define-simple-assertion (name param ...) expr ...)
The define-simple-assertion macro constructs an assertion
called name that takes the params as arguments and an
optional message and evaluates the exprs. The assertion
fails if the result of the exprs is #f. Otherwise the
assertion succeeds. Note that simple assertions cannot
report extra information using with-assertion-info.
The define-simple-assertion macro actually constructs two
assertions: a macro with the given name that collects source
location information, and a function with name name* that
does not collection source location information but can be
used in a higher-order fashion. In my experience
higher-order assertions are useful for testing SchemeUnit
but not much else.
> (define-assertion (name param ...) expr ...)
The define-assertion acts in exactly the same way as the
define-simple-assertion macro, except the assertion only
fails if the macro _fail-assertion_ is called. This allows
more flexible assertions, and in particular more flexible
reporting options.
> (fail-assertion)
The fail-assertion macro raises an exn:test:assertion with
the contents of the assertion information stack.
You are encouraged to submit libraries of assertions to
Schematics.
User Interfaces
---------------
To use the following definitions:
Traditional:
(require (lib "text-ui.ss" "schemeunit"))
PLaneT:
(require (planet "text-ui.ss" ("schematics" "schemeunit.plt" 1)))
OR
Traditional:
(require (lib "graphical-ui.ss" "schemeunit"))
PLaneT:
(require (planet "graphical-ui.ss" ("schematics" "schemeunit.plt" 1)))
SchemeUnit provides two user interfaces: a text UI and a
graphical UI.
The text UI is run via the function
> (test/text-ui test)
The given test is run and the result of running it output to
the current-output-port. The output is compatable with the
(X)Emacs next-error command (as used, for example, by
(X)Emac's compile function).
The graphical UI is run via the function
> (test/graphical-ui test)
When run under DrScheme, the graphical UI provides tracebacks and
hyperlinked errors just like those provided by DrScheme's "bug" icon.
Command-line tool
-----------------
There is also a command-line interface for SchemeUnit that comes with
the traditional bundle (not available for PLaneT).
Once SchemeUnit has been installed and set up, there will be an
executable called $PLT/bin/schemeunit (or $PLT/schemeunit.exe on
Windows). This program requires a module specification with a provided
name bound to the test to be executed.
$plt/bin/schemeunit my-tests '(lib "my-tests.ss" "my-collect")'
(Notice the single quotes around the entire module specification.)
The enhanced gui runs the test found at the location as normal, but if
you edit your code and click the "refresh" button in the SchemeUnit
frame, the tests will be reloaded with your changes and rerun.
Glassbox Testing
----------------
To use the following definitions:
Traditional:
(require (lib "util.ss" "schemeunit"))
PLaneT:
(require (planet "util.ss" ("schematics" "schemeunit.plt" 1)))
Sometimes it is useful to test definitions that are not
exported by a module. SchemeUnit supports this via the
_require/expose_ macro.
> (require/expose module (id ...))
The require/expose macro is like a normal require form,
except it can require definitions that are defined but not
provided (i.e. exported) by a module. The ids are the
identifiers that are to be exposed in the current module or
namespace. Module is a module specifier (as given to
require) where the ids are found.
Other Utilities
---------------
To use the following definitions:
Traditional:
(require (lib "util.ss" "schemeunit"))
PLaneT:
(require (planet "util.ss" ("schematics" "schemeunit.plt" 1)))
The _make-test-suite*_ macro provided a shortcut for the
common case of a test suite that simply defines a number of
test cases.
> (make-test-suite* name (test-case-name test-case-expr ...) ...)
Makes a test suite with the given name that contains test
cases with the given names and actions.
> (assert-regexp-match regex-or-string string-or-port [message])
An assertion that tests if a regular expression (string or
regexp) matches a string or port.
Tips For Using SchemeUnit
=========================
I create one module of tests for each module of code. If
the module is called "foo" the test module is called
"foo-test" and exports a single test suite, called
"foo-tests". For each project (or collection) called, say,
"bar" I have a single module "all-bar-tests" that exports a
single test suite (called "all-bar-tests") which collects
together all the test suites for that project. I often
create another file, called "run-tests.ss" which runs
"all-bar-tests' using the text user interface. To run all
the project's tests I can then simply execute "run-test.ss"
in Dr/MzScheme or (X)Emacs.
To run tests from (X)Emacs the following command is can be
given as the "Compile command" to compile:
mzscheme -qe '(begin (load "run-test.ss") (exit))'
This command causes mzscheme to execute the code in
"run-test.ss" and then exit. If some preventing loading of
"run-test.ss", for example a syntax error, mzscheme will
pause waiting for input and you will have to kill it next
time you run compile ((X)Emacs asks if you want to do this).
Defining your own assertions is one of the most powerful
features of SchemeUnit. Whenever you find yourself writing
out the same series of assertions define your own assertion
and use that instead. Your assertions will act just like
the pre-defined assertions; they take optional message
strings and will display locations and parameters. They are
really easy to create as well. For instance, to define an
assertion called "assert-is-2" that asserts a value is equal
to 2, you'd just evaluate
(define-simple-assertion (assert-is-2 x)
(= x 2))
Assertions compose as well, so you could write:
(define-assertion (assert-is-2 x)
(assert = x 2))
If you find you're creating a library of assertions please
submit them to us so we can share them with all SchemeUnit
users.
Extended Example
================
This example test suite is included in the SchemeUnit
distribution as "example.ss":
;; The tests below are intended as examples of how to use the test
;; package. They test PLT Scheme's arithmetic operations and some
;; simple file reading
(require (lib "test.ss" "schemeunit"))
(require (lib "text-ui.ss" "schemeunit"))
(test/text-ui
(make-test-suite
"Example tests"
(make-test-suite
"Arithmetic tests"
(make-test-case "Multiply by zero"
(assert = (* 1234 0) 0))
(make-test-case "Add zero"
(assert = (+ 1234 0) 1234))
(make-test-case "Multiply by one"
(assert = (* 123.0 1) 123))
(make-test-case "Add one"
(assert = (+ 123.0 1) 124))
(make-test-case "Expt 0 0"
(assert = 1 (expt 0 0)))
(make-test-case "Expt 0 1"
(assert = 0 (expt 0 1)))
(make-test-case "Expt 0.0 0.0"
(assert = 1.0 (expt 0.0 0.0)))
(make-test-case "Expt 0.0 1.0"
(assert = 0.0 (expt 0.0 1.0)))
)
(make-test-suite
"File tests"
;; An example with a teardown action
(let ((port (open-input-string "this is a test string")))
(make-test-case
"String port read"
(assert-equal? "this is a test string" (read-line port))
teardown
(close-input-port port)))
;; An example with a setup and a teardown action
(make-test-case
"File port read"
(with-input-from-file "test.dat"
(lambda ()
(assert-equal? "foo" (read))))
setup
(with-output-to-file "test.dat"
(lambda ()
(write "foo")))
teardown
(delete-file "test.dat"))
)
))
Further Reading
===============
The paper "Two Little Languages: SchemeUnit and SchemeQL"
(http://schematics.sourceforge.net/schemeunit-schemeql.ps)
describes the design rationale and implementation of the 1.x
series of SchemeUnit.
There are copious comments in the source code. The file
"example.ss", provided as part of SchemeUnit, is a good
example of a reasonably complex test suite.
Good references for the Extreme Programming methodology
include
- http://c2.com/cgi-bin/wiki?ExtremeProgrammingRoadmap
- http://www.extremeprogramming.org
- http://www.xprogramming.com
- http://www.junit.org; JUnit}, the Java unit test
framework.
Most of the code at Schematics
(http://schematics.sourceforge.net/) has extensive test
suites written using SchemeUnit.
Acknowledgements
================
The following people have contributed to SchemeUnit:
- Richard Cobbe provided require/expose
- John Clements suggested several new assertions
- Jose A. Ortega Ruiz alerted me a problem in the
packaging system and helped fix it.
- Sebastian H. Seidel provided help packaging SchemeUnit
into a .plt
- Don Blaheta provided the method for grabbing line number
and file name in assertions
- Patrick Logan ported example.ss to version 1.3
- The PLT team made PLT Scheme
- The Extreme Programming community started the whole
testing framework thing
![[logo]](/logo.png){kind=logo}