Whalesong: a Racket to JavaScript compiler
Danny Yoo <[email protected]>
Source code can be found at: https://github.com/dyoo/whalesong. The latest version of this document lives in http://hashcollision.org/whalesong.
Current commit head is e03398802c194598f864da37dac4c2e002ab8e5e.
Warning: this is work in progress!
1 Introduction
Whalesong is a compiler from Racket to JavaScript; it takes Racket programs and translates them so that they can run stand-alone on a user’s web browser. It should allow Racket programs to run with (hopefully!) little modification, and provide access through the foreign-function interface to native JavaScript APIs. The included runtime library supports the numeric tower, an image library, and a framework to program the web in functional event-driven style.
The GitHub source repository to Whalesong can be found at https://github.com/dyoo/whalesong.
Prerequisites: at least Racket 5.1.1, and a Java 1.6 SDK.
2 Getting started
2.1 Installing Whalesong
At the time of this writing, although Whalesong has been deployed to PLaneT, the version on PLaneT is out of date. I’ll be updating the PLaneT package as soon as Whalesong starts to stabilize, but the system as a whole is still in some flux.
You may want to get the latest sources instead of using the version on PLaneT. Doing so requires doing a little bit of manual work. The steps are:
Check Whalesong out of Github.
Set up the PLaneT development link to your local Whalesong instance.
Run raco setup over Whalesong to finish the installation
$ git clone git://github.com/dyoo/whalesong.git |
$ planet link dyoo whalesong.plt 1 0 whalesong |
$ raco setup -P dyoo whalesong.plt 1 0 |
$ ./whalesong |
Expected one of the following: [build, get-runtime, get-javascript]. |
Note: whenever Whalesong’s source code is updated from Github, please re-run the raco setup. Otherwise, Racket will try to recompile Whalesong on every single use of Whalesong, which can be very expensive.
2.2 Making Standalone .xhtml files with Whalesong
Let’s try making a simple, standalone executable. At the moment, the program must be written in the base language of (planet dyoo/whalesong). This restriction unfortunately prevents arbitrary racket/base programs from compiling at the moment; the developers (namely, dyoo) will be working to remove this restriction as quickly as possible.
$ racket hello.rkt |
hello world |
$ |
$ whalesong build hello.rkt |
|
$ ls -l hello.xhtml |
-rw-rw-r-- 1 dyoo nogroup 692213 Jun 7 18:00 hello.xhtml |
The generated program can be downloaded here: dom-play.xhtml
"dom-play.rkt"
#lang planet dyoo/whalesong ;; Uses the JavaScript FFI, which provides bindings for: ;; $ and call (require (planet dyoo/whalesong/js)) ;; insert-break: -> void (define (insert-break) (call-method ($ "<br/>") "appendTo" body) (void)) ;; write-message: any -> void (define (write-message msg) (void (call-method (call-method (call-method ($ "<span/>") "text" msg) "css" "white-space" "pre") "appendTo" body))) ;; Set the background green, and show some content ;; on the browser. (void (call-method body "css" "background-color" "lightgreen")) (void (call-method ($ "<h1>Hello World</h1>") "appendTo" body)) (write-message "Hello, this is a test!") (insert-break) (let loop ([i 0]) (cond [(= i 10) (void)] [else (write-message "iteration ") (write-message i) (insert-break) (loop (add1 i))]))
2.3 Using Whalesong functions from JavaScript
Whalesong also allows functions defined from Racket to be used from JavaScript. As an example, we can take the boring factorial function and define it in a module called "fact.rkt":
The files can also be downloaded here:with generated JavaScript binaries here:
"fact.rkt"
#lang planet dyoo/whalesong (provide fact) (define (fact x) (cond [(= x 0) 1] [else (* x (fact (sub1 x)))]))
$ whalesong get-javascript fact.rkt > fact.js |
$ ls -l fact.js |
-rw-r--r-- 1 dyoo dyoo 27421 2011-07-11 22:02 fact.js |
$ whalesong get-runtime > runtime.js |
$ ls -l runtime.js |
-rw-r--r-- 1 dyoo dyoo 544322 2011-07-11 22:12 runtime.js |
"index.html"
<!DOCTYPE html>
<html>
<head>
<script src="runtime.js"></script>
<script src="fact.js"></script>
<script>
// Each module compiled with 'whalesong get-runtime' is treated as a
// main module. invokeMains() will invoke them.
plt.runtime.invokeMains();
plt.runtime.ready(function() {
// Grab the definition of 'fact'...
var myFactClosure = plt.runtime.lookupInMains('fact');
// Make it available as a JavaScript function...
var myFact = plt.baselib.functions.asJavaScriptFunction(
myFactClosure);
// And call it!
myFact(function(v) {
$('#answer').text(v.toString());
},
function(err) {
$('#answer').text(err.message).css("color", "red");
},
10000
// "one-billion-dollars"
);
});
</script>
</head>
<body>
The factorial of 10000 is <span id="answer">being computed</span>.
</body>
</html>
Replacing the 10000 with "one-billion-dollars" should reliably produce a proper error message.
3 Using whalesong
Whalesong provides a command-line utility called whalesong for translating Racket to JavaScript. It can be run in several modes:
To create standalone XHTML documents
To output the compiled JavaScript as a single ".js" file
To output the compiled JavaScript as several ".js" files, one per module. (this isn’t done yet...)
$ whalesong build [name-of-racket-file] |
Almost all of the whalesong commands support two command line options:
–compress-javascript: Use Google Closure’s JavaScript compiler to significantly compress the JavaScript. Using this currently requires a Java 1.6 JDK.
–verbose: write verbose debugging information to standard error.
For more advanced users, whalesong can be used to generate JavaScript in non-standalone mode. This gives the web developer more fine-grained control over how to control and deploy the outputted program.
3.1 build
Given the name of a program, this builds a standalone ".xhtml" file into the current working directory that executes the program in a web browser.
The ".xhtml" should be self-contained, with an exception: if the file uses any external resources by using define-resource, those resources are written into a subdirectory called "res" under the current working directory.
3.2 get-javascript
Given the name of a program, writes the JavaScript to standard output, as well as its dependent modules. The outputted file is meant to be used as a SCRIPT source.
By default, the given program will be treated as a main module. All main modules will be executed when the JavaScript function plt.runtime.invokeMains() is called.
3.3 write-javascript-files
[NOT DONE YET] (needs to write a MANIFEST file?) (this almost seems like we need some concept of a JAR... )
3.4 write-resources
[NOT DONE YET]
3.5 get-runtime
Prints out the core runtime library that the files generated by get-javascript depend on.
4 Including external resources with (planet dyoo/whalesong:1:2/resource)
(require (planet dyoo/whalesong:1:2/resource)) |
Programs may need to use external file resources that aren’t themselves Racket programs, but instead some other kind of data. Graphical programs will often use ".png"s, and web-related programs ".html"s, for example. Whalesong provides the (planet dyoo/whalesong:1:2/resource) library to refer and use these external resources. When Whalesong compiles a program into a package, these resources will be bundled alongside the JavaScript-compiled output.
(define-resource id path-string) |
#lang planet dyoo/whalesong (require (planet dyoo/whalesong/resource)) (define-resource my-whale-image-resource "humpback.png")
(resource->url a-resource) → string? |
a-resource : resource? |
#lang planet dyoo/whalesong (require (planet dyoo/whalesong/resource) (planet dyoo/whalesong/image)) (define-resource my-whale-image-resource "humpback.png") (define WHALE-IMAGE (bitmap/url (resource->url my-whale-image-resource)))
5 The JavaScript API
body : any/c |
(call-method object method-name arg ...) → any/c |
object : any/c |
method-name : string? |
arg : any/c |
($ locator) → any/c |
locator : any/c |
(call-method ($ "<h1>Hello World</h1>") "appendTo" body)
(in-javascript-context?) → boolean |
6 World programming
Whalesong provides a library to support writing functional I/O programs (A Functional I/O System). Here’s an example of such a world program:
[FIXME: embed a world program here.]
7 Internals
Please skip this section if you’re a regular user: this is really notes internal to Whalesong development, and is not relevant to most people.
These are notes that describe the internal details of the implementation, including the type map from Racket values to JavaScript values. It should also describe how to write FFI bindings, eventually.
7.1 Architecture
The basic idea is to reuse most of the Racket compiler infrastructure. We use the underlying Racket compiler to produce bytecode from Racket source; it also performs macro expansion and module-level optimizations for us. We parse that bytecode using the compiler/zo-parse collection to get an AST, compile that to an intermediate language, and finally assemble JavaScript.
AST IL |
parse-bytecode.rkt -----> compiler.rkt ----> assembler.rkt |
|
The IL is intended to be translated straightforwardly. We currently have an assembler to JavaScript "js-assembler/assemble.rkt", as well as a simulator in "simulator/simulator.rkt". The simulator allows us to test the compiler in a controlled environment.
7.2 parser/parse-bytecode.rkt
(We try to insulate against changes in the bytecode structure by using the version-case library to choose a bytecode parser based on the Racket version number. Add more content here as necessary...)
7.3 compiler/compiler.rkt
This translates the AST to the intermediate language. The compiler has its origins in the register compiler in Structure and Interpretation of Computer Programs with some significant modifications.
Since this is a stack machine, we don’t need any of the register-saving infrastructure in the original compiler. We also need to support slightly different linkage structures, since we want to support multiple value contexts. We’re trying to generate code that works effectively on a machine like the one described in http://plt.eecs.northwestern.edu/racket-machine/.
The intermediate language is defined in "il-structs.rkt", and a simulator for the IL in "simulator/simulator.rkt". See "tests/test-simulator.rkt" to see the simulator in action, and "tests/test-compiler.rkt" to see how the output of the compiler can be fed into the simulator.
val: value
proc: procedure
argcount: number of arguments
env: environment stack
control: control stack
7.4 js-assembler/assemble.rkt
The intent is to potentially support different back end generators for the IL. "js-assembler/assemble.rkt" provides a backend for JavaScript.
The JavaScript assembler plays a few tricks to make things like tail calls work:
Each basic block is translated to a function taking a MACHINE argument.
Every GOTO becomes a function call.
The head of each basic-blocked function checks to see if we should trampoline (http://en.wikipedia.org/wiki/Trampoline_(computers))
We support a limited form of computed jump by assigning an attribute to the function corresponding to a return point. See the code related to the LinkedLabel structure for details.
Otherwise, the assembler is fairly straightforward. It depends on library functions defined in "runtime-src/runtime.js". As soon as the compiler stabilizes, we will be pulling in the runtime library in Moby Scheme into this project. We are right in the middle of doing this, so expect a lot of flux here.
The assembled output distinguishes between Primitives and Closures. Primitives are only allowed to return single values back, and are not allowed to do any higher-order procedure calls. Closures, on the other hand, have full access to the machine, but they are responsible for calling the continuation and popping off their arguments when they’re finished.
7.5 Values
All values should support the following functions
plt.runtime.toDomNode(x, mode): produces a dom representation. mode can be either ’write’, ’display’, or ’print’
plt.runtime.equals(x, y): tests if two values are equal to each other
7.5.1 Numbers
Numbers are represented with the js-numbers JavaScript library. We re-exports it as a plt.baselib.numbers namespace which provides the numeric tower API.
Example uses of the plt.baselib.numbers library include:
- Creating integers:
42
16
- Creating big integers:
plt.baselib.numbers.makeBignum("29837419826")
- Creating floats:
plt.baselib.numbers.makeFloat(3.1415)
- Predicate for numbers:
plt.baselib.numbers.isSchemeNumber(42)
- Adding two numbers together:
plt.baselib.numbers.add(42, plt.baselib.numbers.makeFloat(3.1415))
- Converting a plt.baselib.numbers number back into native JavaScript floats:
plt.baselib.numbers.toFixnum(...)
Do all arithmetic using the functions in the plt.baselib.numbers namespace. One thing to also remember to do is apply plt.baselib.numbers.toFixnum to any native JavaScript function that expects numbers.
7.5.2 Pairs, NULL, and lists
Pairs can be constructed with plt.runtime.makePair(f, r). A pair is an object with first and rest attributes. They can be tested with plt.runtime.isPair();
The empty list value, plt.runtime.NULL, is a single, distinguished value, so compare it with ===.
var aList = plt.runtime.makeList(3, 4, 5); |
The predicate plt.runtime.isList(x) takes an argument x and reports if the value is chain of pairs, terminates by NULL. At the time of this writing, it does NOT check for cycles.
7.6 Vectors
ref(n): get the nth element
set(n, v): set the nth element with value v
length: the length of the vector
7.7 Strings
Immutable strings are represented as regular JavaScript strings.
Mutable strings haven’t been mapped yet.
7.8 VOID
The distinguished void value is plt.runtime.VOID; functions implemented in JavaScript that don’t have a useful return value should return plt.runtime.VOID.
7.9 Undefined
The undefined value is JavaScript’s undefined.
7.10 EOF
The eof object is plt.runtime.EOF
7.10.1 Boxes
box.get(): returns the value in the box |
box.set(v): replaces the value in the box with v |
7.10.2 Structures
var Color = plt.runtime.makeStructureType( |
'color', // name |
false, // parent structure type |
3, // required number of arguments |
0, // number of automatically-filled fields |
false, // OPTIONAL: the auto-v value |
false // OPTIONAL: a guard procedure |
); |
constructor: create an instance of a structure type.
For example,var aColor = Color.constructor(3, 4, 5);
creates an instance of the Color structure type.predicate: test if a value is of the given structure type.
For example,Color.predicate(aColor) --> true
Color.predicate("red") --> false
accessor: access a field of a structure.
For example,var colorRed = function(x) { return Color.accessor(x, 0); };
var colorGreen = function(x) { return Color.accessor(x, 1); };
var colorBlue = function(x) { return Color.accessor(x, 2); };
mutator: mutate a field of a structure.
For example,var setColorRed = function(x, v) { return Color.mutator(x, 0, v); };
Color.type.prototype.toString = function() { |
return "rgb(" + colorRed(this) + ", " |
+ colorGreen(this) + ", " |
+ colorBlue(this) + ")"; |
}; |
7.11 Tests
The test suite in "tests/test-all.rkt" runs the test suite. You’ll need to run this on a system with a web browser, as the suite will evaluate JavaScript and make sure it is producing values. A bridge module in "tests/browser-evaluate.rkt" brings up a temporary web server that allows us to pass values between Racket and the JavaScript evaluator on the browser for testing output.
7.12 What’s in js-vm that’s missing from Whalesong?
(This section should describe what needs to get done next.)
immutable strings
numbers
pairs
null
void
vectors
immutable vectors
regexp
byteRegexp
character
placeholder
path
bytes
immutable bytes
keywords
hash
hasheq
struct types
exceptions
thread cells
big bang info
worldConfig
effectType
renderEffectType
readerGraph
)
abort-current-continuation
andmap
append
apply
argmax
argmin
arity-at-least-value
arity-at-least?
assoc
assq
assv
boolean=?
boolean?
box-immutable
box?
build-list
build-string
build-vector
byte?
bytes
bytes->immutable-bytes
bytes->list
bytes-append
bytes-copy
bytes-fill!
bytes-length
bytes-ref
bytes-set!
bytes<?
bytes=?
bytes>?
bytes?
caaar
caadr
caar
cadar
cadddr
caddr
cadr
call-with-continuation-prompt
call-with-current-continuation
call-with-values
call/cc
cdaar
cdadr
cdar
cddar
cdddr
cddr
char->integer
char-alphabetic?
char-ci<=?
char-ci<?
char-ci=?
char-ci>=?
char-ci>?
char-downcase
char-lower-case?
char-numeric?
char-upcase
char-upper-case?
char-whitespace?
char<=?
char<?
char=?
char>=?
char>?
char?
complex?
compose
cons?
continuation-mark-set->list
continuation-mark-set?
continuation-prompt-tag?
current-continuation-marks
current-inexact-milliseconds
current-seconds
default-continuation-prompt-tag
e
eighth
empty
empty?
eof
eof-object?
equal~?
even?
exact->inexact
exn-continuation-marks
exn-message
exn:fail:contract:arity?
exn:fail:contract:divide-by-zero?
exn:fail:contract:variable?
exn:fail:contract?
exn:fail?
exn?
explode
false
false?
fifth
filter
first
foldl
foldr
for-each
fourth
gensym
hash-for-each
hash-map
hash-ref
hash-remove!
hash-set!
hash?
identity
immutable?
implode
inexact->exact
inexact?
int->string
integer->char
js-function?
js-object?
js-value?
length
list->bytes
list-tail
make-arity-at-least
make-bytes
make-continuation-prompt-tag
make-exn
make-exn:fail
make-exn:fail:contract
make-exn:fail:contract:arity
make-exn:fail:contract:divide-by-zero
make-exn:fail:contract:variable
make-hash
make-hasheq
make-placeholder
make-reader-graph
make-struct-type
make-thread-cell
map
max
memf
memq
memv
min
negative?
null
odd?
ormap
pi
placeholder-get
placeholder-set!
positive?
posn?
print-values
quicksort
raise
rational?
real?
remove
replicate
rest
second
seventh
sixth
sleep
sort
string
string->immutable-string
string->int
string->list
string-alphabetic?
string-ci<=?
string-ci<?
string-ci=?
string-ci>=?
string-ci>?
string-copy
string-fill!
string-ith
string-lower-case?
string-numeric?
string-ref
string-upper-case?
string-whitespace?
string<=?
string<?
string>=?
string>?
struct-accessor-procedure?
struct-constructor-procedure?
struct-mutator-procedure?
struct-predicate-procedure?
struct-type?
struct?
subbytes
symbol=?
third
throw-cond-exhausted-error
true
undefined?
values
vector?
verify-boolean-branch-value
void?
write
xml->s-exp
(I should catalog the bug list in GitHub, as well as the feature list, so I have a better idea of what’s needed to complete the project.)
(We also need a list of the primitives missing that prevent us from running racket/base; it’s actually a short list that I’ll be attacking once things stabilize.)
8 The Whalesong language
(require (planet dyoo/whalesong:1:2/lang/base)) |
This needs to at least show all the bindings available from the base language.
true : boolean |
false : boolean |
pi : number |
e : number |
(let/cc id body ...) |
(null? ...) |
(not ...) |
(eq? ...) |
(equal? ...) |
(void ...) |
8.1 IO
(current-output-port ...) |
(current-print ...) |
(write ...) |
(write-byte ...) |
(display ...) |
(newline ...) |
(format ...) |
(printf ...) |
(fprintf ...) |
(displayln ...) |
8.2 Numeric operations
(number? ...) |
(+ ...) |
(- ...) |
(* ...) |
(/ ...) |
(= ...) |
(add1 ...) |
(sub1 ...) |
(< ...) |
(<= ...) |
(> ...) |
(>= ...) |
(abs ...) |
(quotient ...) |
(remainder ...) |
(modulo ...) |
(gcd ...) |
(lcm ...) |
(floor ...) |
(ceiling ...) |
(round ...) |
(truncate ...) |
(numerator ...) |
(denominator ...) |
(expt ...) |
(exp ...) |
(log ...) |
(sin ...) |
(sinh ...) |
(cos ...) |
(cosh ...) |
(tan ...) |
(asin ...) |
(acos ...) |
(atan ...) |
(sqr ...) |
(sqrt ...) |
(integer-sqrt ...) |
(sgn ...) |
(make-rectangular ...) |
(make-polar ...) |
(real-part ...) |
(imag-part ...) |
(angle ...) |
(magnitude ...) |
(conjugate ...) |
(string->number ...) |
(number->string ...) |
(random ...) |
(exact? ...) |
(integer? ...) |
(zero? ...) |
8.3 String operations
(string? s) |
(string=? ...) |
(string->symbol ...) |
(string-length ...) |
(string-append ...) |
8.4 Symbol operations
(symbol? ...) |
8.5 List operations
(pair? ...) |
(cons ...) |
(car ...) |
(cdr ...) |
(list ...) |
(length ...) |
(append ...) |
(reverse ...) |
(map ...) |
(for-each ...) |
(member ...) |
(list-ref ...) |
(memq ...) |
(assq ...) |
8.6 Vector operations
(vector? ...) |
(make-vector ...) |
(vector ...) |
(vector-length ...) |
(vector-ref ...) |
(vector-set! ...) |
(vector->list ...) |
(list->vector ...) |
9 Acknowledgements
jshashtable (http://www.timdown.co.uk/jshashtable/)
js-numbers (http://github.com/dyoo/js-numbers/)
JSON (http://www.json.org/js.html)
jquery (http://jquery.com/)
Google Closure Compiler (http://code.google.com/p/closure-compiler/)
The following folks have helped tremendously in the implementation of Whalesong by implementing libraries, giving guidence, and suggesting improvements:
Ethan Cecchetti
Scott Newman
Zhe Zhang
Jens Axel Søgaard
Shriram Krishnamurthi
Emmanuel Schanzer