►Whalesong: a Racket to Java Script compiler
1 Introduction
2 Getting started
2.1 Installing Whalesong
2.2 Running Whalesong
3 Extended example
4 Reference
4.1 The "whalesong" command-line
5 Internals
5.1 Architecture
5.2 parser/ parse-bytecode.rkt
5.3 compiler/ compiler.rkt
5.4 js-assembler/ assemble.rkt
5.5 Tests
5.6 Incomplete features
6 Acknowledgements
Version: 5.1.1

Whalesong: a Racket to JavaScript compiler

Danny Yoo <[email protected]>

Warning: this is work in progress!

1 Introduction

Whalesong is a compiler from Racket to JavaScript; it takes Racket programs and translates them so that they can run stand-alone on a user’s web browser. It should allow Racket programs to run with little modification, and provide access through the foreign-function interface to native JavaScript APIs. The included runtime library also includes a framework to programming the web in functional event-driven style.

The GitHub source repository to Whalesong can be found at https://github.com/dyoo/whalesong.

Prerequisites: at least Racket 5.1.1, and a Java 1.6 SDK.

2 Getting started

2.1 Installing Whalesong

At the time of this writing, Whalesong hasn’t been deployed to PLaneT yet, so getting it requires doing a little bit of manual work. The steps are:

We can check it out of the source repository in GitHub; the repository can be checked out by using git clone. At the command-line, clone the tree with:

 $ git clone git://github.com/dyoo/whalesong.git

This should check the repository in the current directory.

Next, let’s set up a PLaneT development link. Make sure you are in the parent directory that contains the "whalesong" repository, and then run this on your command line:

$ planet link dyoo whalesong.plt 1 0 whalesong

Finally, we need to set up Whalesong with raco setup. Here’s how to do this at the command line:

$ raco setup -P dyoo whalesong.plt 1 0

This should compile Whalesong, as well as set up the "whalesong" executable. Any time the source code in "whalesong" changes, we should repeat this raco setup step again.

At this point, you should be able to rung "whalesong" from the command line.

$ ./whalesong

Expected one of the following: [build, get-runtime, get-javascript].

and if this does appear, then Whalesong should be installed successfully.

2.2 Running Whalesong

Let’s try making a simple, standalone executable. At the moment, the program must be written in the base language of (planet dyoo/whalesong). This restriction currently prevents arbitrary racket/base programs from compiling, and the developers will be working to remove this restriction.

Write a "hello.rkt" with the following content

"hello.rkt"

#lang planet dyoo/whalesong
(display "hello world")
(newline)
This program can be executed in Racket,

$ racket hello.rkt

hello world

$

and it can also be packaged with "whalesong".

$ whalesong build hello.rkt

 

$ ls -l hello.xhtml

-rw-rw-r-- 1 dyoo nogroup 692213 Jun  7 18:00 hello.xhtml

Running whalesong build on a Racket program will produce a self-contained ".xhtml" file. If you open this file in your favorite web browser, you should see a triumphant message show on screen.

We can do something slightly more interesting. Let’s write a Whalesong program that accesses the JavaScript DOM. Call this file "dom-play.rkt".

"dom-play.rkt"

#lang planet dyoo/whalesong
 
;; Uses the JavaScript FFI, which provides bindings for:
;; $ and call
(require (planet dyoo/whalesong/js))
 
;; insert-break: -> void
(define (insert-break)
  (call ($ "<br/>") "appendTo" body)
  (void))
 
;; write-message: any -> void
(define (write-message msg)
  (void (call (call (call ($ "<span/>") "text" msg)
                    "css" "white-space" "pre")
              "appendTo"
              body)))
 
;; Set the background green, and show some content
;; on the browser.
(void (call body "css" "background-color" "lightgreen"))
(void (call ($ "<h1>Hello World</h1>") "appendTo" body))
(write-message "Hello, this is a test!")
(insert-break)
(let loop ([i 0])
  (cond
   [(= i 10)
    (void)]
   [else
    (write-message "iteration ") (write-message i)
    (insert-break)
    (loop (add1 i))]))
This program uses the JQuery API provided by (planet dyoo/whalesong/js), as well as the native JavaScript FFI to produce output on the browser. If w run Whalesong on this program, and view the resulting "dom-play.xhtml" in your web browser, we should see a pale, green page with some output.

3 Extended example

4 Reference

(This section should describe the whalesong language.)

4.1 The "whalesong" command-line

(This section should describe the whalesong launcher and the options we can use.)

(We want to add JavaScript compression here as an option.)

(We also need an example that shows how to use the get-javascript and get-runtime commands to do something interesting...)

5 Internals

Please skip this section if you’re a regular user: this is really notes internal to Whalesong development, and is not relevant to most people.

These are notes that describe the internal details of the implementation, including the type map from Racket values to JavaScript values. It should also describe how to write FFI bindings, eventually.

5.1 Architecture

The basic idea is to reuse most of the Racket compiler infrastructure. We use the underlying Racket compiler to produce bytecode from Racket source; it also performs macro expansion and module-level optimizations for us. We parse that bytecode using the compiler/zo-parse collection to get an AST, compile that to an intermediate language, and finally assemble JavaScript.

                    AST                 IL

parse-bytecode.rkt -----> compiler.rkt ----> assembler.rkt

 

The IL is intended to be translated straightforwardly. We currently have an assembler to JavaScript "js-assembler/assemble.rkt", as well as a simulator in "simulator/simulator.rkt". The simulator allows us to test the compiler in a controlled environment.

5.2 parser/parse-bytecode.rkt

(We try to insulate against changes in the bytecode structure by using the version-case library to choose a bytecode parser based on the Racket version number. Add more content here as necessary...)

5.3 compiler/compiler.rkt

This translates the AST to the intermediate language. The compiler has its origins in the register compiler in Structure and Interpretation of Computer Programs with some significant modifications.

Since this is a stack machine, we don’t need any of the register-saving infrastructure in the original compiler. We also need to support slightly different linkage structures, since we want to support multiple value contexts. We’re trying to generate code that works effectively on a machine like the one described in http://plt.eecs.northwestern.edu/racket-machine/.

The intermediate language is defined in "il-structs.rkt", and a simulator for the IL in "simulator/simulator.rkt". See "tests/test-simulator.rkt" to see the simulator in action, and "tests/test-compiler.rkt" to see how the output of the compiler can be fed into the simulator.

The assumed machine is a stack machine with the following atomic registers:

  • val: value

  • proc: procedure

  • argcount: number of arguments

and two stack registers:

  • env: environment stack

  • control: control stack

5.4 js-assembler/assemble.rkt

The intent is to potentially support different back end generators for the IL. "js-assembler/assemble.rkt" provides a backend for JavaScript.

The JavaScript assembler plays a few tricks to make things like tail calls work:

Otherwise, the assembler is fairly straightforward. It depends on library functions defined in "runtime-src/runtime.js". As soon as the compiler stabilizes, we will be pulling in the runtime library in Moby Scheme into this project. We are right in the middle of doing this, so expect a lot of flux here.

The assembled output distinguishes between Primitives and Closures. Primitives are only allowed to return single values back, and are not allowed to do any higher-order procedure calls. Closures, on the other hand, have full access to the machine, but they are responsible for calling the continuation and popping off their arguments when they’re finished.

5.5 Tests

The test suite in "tests/test-all.rkt" runs the test suite. You’ll need to run this on a system with a web browser, as the suite will evaluate JavaScript and make sure it is producing values. A bridge module in "tests/browser-evaluate.rkt" brings up a temporary web server that allows us to pass values between Racket and the JavaScript evaluator on the browser for testing output.

5.6 Incomplete features

(This section should describe what needs to get done next.)

The only types that are mapped so far are

  • immutable strings

  • numbers

  • pairs

  • null

  • void

  • vectors

We need to bring around the following types previously defined in js-vm: (This list will shrink as I get to the work!)

  • immutable vectors

  • regexp

  • byteRegexp

  • character

  • box

  • placeholder

  • path

  • bytes

  • immutable bytes

  • keywords

  • hash

  • hasheq

  • color

  • structs

  • struct types

  • exceptions

  • thread cells

  • big bang info

  • worldConfig

  • effectType

  • renderEffectType

  • readerGraph

What are the list of primitives in "js-vm-primitives.js"? They are:

  • *

  • +

  • -

  • /

  • <

  • <=

  • =

  • =~

  • >

  • >=

  • abort-current-continuation

  • abs

  • acos

  • add1

  • andmap

  • angle

  • append

  • apply

  • argmax

  • argmin

  • arity-at-least-value

  • arity-at-least?

  • asin

  • assoc

  • assq

  • assv

  • atan

  • boolean=?

  • boolean?

  • box

  • box-immutable

  • box?

  • build-list

  • build-string

  • build-vector

  • byte?

  • bytes

  • bytes->immutable-bytes

  • bytes->list

  • bytes-append

  • bytes-copy

  • bytes-fill!

  • bytes-length

  • bytes-ref

  • bytes-set!

  • bytes<?

  • bytes=?

  • bytes>?

  • bytes?

  • caaar

  • caadr

  • caar

  • cadar

  • cadddr

  • caddr

  • cadr

  • call-with-continuation-prompt

  • call-with-current-continuation

  • call-with-values

  • call/cc

  • car

  • cdaar

  • cdadr

  • cdar

  • cddar

  • cdddr

  • cddr

  • cdr

  • ceiling

  • char->integer

  • char-alphabetic?

  • char-ci<=?

  • char-ci<?

  • char-ci=?

  • char-ci>=?

  • char-ci>?

  • char-downcase

  • char-lower-case?

  • char-numeric?

  • char-upcase

  • char-upper-case?

  • char-whitespace?

  • char<=?

  • char<?

  • char=?

  • char>=?

  • char>?

  • char?

  • complex?

  • compose

  • conjugate

  • cons

  • cons?

  • continuation-mark-set->list

  • continuation-mark-set?

  • continuation-prompt-tag?

  • cos

  • cosh

  • current-continuation-marks

  • current-inexact-milliseconds

  • current-inspector

  • current-print

  • current-seconds

  • default-continuation-prompt-tag

  • denominator

  • display

  • e

  • eighth

  • empty

  • empty?

  • eof

  • eof-object?

  • eq?

  • equal?

  • equal~?

  • eqv?

  • error

  • even?

  • exact->inexact

  • exact?

  • exn-continuation-marks

  • exn-message

  • exn:fail:contract:arity?

  • exn:fail:contract:divide-by-zero?

  • exn:fail:contract:variable?

  • exn:fail:contract?

  • exn:fail?

  • exn?

  • exp

  • explode

  • expt

  • false

  • false?

  • fifth

  • filter

  • first

  • floor

  • foldl

  • foldr

  • for-each

  • format

  • fourth

  • gcd

  • gensym

  • hash-for-each

  • hash-map

  • hash-ref

  • hash-remove!

  • hash-set!

  • hash?

  • identity

  • imag-part

  • immutable?

  • implode

  • inexact->exact

  • inexact?

  • int->string

  • integer->char

  • integer-sqrt

  • integer?

  • js-function?

  • js-object?

  • js-value?

  • lcm

  • length

  • list

  • list*

  • list->bytes

  • list->string

  • list->vector

  • list-ref

  • list-tail

  • list?

  • log

  • magnitude

  • make-arity-at-least

  • make-bytes

  • make-continuation-prompt-tag

  • make-exn

  • make-exn:fail

  • make-exn:fail:contract

  • make-exn:fail:contract:arity

  • make-exn:fail:contract:divide-by-zero

  • make-exn:fail:contract:variable

  • make-hash

  • make-hasheq

  • make-placeholder

  • make-polar

  • make-reader-graph

  • make-rectangular

  • make-string

  • make-struct-field-accessor

  • make-struct-field-mutator

  • make-struct-type

  • make-thread-cell

  • make-vector

  • map

  • max

  • member

  • memf

  • memq

  • memv

  • min

  • modulo

  • negative?

  • newline

  • not

  • null

  • null?

  • number->string

  • number?

  • numerator

  • odd?

  • ormap

  • pair?

  • pi

  • placeholder-get

  • placeholder-set!

  • positive?

  • posn?

  • print-values

  • printf

  • procedure-arity

  • procedure-arity-includes?

  • procedure?

  • quicksort

  • quotient

  • raise

  • random

  • rational?

  • real-part

  • real?

  • remainder

  • remove

  • replicate

  • rest

  • reverse

  • round

  • second

  • set-box!

  • seventh

  • sgn

  • sin

  • sinh

  • sixth

  • sleep

  • sort

  • sqr

  • sqrt

  • string

  • string->immutable-string

  • string->int

  • string->list

  • string->number

  • string->symbol

  • string-alphabetic?

  • string-append

  • string-ci<=?

  • string-ci<?

  • string-ci=?

  • string-ci>=?

  • string-ci>?

  • string-copy

  • string-fill!

  • string-ith

  • string-length

  • string-lower-case?

  • string-numeric?

  • string-ref

  • string-set!

  • string-upper-case?

  • string-whitespace?

  • string<=?

  • string<?

  • string=?

  • string>=?

  • string>?

  • string?

  • struct-accessor-procedure?

  • struct-constructor-procedure?

  • struct-mutator-procedure?

  • struct-predicate-procedure?

  • struct-type?

  • struct?

  • sub1

  • subbytes

  • substring

  • symbol->string

  • symbol=?

  • symbol?

  • tan

  • third

  • throw-cond-exhausted-error

  • true

  • truncate

  • unbox

  • undefined?

  • values

  • vector

  • vector->list

  • vector-length

  • vector-ref

  • vector-set!

  • vector?

  • verify-boolean-branch-value

  • void

  • void?

  • write

  • xml->s-exp

  • zero?

(I should catalog the bug list in GitHub, as well as the feature list, so I have a better idea of what’s needed to complete the project.)

(We also need a list of the primitives missing that prevent us from running racket/base; it’s actually a short list that I’ll be attacking once things stabilize.)

6 Acknowledgements

Whalesong uses code and utilities from the following external projects: