Wednesday, August 13, 2008

The Best State Machine is a Coroutine..

Occasionally, people will ask me why I think Wasp is better than other Lisps for network programming. One of the big reasons is its pervasive use of lightweight processes as coroutines, a technique often used by Erlang programmers and, occasionally, advanced Scheme libraries. An example, from "lib/collate-filter":
;;; -- SNIP --
     ;;; "in" is an input channel, waiting on it causes a process to block.
     ;;; "out" is an output channel, which connects to the next coroutine.

      (define buf (make-string))
    
      (define (wait-for-any)
        (forever
          (define evt (wait in))
          (when (string? evt)
            (string-append! buf evt)
            (return))
          (send evt out)))
      
      (define (wait-for-field len)
        (while (< (string-length buf) len)
          (wait-for-any))
        (string-read! buf len))
    
      (define (wait-for-int)
        (string->quad (wait-for-field 4)))
Wait-for-int causes the process to idle until it has four bytes in the buffer, then permits the process to continue. If a process got 30 bytes at once, then it would be able to read 7 integers successively without having to idle at all, then would need to idle for at least 2 more bytes. Also visible, above, is some logic forwarding any status messages outwards; that is a common idiom found in filters -- if they can't handle the event, they assume that something further along in the chain would. A filter is a process that consumes data from an input and yields results to an output, acting very much like a coroutine. Since they can be composed into chains, Wasp can make many tricky data flows easy to express and understand. For example, MOSREF's outbound cryptography chain:
;;; -- SNIP --
     (input-chain recv
                  (decrypt-filter recv-decrypt)
                  (check-collation-filter)
                  (check-checksum-filter crc32))
The first argument is the input that the next filter will wait on; each subsequent form spawns a new coroutine, supplying them with the supplied arguments for initialization. The result is a new input that provides a decrypted and carefully groomed sequence of messages from another MOSREF node. It's almost embarassing how simple that looks, like much of MOSREF's code. Much of my effort has been invested in Wasp Lisp for complicated I/O applications.