Wasp Lisp Developments

WaspDoc Bug Fixes

2009-12-12T00:49:00.000-08:00

Looks like there were a couple TConc-related bugs lying around in the Waspdoc argument parser. Fixed for posterity.. Perhaps I will eventually get around to finishing waspdoc/check-source.ms and waspdoc/c-file.ms.

Fixing Static vs. Dynamic

2009-12-06T00:20:00.001-08:00

Courtesy of Chris Double, Wasp's build system will now statically link libevent on Linux and other Unixen for the stub virtual machine. This improves the portability of standalone executables compiled using Wasp's stub virtual machine, since libevent is not quite ubiquitous.

Also patched was an annoying little warning in "vm/salsa.c" related to an implicit pointer cast, and the process loop now performs a 100 microsecond sleep when all processes are polling. (Previously, this was handled by the OS process, which doesn't correctly handle the possibility of more than one process polling for external events.)

Portable Terminal I/O with SCUD

2008-12-26T05:53:00.000-08:00

A new release will be uploaded, today, 0.4.1, with some minor bugfixes and a simple terminal abstraction library, "lib/scud." This library supports cursor relocation, changing terminal colors, and clearing the screen on both VT100-capable terminals and Windows. Also, a new primitive has been added, "unbuffer-console", which puts the Windows and UNIX standard input into a raw, unbuffered mode. This makes it possible to capture individual keypresses by the user without waiting for linebreaks.

Why the fuss with terminals? Because I have been working on a lot of console mode applications, and this module gives me the bare minimum while supporting both Windows and UNIX platforms. There is still a lot of planning and work ongoing with 0.5, but not much of it is suitable for release, yet.

Happy holidays!

Kicking Wasp 0.4 Out of the Door..

2008-09-18T02:46:00.001-07:00

This has been languishing in my To Do list for the past couple weeks, introducing Wasp Lisp 0.4. This version provides a lot of little improvements in the network and console I/O on Windows and Linux, and, more importantly, contains a complete port of MOSREF 2.0 from MOSREF 2.0b3. All of the functionality we demonstated two years ago from MOSREF is in this newest port of MOSREF, but with different ciphers. Stability and speed have greatly improved, compared to the 2006 prototype, largely due to the use of the Salsa/20 stream cipher for the session cipher.

Work will begin soon on Wasp 0.5, which consist of more primitive namespace cleanup, bug fixes and, hopefully, WaspDoc.

Link Dump: Plugins and DLLs

2008-09-02T04:36:00.000-07:00

MOSREF requires the creation of statically linked "monolithic" binaries for Drones. Wasp, prior to 0.3, supported loading plugins written in C by bundling the entire virtual machine into a shared object file. The following is a list of links I have been looking at, trying to find a way to support plugins while keeping the virtual machine itself in the executable:

FlexDLL -- Emulates UNIX dlopen on Win32.
EDLL -- Reimplements the dynamic linker for Windows; also catalogues common workarounds for the DLL plugin paradox.
DLL Creation in MinGW -- A succinct tutorial on LoadLibrary / GetProcAddress.
The University of Colorado MinGW FAQ -- Contains a nice list of common MinGW gotchas.
The Program Library HOWTO -- More about how to design libraries on UNIX systems than Win32, but discusses those techniques thoroughly.
Wikipedia: Dynamic Loading -- Contains an interesting technique for DLLs to access the original process's symbols.
Dynamic Linking in Linux and Windows, part two -- More in-depth discussion of DLLs and their limitations.

The criteria for a successful solution:

Permits a plugin subsystem to call routines in the virtual machine. (Blocks naked use of DLLs and LoadLibrary / GetProcAddress.)
Shares data structures in the virtual machine process with all plugins.
Does not bloat the virtual machine.
Does not require special effort by plugin authors. (Disqualifying passing a structure of API function pointers to the plugin.)
Does not tightly couple the plugin to the binary. (Disqualifying various --output-implib tricks.)
Does not excessively complicate the build process. (Disqualifying LibTool.)

MOSREF Shell Shock..

2008-09-01T08:58:00.000-07:00

Work has been quiet but steady on MOSREF, with some very intense work on restoring the "proxy" command and improving the Console's resilience against malfunctioning Drones and noise. All of the functionality from MOSREF 2.0b3 has been restored, with the sole exception of the "scan" command; it took me longer than expected, but a very subtle bug in vm/os.c was locking up Drones when a specific sequence of proxy sessions was introduced. I also discovered an interesting hiccup in GCC when compiling vm/format.c with -Os or -O3. Until I get a chance to examine the optimizations involved, the Makefile will default to compiling with the more conservative -O2 optimization flags.

Progress on MOSREF..

2008-08-17T06:10:00.000-07:00

As I have mentioned in the last few posts, MOSREF is being ported to the Wasp Virtual Machine. There have been a few changes in this new version, mostly the reduction of code complexity, and some improved handling of man in the middle attacks. Previously, each message had a hash used to detect transport compromise, but this did not prevent a permuted message header from tricking the Console or a Drone into waiting an indefinite amount of time for a ludicrous amount of data. The new model uses a two-tier checksum model; each message is composed of zero or more blocks. The message has its checksum, and each block has its own checksum. To increase the difficulty of spoofing valid blocks, the block sizes are randomly sized, which ensures that the offset of the third and successive CRCs are difficult to predict. Next up, getting the SOCKS 5 proxy working again and further code complexity reductions.

The Best State Machine is a Coroutine..

2008-08-13T23:55:00.000-07:00

Occasionally, people will ask me why I think Wasp is better than other Lisps for network programming. One of the big reasons is its pervasive use of lightweight processes as coroutines, a technique often used by Erlang programmers and, occasionally, advanced Scheme libraries. An example, from "lib/collate-filter":

;;; -- SNIP --
     ;;; "in" is an input channel, waiting on it causes a process to block.
     ;;; "out" is an output channel, which connects to the next coroutine.

      (define buf (make-string))
    
      (define (wait-for-any)
        (forever
          (define evt (wait in))
          (when (string? evt)
            (string-append! buf evt)
            (return))
          (send evt out)))
      
      (define (wait-for-field len)
        (while (< (string-length buf) len)
          (wait-for-any))
        (string-read! buf len))
    
      (define (wait-for-int)
        (string->quad (wait-for-field 4)))

Wait-for-int causes the process to idle until it has four bytes in the buffer, then permits the process to continue. If a process got 30 bytes at once, then it would be able to read 7 integers successively without having to idle at all, then would need to idle for at least 2 more bytes. Also visible, above, is some logic forwarding any status messages outwards; that is a common idiom found in filters -- if they can't handle the event, they assume that something further along in the chain would. A filter is a process that consumes data from an input and yields results to an output, acting very much like a coroutine. Since they can be composed into chains, Wasp can make many tricky data flows easy to express and understand. For example, MOSREF's outbound cryptography chain:

;;; -- SNIP --
     (input-chain recv
                  (decrypt-filter recv-decrypt)
                  (check-collation-filter)
                  (check-checksum-filter crc32))

The first argument is the input that the next filter will wait on; each subsequent form spawns a new coroutine, supplying them with the supplied arguments for initialization. The result is a new input that provides a decrypted and carefully groomed sequence of messages from another MOSREF node. It's almost embarassing how simple that looks, like much of MOSREF's code. Much of my effort has been invested in Wasp Lisp for complicated I/O applications.

WaspVM Focus for 0.4..

2008-08-12T21:54:00.000-07:00

In the key of Hubert J. Farnsworth, Good news, everyone! Wes Brown has offered to fund a port of MOSREF 2.0 to WaspVM, along with some other things. WaspVM 0.4 will contain the necessary primitive functionality to support MOSREF, and some modifications to the compiler to restore cross-platform compilation. After that, I will port MOSREF back to WaspVM, and maintain MOSREF until at least the release of Wasp Lisp 1.0. Wes has a couple more contributions to the project, but we need to hammer out the details before I make more announcements.

Wasp Lisp 0.3 on Mac OS X Fixed

2008-08-10T04:57:00.000-07:00

There is a bug in LibEvent on Mac OS X when using any I/O scheduling method more sophisticated than select(2); Wasp Lisp 0.3.2 provides support for Mac OS X by disabling the more efficient kqueue(2) and poll(2) I/O schedulers on that platform and that platform only. It's disappointing, really, and any Mac OS X aficionado is more than welcome to find a more satisfactory way to achieve this -- I only use OS X when under duress. :)

Barely 24 Hours In..

2008-08-08T05:53:00.000-07:00

.. and someone has already poked holes in the new I/O loop. I've cleaned up some issues related to the I/O loop, noninteractive programs, and some inconsistent I/O states that could crop up from exotic interleaving of the wait-input, send-output and exit primitives. Version 0.3.1 is now up on "Wasp Lisp Releases," with fixes for all the issues above and a wee bit more work on WaspDoc. Wes Brown has found some I/O problems on Mac OS X Leopard; I will dig out my old Mac OS X box and see what the issue is this evening or tomorrow. If it isn't just a compile problem, 0.3.2 may be following in the next 24 hours. Viva La Beta!

Ephemeral Security Found at an Archive Near You

2008-08-07T18:25:00.001-07:00

Found this when digging for some old EphSec information for an email: Ephemeral Security: Mosquito Lisp Easy to Learn Looks like Archive.Org backed up the entire EphSec blog..

WaspVM 0.3 Released!

2008-08-07T05:49:00.000-07:00

After nine months in development, WaspVM 0.3 is complete. 0.3 will be the last alpha release of WaspVM, with development focus moving on to improving the reliability and usability of the functionality present in WaspVM, and writing documentation. Since WaspVM 0.2, the autoconf build process has been removed, returning to a Mosquito-style Makefile.cf; we have also seen some serious improvements in the I/O loop, including bugfixes for some issues that have been part of MOSVM and WASPVM for years. (Mostly win32-related.. And someone please kick me before I autoconf again..) The source package and prebuilt windows executables can be found at Wasp Lisp Releases. Please, try these packages out and let me know if you have any problems building, installing or using them. Every bug found now is a bug that won't be there for Wasp 1.0..

Win32 and LibEvent

2008-08-07T05:05:00.000-07:00

I found a hack that lets my win32 console workaround cohabitate with LibEvent for the present; if I wedge a SleepEx( 0, TRUE ) before I let event_loop take away my CPU, the console can take advantage of QueueAPC to post any I/O updates to the main virtual machine thread. It's a grotesque hack, but necessary because LibEvent's WIN32 logic uses WIN32's select emulation. Long term, I need to fire off a patch upstream to LibEvent, removing the poor performance of select, and moving to the more modern I/O completion callbacks. (Well, as modern as anything derived from VMS network programming can be..) But, for now, I can safely release WaspVM 0.3 and see about getting work done on documentation and perfection.

HTTP Server Drag Racing

2008-08-06T07:30:00.000-07:00

I've finished another wave of tweaks to the Wasp I/O event loop, basically just cleaning up some leaks and crashers that snuck in. The current product is pretty stable, but still not ready to build on Windows. (To support the console on Windows, I have to not use LibEvent, which is disappointing as always..) To stress the garbage collector, I have been using the Siege load test utility against the Wasp lib/http-server module. Even at 250 concurrent connections, Wasp manages to perform comparably to HTTP servers written in other dynamic languages. There are sporadic bits of latency, though, due to the stop-the-world garbage collector used by Wasp. (It'll be replaced after 1.0.. Trying to keep my priorities straight, here..) For comparisons between Wasp and THTTPD, see "Drag Racing Against THTTPD Using Siege." It's not terribly rigorous -- just the sort of thing that crops up when you have load testing tools lying around. :) (As always, the bleeding edge source is available at sourceforge; as soon as I get Win32 builds where I want them, I'll cut a new distribution for those chained to Microsoft..)

Temptations and Invisible Peer Pressure

2008-08-03T07:27:00.000-07:00

I have developed a little hobby over the past couple years -- going out and finding mentions of Mosquito, MOSREF or Ephemeral Security that also suppose that the Mosquito project has gone dead and I have disappeared off the face of the earth. It is really my own fault, Wes has always been better than me at self-promotion; I do things like Mosquito just to entertain myself without considering that other people might find them useful. Most of these mentions are people who were in the DC14 audience who were really looking forwards to using MOSREF 2.0 for their own research and professional use. I feel a certain amount of sadness when I realize that I disappointed these people when I veered off into language design. People seem to be daunted by the challenge of learning Mosquito Lisp and making their own improvements to MOSREF, which is a shame. Perhaps in the near future I'll take a day or two to port MOSREF for those hopeful few who are still interested. It'll be a nice holiday from doing the responsible thing: focusing on getting Wasp to a production-quality level.

Return of the I/O Loop

2008-07-29T19:21:00.000-07:00

The I/O loop, using Niels Provos' LibEvent, is back in the WaspVM trunk, along with some of the TCP/IP functionality and expanded timeout functionality. This doesn't work on Windows at the moment, because Windows requires Sockets, the Console and Devices to be treated differently; I will probably have to write a new I/O loop from scratch for Windows. (We had this problem in Mosquito, too; we just ignored the console, and used netcat to provide a console interface..) I know that the I/O loop was supposed to be deferred until after WaspDoc, but we were getting into some dire straits at work, with a few devices with very primitive serial chipsets. If we uploaded parameters faster than roughly 16 bytes per 100ms, the buffers would overflow and the device would crash. The new pause primitive, which accepts an optional duration in milliseconds, lets a process rest periodically to let the wretched controller to digest each transmission. Originally, I wrote this script using Python, but.. Bleh. Too complex. :)

Interoperability Using Google Buffers

2008-07-09T07:01:00.000-07:00

Work has been steady but slow on Wasp; as usual, I have been less interested in building that crucial foundation for a Wasp community than in using Wasp itself for my pet projects. Work on Wasp Core has been centered around getting the old LibEvent event loop restored, and backporting the TCP listener from Mosquito. Parts of WaspDoc have been backported as well, so it is easier to see what a Wasp Lisp module imports and exports. Outside of the Core, I have been having a lot of fun porting Google's Protocol Buffers API to Wasp. Google's protocol design resembles a less verbose version of the "Sickle" protocol I have used internally in my MUD servers; very elegant and more compact than my designs. The "protocol compiler" used by Google reminds me a lot of the Quincy protocol compiler we used at Ephemeral Security in IPAF, a protocol analysis framework that could be considered a predecessor to Mosquito's concurrency model. It is difficult to balance laying groundwork for Wasp users with my own entertainment.. It doesn't help that my wife is on me to finish the latest MUD server for her friends to use.

WaspVM Rewind Installs

2008-06-10T10:27:00.000-07:00

Just a quick update; the build system for WaspVM Rewind now builds subsystems again, the install script works, and a few warts that used to trouble the subsystems are gone. I need to write a cron job that watches for me trying to bugger up my projects with autotools and applies a startling electric current to my chair..

The Wasp Lisp Rewind

2008-06-07T04:55:00.000-07:00

First, some back of the envelope statistics: there are 283 primitives in the WaspVM virtual machine, as of 2008/05/25. Of these, 116 are used by the Wasp REPL -- determined by some clever partial compilation and finding the subset of global variable references that have the same name as a primitive. None of these primitives have had documentation outside of the CHANGES file since WaspVM forked from MOSVM. Few of these have had test cases written for them, and none of these test cases could be considered even slightly rigorous. For a project which wants to achieve a production-quality environment, this is dire news.

When I made my decision last week to increase my efforts with Wasp Lisp, it was obvious that feature creep was running away with the language (again), and it would be necessary to get out the refactoring chainsaw. My new objective is to get the virtual machine primitive count down to those currently used by the compiler, then improve code, test, and work on documentation quality.

After we have 100% test coverage of primitive functionality, and 100% reference documentation of primitive objects contained by the Wasp Virtual Machine, it will be safe to bring back these features, along with a number of modules that have bitrotted since the fork.

It will not be glorious work, but the resulting polish should be worth it. The winnowing portion should be fairly quick, I have already removed a couple dozen and damned autoconf in the new WaspVM Rewind branch.

Surprised by Loud Noises..

2008-06-06T05:04:00.001-07:00

A few days ago, Wes Brown wrote an article on Matasano Chargen about design choice I made for the Mosquito Secure Remote Execution Framework (MOSREF) to go from Lua to Mosquito Lisp (MOSVM). Within a matter of hours there were some very interesting questions about why we had stopped working on Mosquito Lisp, and the usual outrage about my dislike for Lua. I was surprised to see how many people wanted to know why development of MOSVM had stopped; I had made some faulty assumptions about the community's interest in MOSREF, MOSVM, and what would happen when Wes and I moved on to other things. There has been some activity on Matasano Chargen and Pensieri di un Lunatico Minore about the future of Mosquito Lisp and MOSVM, reminding me that there were programmers interested in the MOSREF project because of our Lisp dialect, as well as security professionals. While I have continued working on my fork of MOSVM, Wasp Lisp, I have not been terribly public about it because I thought there was no audience. Due to this reminder that other people are, indeed, watching and interested, I will bump up Wasp in my personal priorities -- above Kabuki and KoVM; I really would love to have a production quality application of the concepts we introduced in Mosquito Lisp.

Return of Networking

2008-01-21T08:40:00.001-08:00

The latest patch restores (tcp-connect addr portno) -> connection and (resolve-ipv4 hostname) -> addr to WaspVM, using LibEvent for the network loop. At the moment, this has only been tested on Linux, and I have not had a chance to port it over to Win32, as I do not have an NT box up at the moment. I am getting a fair amount done, now that I have an Eee PC to drag into work with me -- I can tease away at the bug list at lunch. Another interesting change is the addition of arguments to spawn; spawn now accepts functions of any arity, and zero or more optional arguments to be supplied to the function when the process begins. For example:

>> (spawn print "-- Hello\n")
:: [process 0x12345]
-- Hello
>>

This capability has been in the Process API for a while, I have just been too busy with other things to tweak the spawn primitive.

Building on 64-bit Linux

2007-12-02T15:08:00.000-08:00

WaspVM and Mosquito were never intended to run natively on 64-bit architectures. While efforts were made to keep any size assumptions under control, it appears WaspVM isn't building properly in native 64-bit architectures. WaspVM will build if you specify "gcc32" as your CC, but it still won't bootstrap. ("CC=gcc32 ./build")

Mixing a Wasp VM With Threaded Applications

2007-11-24T07:02:00.000-08:00

My driving problem for Wasp has a GUI written in Qt. Like most GUI libraries, Qt is callback-happy, and expects to be the outer loop for the application; to get around this, I need to let the Wasp Virtual Machine run as its own thread, and use channels to bridge between the two. Since Wasp is not designed to be terribly friendly to native OS threads, with quite a few global variables in the Virtual Machine, I have added two C functions that pause the VM and resume it: wasp_seize_vm( ) and wasp_release_vm( ), respectively. These two functions wrap a simple mutex that is locked while the inner Wasp process loop is running, and released momentarily when Wasp is transitioning from one process to another. The GUI application uses wasp_seize_vm( ) to grab control of the GC system, send a message to the event channel, then release control with wasp_release_vm( ). With care, if you squint enough, the threads seem to act like Wasp processes -- as long as you don't touch any Wasp-managed objects. I use Qt's notifiation APIs in the GUI wrapper library to let Qt know when it needs to do the same. Vacation's over, time to drive home and start getting ready for work. (As opposed to Wasp, which is still strictly goofing off.)

WaspDoc Dump Module / Source Feature

2007-11-24T06:55:00.000-08:00

"Dump Module" is a neat new toy in the WaspDoc tree that grew out of WaspDoc's source parser. Given a source file, "waspdoc check source " will open the file, analyze its imports and exports, then print them to standard output. As you can see in the example below, it isn't perfect; things like doing (define func (lambda args ...)) will look like a variable export to WaspDoc, but it can be really handy when you need to know what the arguments for a given function are. Hopefully, by next weekend, WaspDoc will be able to do this for primitives as well; this data is used by WaspDoc to generate skeleton documentation files.

One Last Thing -- Dir-Read and Dir-Path

2007-11-19T06:39:00.000-08:00

Added two more primitives to support WaspDoc before I get away from my build environment: (dir-files path), which lists files contained by a directory, (Well, files, directories, named pipes, it's not terrible discriminating..) and (dir-path? path) which works much like (file-path? path) to identify valid directory paths. Both of these primitives will work under POSIX and Windows, thanks to MinGW.

Iterations Over Sequences

2007-11-18T19:06:00.001-08:00

Mosquito's implementation of iterators was derived from "SRFI-1: List Library," and restricted itself mostly to iterating over lists, much like the original. Since Mosquito and Wasp have much richer type libraries, it makes sense to generalize these functions to support other primitive sequence types, such as Sets, Dictionaries, Vectors and Channels. (Yes, channels can be considered sequences of a sort.) It is now possible to do the following:

>> (for-each send (current-input))

The input yielded by (current-input) is applied to the (iter value) function, which yields an iteration thunk. That thunk is called repeatedly by for-each until it raises a (done why) error. A slightly less verbose version:

>> (for-each send wait)

The iterator for a function is the function itself; iter makes the assumption that the supplied function is a generator. Part of this was already done in Mosquito, but only a couple functions supported it, it was never documented, and most sequences had to be converted into an intermediate list. Those oversights have been corrected. I am going on vacation this week, so, either expect a lot of changes, or none at all, next weekend. Waspdoc now lists all functions exported by a module, identifies its imports, and is coming along nicely.

Improvements in the Wasp Debugger..

2007-11-10T08:11:00.001-08:00

As you can see from the screenshot, Wasp's trace mode now works only on the process that enters trace. This should make it much easier to trace coroutines and subprocesses, especially in the REPL. (Previously, you often got an eyeful of the REPL's compiler busily chopping up your expressions.) Any sub-processes spawned by a process in trace mode will also start out in trace mode. Keep in mind that the REPL runs each expression within the REPL's process, so typing (enable-trace) at the REPL prompt is ill-advised. (For those of you who couldn't resist, just type in (disable-trace) to turn it back off when your terminal catches up..) Also, the output has been cleaned up quite a bit -- only call frames and returned values are displayed. I think that Wasp's compiler has matured enough for end users to resent seeing every variable lookup, argument push and jump.

Welcome to Wasp Lisp 0.2!

2007-11-10T02:06:00.001-08:00

Wasp Lisp 0.2 is ready for the public! The REPL is back in this version, and it seems to work well on my two test platforms, Linux/x86 and WinNT. As in 0.1, the network event loop is still missing, subsystems won't build automatically, although they DO load on both platforms, and Wasp still won't produce standalone executables. I know I said that 0.2 would have the network loop back, but people have been asking questions about Wasp again, so it is time to get something out the door. For 0.3, I want to include a preliminary version of the WaspDoc documentation tool so a new Wasp Lisp manual can be written with the assistance of the user community. (All three of you.. Well, if you build it, either they'll come or they'll still prefer DrScheme..) In 0.4, the event loop will return, and Wasp will probably enter a feature freeze. I expect to focus on unit tests, build processes and documentation. Wasp and Mosquito have never had a real production quality release -- something I would like to change. For Source and Win32 Builds, see the possibly temporary distribution site.

Version 0.1 Released! Kind of..

2007-06-22T20:30:00.000-07:00

To avoid becoming the next Debian project manager, I decided it is probably time to release version 0.1 of WaspVM. This version compiles and executes programs, but does not read input from the user correctly, does not do asynchronous I/O, does not build subsystems, and probably does not build on Windows. But, it does contain the new Channel and Connection semantics, and it will be the last version that does not provide integration with native threads. Version 0.2 will use pthreads and libevent to create a worker thread to manage I/O concurrent with the virtual machine. I will also add the ability to build subsystems, which has been omitted because I am still trying to get comfortable with autoconf. After 0.2, we will probably see Alpha One, and work will begin on the Wasp Documentation project. With the lessons learned from WaspDoc and Alpha One, the Beta process and march to 1.0 will begin. I'm setting a project deadline of roughly 2017 for 1.0.. Learned my lesson with that December target. :) http://code.launchpad.net/~swdunlop/waspvm/0.1

So I don't misplace it.. A BSD licensed readline clone..

2007-06-21T08:46:00.001-07:00

Perhaps calling it a clone is disingenuous; WaspVM, like MOSVM, is LGPL'd and incompatible with GNU's readline implementation. While I could probably do some horrible two-step around the license requirements, it is more honest to just use another library. http://www.cs.utah.edu/~bigler/code/libedit.html (For the curious legal minds -- since GNU Readline is GPL, it cannot be linked into our LGPL source code. But, a GPL stub application could be linked to our LGPL interpreter library and the GPL Readline, acting as some legal insulation. This kind of perversity keeps the lawyers of the world in business..)

Getting Closer..

2007-06-03T18:04:00.000-07:00

The thicket of bugs related to the I/O changes afflicting the Wasp REPL are starting to thin out; I cannot wait for some of the changes I have planned for debugging Wasp programs -- even I get tired of weaving through low level trace output. The current issue appears to be a deadlock that forms after the first line is received by the Wasp REPL. Deadlocks in Wasp cause the virtual machine to terminate without comment; I need to change this to emit more information about the current processes, and what they are waiting on. Still no idea what is going on over at EphSec.. Wish people would stop asking me. When I hit Alpha One, I'll start making Wasp more visible to the Lisp community to test the waters.

2007-05-11T12:28:00.000-07:00

It's May.. Like most deadlines, everything crept by.. The new job tends to leave me exhausted at the end of the day, so I have been slow in achieving my objective with WaspVM. It didn't help that I made the mistake of using Google Code, which is horribly raw and unready, for managing the public source tree. The new source tree may be found at the Ubuntu project's Launchpad; while this, too, is relatively raw, it doesn't force me to use Subversion, which automatically makes it a hell of a lot better than Google Code. (Bazaar is like a more readable, less self-imploding Darcs.) The build tree self compiles, but does not do any of the following:

Install. It's a bit early for installing this version, it's very immature.
REPL. I still need to adjust all of the REPL source files for the new Channel semantics.
TCP Connections. The I/O monitor is going to be moved into a subsystem.

Once all of these things work, I will cut an alpha one release, and start laying down the framework for a new documentation project. While I still have access to the old Mosquito reference data, it was never clearly licensed and its quality was somewhat uneven -- I had very little input in the documentation process, and was never satisfied with it. Which brings us to the next question: where in the world is Ephemeral Security? I can only guess that things have gone awry, and Wes Brown has decided to let the whole thing lapse into silence. Since we split on less than amicable terms, I would probably be the last to know..

Christmas Status Report

2006-12-25T17:00:00.000-08:00

Things are proceeding slower than I would have liked for Alpha One. The good news is that almost all of the new features have been written, but I have not had the time to test and integrate them into the code base. The most painful problem has been with Mac OS X, which has some interesting idiosyncrasies in its dynamic library loader that have made the dynamic subsystems much more complicated to implement. While I appreciate Apple's efforts at breaking new ground with OS X, I really wish they would either clearly document workarounds for their little innovations or keep them out of the POSIX / GNU / C level of their API's. (In specific, the loading program cannot export symbols to be loaded by dynamic libraries loaded at run time. An intermediate shared library that contains these definitions must be used, despite Apple's claims of 100% compatibility with other dlopen/dlsym implementations.) This has forced a near total rewrite of the build system, which, for 1.0, will probably break the ability of waspc to produce standalone binaries. This isn't a terrible thing, since I've wanted to move to autoconf for a while; we have wasted far too much time on our own build system and I would like things like Canadian Cross and VPATH builds. The Sqlite3 subsystem was trivial to write, taking about a night to write and test. I hope this is a good indication of how easy it will be to write third party subsystems for Wasp. Quick, informal tests show it to be somewhat faster than the Python module I have been using, which is reassuring. Work on the Streams and Channels reform has also been a bit of a struggle. I have a fairly clear idea of where I want Channels and Streams to be -- easy to expand, easier to debug, and better performing. And the C files in the current source tree reflect the new Channels fairly well. The miserable part, as usual, is integrating these clean, neat abstractions with the OS.

An' 31 Days Has December..

2006-12-01T06:07:00.000-08:00

The objective - present a first alpha version of the new Wasp Virtual Machine, fork of the Mosquito Virtual Machine I designed for Ephemeral Security, by the beginning of the New Year. For this first alpha, I want the new loadable subsystem API done, at least for UNIX platforms if not Windows, a complete Sqlite API, and an overhaul of Channels and Streams to make them more generic. If possible I also want to have a simple API for native threads to run concurrent with the virtual machine thread for "worker" functionality so we can make peace with the Win32 APIs.