Tuesday, September 26, 2006

Futures and Promises in Javascript

The Alice programming language has some nice concurrency features to facilitate dataflow programming. Futures and Promises allow performing computations and having threads block waiting for the results of those computations in an safe, predictable manner.

There have been discussions in the Narrative Javascript group about how to implement this sort of functionality on the browser client side. Given client side threads, it's fairly easy to implement. I'd done some work previously on implementing client side threads with Narrative Javascript but to try out these ideas I took a slightly different approach. Narrative Javascript changed the internals recently and I couldn't use the manually written CPS transformed functions like my previous threading implementation. This time I reimplemented things using the 'notifier' functionality that it has been replaced with.

For simple threading, instead of writing a scheduler I that has a queue of continuations to resume I just pass everything on to Javascript's 'setTimeout' function:
function Process() {
this.pid = pid_count++;
this.mailbox = new Queue();
this.blocked = false;
}

Process.prototype.resumeAfter = function(func, ms) {
var p = this;
setTimeout(function() { self = p; func(); self = false; }, ms)
}

Process.prototype.resume = function(func) {
this.resumeAfter(func, 10);
}

function concede() {
var n = NjsRuntime.createNotifier();
self.resume(n)
n.wait->()
}

function sleep(ms) {
var n = NjsRuntime.createNotifier();
self.resumeAfter(n, ms)
n.wait->()
}

function spawn(func) {
var n = NjsRuntime.createNotifier();
var p = new Process();
p.resume(function() { func(); n(); });
n.wait->()
return p;
}
A 'future' is just a spawned thread that updates an object when it is complete. A process can block waiting for the result of the future. For this quick example I poll the object to see if it has completed, and give up a timeslice using 'concede' if it hasn't. In an actual framework I'd rewrite this to not poll, but instead to be notified when it is complete - I just wanted to prove the concept first:
function future(func) {
var f = {}
f.finished = false
spawn->(function() { f.result = func->(); f.finished = true; })
return f;
}

function value(f) {
while(!f.finished) {
concede->();
}
return f.result;
}

An example of how this can be used using a 'fib' calculation:
function fib(n) {
concede->()
if(n==0) {
return 1;
}
else if(n==1) {
return 1;
}
else {
return fib->(n-1)+fib->(n-2);
}
}

function test() {
var f1 = future->(function() { return fib->(5); })
var f2 = future->(function() { return fib->(6); })
print(value->(f1) + value->(f2))
}
Two threads are spawned as futures and then the results are used and printed. 'print' is a simple function to display the results in a div on the html page. If the threads have not completed then the process blocks until they are and the results provided.

Promises are implemented in a very similar manner:
function promise() {
var p = {}
p.finished = false;
return p;
}

function fulfill(p, value) {
p.result = value;
p.finished = true;
}

function test2() {
var p = promise();
spawn->(function() { var v = value->(p); print("Thread 1 complete: " + v); });
spawn->(function() { var v = value->(p); print("Thread 2 complete: " + v); });
spawn->(function() { var v = value->(p); print("Thread 3 complete: " + v); });
fulfill(p, 42)
}
Here I spawn three processes waiting on a single promise variable. When that promise is fulfilled the three threads resume and display the values.

You might notice that the process object has a 'mailbox' - this is for Erlang style message passing which I've also implemented. Here's how the lightweight message passing code looks:
function test3() {
var p1 = spawn->(function() {
var m = receive->();
print(m[1]);
send->(m[0], "From p1!")
})

var p2 = spawn->(function() {
send->(p1, [ self, "From p2" ]);
print("p2 sent to p1");
var m = receive->();
print(m);
})
I think this implementation of threading using Narrative Javascript is a bit cleaner than my last implementation since it passes off most of the work to the browser's setTimeout function - and there's less code which is always good. It also uses the standard Narrative Javascript API rather than taking advantage of the internal structure of the generated code.

I'm rewriting the promise and future code not to use polling and tidying it up and I'll make the code and a live example available here over the next few days.

Categories: ,

More on Jwacs

Bill Clementson has a post summarising the lispvam meeting discussing jwacs. A great transcript of the presentation is also available.

Categories: ,

Tuesday, September 19, 2006

Lazy file reading in Factor

I've added a couple of new lazy list routines in Factor to do lazy stream input/output. These are lazy counterparts to the standard contents and lines words.

The new words are 'lcontents' and 'llines'. The first returns a lazy list of all the characters in the file. The second returns a lazy list of all lines in the file. Both of these lazily retrieve from the stream as needed and can be used on files larger than the memory available to Factor:
"test.txt" <file-reader> llines [ . ] leach


While adding these I noticed the the mapping and subset operations didn't memoize their values. This meant the quotation for these operations were being called whenever the 'car' or 'cdr' of the cons was called. I added a <memoized-cons> type that wraps an existing cons and remembers the previous calls to 'car', 'cdr' and 'nil?'. 'lmap' and 'lsubset' automatically wrap their return type in a <memoized-cons>.

Now that they are memoized, the following 'fib' implementation works very fast:
: fib ( n -- )
dup 2 < [
drop 1
] [
dup 1 - "fibs" get lnth swap 2 - "fibs" get lnth +
] if ;

naturals [ fib ] lmap "fibs" set

5 fib
25 fib
60 fib

This creates a lazy list of fibonacci numbers. Retrieving the nth item from the list returns the nth fibonacci number. As the values are automatically memoized as a result of the lazy map operation, susbequent calls to 'fib' are very fast.

Categories:

Sunday, September 17, 2006

Factor code to upload to the S1 MP3 Player

Futher to my post on using Factor to program the S1 MP3 Player, I got some basic code working to upload Z80 code and execute it.

It requires updates to the usb code in my Factor repository:
darcs get http://www.bluishcoder.co.nz/repos/factor

I've created a darcs repository to host the code. It's best to get the repository from within the Factor 'contrib' subdirectory so you can use the standard module system to load the code:
darcs get http://www.bluishcoder.co.nz/repos/s1mp3

libusb
must also be installed. Once all that's done you can test things out by plugging in the MP3 player and navigating to the 'Update Firmware' menu option. The MP3 player then sits waiting to receive code.

The following will load a Z80 assembled file called 'test.bin', upload it to the MP3 player and start executing it:
"contrib/s1mp3" require
USE: s1mp3
"test.bin" s1mp3-run-file


The Z80 file must be assembled with an origin of 0x3400. For a quick test I've included in the repository Carlos Aguilar's LCD autodetect program. The binary is in 'detect/detect.bin' and can e sent to the player with:
"contrib/s1mp3/detect/detect.bin" s1mp3-run-file

This should find details about the LCD display on the device and display it. Hopefully this code will work on all supported Factor platforms as long as libusb is available. This covers Windows, Linux and Mac OS X that I know of.

If you get errors due to 'permissions' it may be because your user does not have permission to access the USB device directly. On Linux this can be fixed by running Factor as root. A better option though is to set up hotplug or udev to change the permission on the device when it is plugged in so you can access it.

Categories: ,

Saturday, September 16, 2006

Writing your own MP3 player firmware

The S1 MP3 Player (or 's1mp3' as it is commonly known as) is a generic type of MP3 that is used and branded by a number of manufacturers. It uses a chip that is firmware updatable, where the firmware is written in Z80 machine code. A number of people have written third party tools allowing writing and running their own code on the MP3 player. The S1MP3 website and wiki has lots of details about this.

A program originally announced on the mailing list called loadram is used to upload Z80 code while the player sits on the 'upload new firmware' screen. This code is run, and can return back to the player. Not much seems to be known about the MP3 players yet but it is possible to write text to the LED screen on some models.

The manufacturers firmware updates can be 'unarchived', and they contain the Z80 binary programs that are run from the firmware menu. For example, the program to run the sound recorder is called RECORDER.AP. This can be replaced by your own Z80 program, repacked into the firmware and updated to the player. From then on, selecting 'Record' from the MP3 player menu will run your new program.

What interests me about this is it gives a small portable reprogrammable device. And hopefully something I can leverage Factor's interactive development system. I wrote an 8080 emulator to play the original Space Invaders arcade game written in Factor. 8080 and Z80 are quite similar. In fact I used Z80 assembler syntax in the 8080 emulator implementation.

It wouldn't be hard to base a Z80 emulator on top of this to allow testing programs to run on the MP3 player. And to write a Z80 assembler in Factor that generated the binaries to upload to the device. 'loadram' was Linux based and has been recently ported to Windows. It uses libusb to communicate with the player. By adding libusb wrappers to Factor it would be possible to assemble and upload directly to the player from a Factor environment - and it should work on Windows, Linux and Mac OS X.

I put together a quick wrapper around libusb and added it to my repository:
darcs get http://www.bluishcoder.co.nz/repos/factor

This has been tested to work on Linux and Mac OS X. I'll try adding a nice Factor wrapper around the alien definitions and get 'loadram' ported.

Note that to implement the alien wrappers I had to modify Factor's alien C structure wrappers to accept arrays of characters. The patch for this is in my repository but may not necessarily be pulled by Slava into the main Factor repository. I need to tidy it up and work on it before that might happen.

Once libusb is working and loadram is ported I'll try getting the Z80 emulator and disassembler going.

If you're looking for suitable players, Dick Smith Electronics in New Zealand, has at least two models that work. The first is their DSE branded 128MB MP3 player, model number A2439. This is a very basic model and is quite cheap. It has a black and white LCD screen which can be written too using the S1MP3 open source code.

Another model they have is the AStone Samba AV MP4 player. This has a colour OLED display and can play video and pictures - on its very tiny screen. Calling it an MP4 player is a bit of a stretch. It can't actually play MP4's. You use their client software to convert videos in other formats to an AMV format which has awful quality (not that it matters on the tiny screen) but otherwise works ok. The downside of this player is it seems that there is no known way of writing to the colour OLED displays yet.

What sort of things could be written to run on the MP3 players? I'm not sure yet but it'll be an interesting 'spare time' project to play with.

Categories: ,

Factor Lazy Lists Revisited

I originally wrote the Lazy Lists library in Factor to support the Parser Combinators library. It was one of my first libraries in Factor and was probably not very well written. Matthew Willis worked on it for the recent Factor releases to get it working with the removal of the standard list datatype and made it a lot better in places.

Both the lazy lists and parser combinators worked by building up quotations manually to defer computation, and having these quotations called when required. This unfortunately made the library a bit difficult to debug and understand. While using the parser combinators library for a recent project I found it quite hard to work some things out and found a couple of bugs in the parser combinators - and areas where the lazy lists weren't lazy enough.

To resolve the issues I rewrote the lazy lists library in a slightly different style and plan to redo the parser combinators in a similar manner. Instead of building up quotations I now use tuples to hold the information, with generic functions to call to process them.

I created a generic function based protocol for lists. It has three generic functions:
GENERIC: car  ( cons -- car )
GENERIC: cdr ( cons -- cdr )
GENERIC: nil? ( cons -- bool )


A 'cons' tuple for normal non-lazy lists, with the obvious implementation was the starting point:
TUPLE: cons car cdr ;
M: cons car ( cons -- car )
cons-car ;

M: cons cdr ( cons -- cdr )
cons-cdr ;

: nil ( -- cons )
T{ cons f f f } ;

M: cons nil? ( cons -- bool )
nil eq? ;

This gives the functionality of ordinary lists that used to exist in Factor. To make actual lazy lists I use the 'force' and <promise> words from Matthew's lazy list rewrite. A <promise> is a wrapper around a quotation that calls that quotation when 'force' is called on it, and memoizes the value to return directly on later 'force' calls. A lazy list has a promise for both the 'car' and 'cdr':
: lazy-cons ( car cdr -- promise ) 
>r <promise> r> <promise> <cons>
T{ promise f f t f } clone [ set-promise-value ] keep ;

M: promise car ( promise -- car )
force car force ;

M: promise cdr ( promise -- cdr )
force cdr force ;

M: promise nil? ( cons -- bool )
force nil? ;

Notice that the lazy list itself is a promise. And the methods are specialized on the <promise> type. So not only are the 'car' and 'cdr' promises, but so is the list itself. The 'cdr' of a lazy list is a promise, which when forced, returns something that conforms to the list protocol. This means parts of the list can be lazy, and parts non-lazy

A lazy map operation is the first word I implemented on top of this generic protocol. Previous implementations used quotations that were automatically generated and were quite complicated. In this new implementation I created a <lazy-map> tuple that implemented the list protocol which turned out to be much simpler:
TUPLE: lazy-map cons quot ;

: lmap ( list quot -- result )
over nil? [ 2drop nil ] [ <lazy-map> ] if ;

M: lazy-map car ( lazy-map -- car )
[ lazy-map-cons car ] keep
lazy-map-quot call ;

M: lazy-map cdr ( lazy-map -- cdr )
[ lazy-map-cons cdr ] keep
lazy-map-quot lmap ;

M: lazy-map nil? ( lazy-map -- bool )
lazy-map-cons nil? ;

Basically the 'lmap' call itself just returns a <lazy-map> which contains the quotation and list to map over. Calls to 'car' will call the quotation on the head of the list, while 'car' returns a new <lazy-map> that holds the same quotation and the 'cdr' of the original list.

The 'take' operation returns the first 'n' items in the list (whether lazy or not). It's lazy implementation is:
TUPLE: lazy-take n cons ;

: ltake ( n list -- result )
over zero? [ 2drop nil ] [ <lazy-take> ] if ;

M: lazy-take car ( lazy-take -- car )
lazy-take-cons car ;

M: lazy-take cdr ( lazy-take -- cdr )
[ lazy-take-n 1- ] keep
lazy-take-cons cdr ltake ;

M: lazy-take nil? ( lazy-take -- bool )
lazy-take-n zero? ;

Given a word 'naturals' to return an infinite list of natural numbers, the squares of the first 10 numbers can be returned as:
naturals [ dup * ] lmap 10 swap ltake

The result is a lazy list itself. No actual computation is done yet until 'car' or 'cdr' is called on the result.

More code implements 'subset' to filter out items from the lists, append lazy lists in a lazy manner, etc. The squares of all odd natural numbers are expressed as:
naturals [ odd? ] lsubset [ dup * ] lmap

It would be possible to add to this to implement the protocol for other things like lines in a file. This way you can lazily read through a large file line by line. Although this could be done in the previous lists code I think this approach makes things easier to read. As part of the refactoring I also wrote integrated Factor help for all the non-internal words.

I'm not sure how 'concatenative' this approach is or whether it's the right approach for 'stack' based languages but it seems to work well for Factor. I'll work on the parser combinators, fitting them into a similar style, and see how it works out with the current project.

Categories:

Tuesday, September 12, 2006

Javascript with Advanced Continuation Support

Bill Clementson has an announcement for the next Vancouver Lisp Users Group meeting. This one is about a web framework called jwacs, Javascript with Advanced Continuation Support.

It's an interesting library that seems to fit into a similar space to Narrative Javascript. It compiles a superset of Javascript, with support for continuations, into normal Javascript which is then run on the browser. Their demo page has a number of examples with source. The compiler from jwacs to Javascript is written in Common Lisp.

It looks to me like a number of people are starting to use the 'asynchronous operations written in a synchronous style' in Javascript on the browser, bringing the good things that you get with continuation based web servers to the client side.

Categories: ,

Wednesday, September 06, 2006

Serializable Gadgets

As Slava has noted, I've managed to get the serialization code to the point where it serializes user interface gadgets. To demonstrate it I serialized an in-progress game of Space Invaders, uploaded the serialized file to my web server and someone else on IRC downloaded it, deserialized it, and continued where the game left off.

Given the gadget on the stack, serialization to a file was as easy as:
[ 
"filename.ser" <file-writer> [
unparent serialize
] with-stream
] with-serialized

To get the instance running again:
[ 
"filename.ser" <file-writer> [
deserialize
] with-stream
] with-serialized "Space Invaders" open-titled-window


Categories:

Tuesday, September 05, 2006

Pattern Matching in Factor

For the concurrency library in Factor I've struggled to come up with good ways of writing the main loop of processes that handle messages. Erlang makes it so nice being able to create tuples and break them apart using pattern matching.

I've gone from using Factor arrays and breaking them apart with 'first', etc to using generic functions. The latter works well but ends up quite verbose for small servers. You need to create tuple types for the different messages - and handling tagged messages to reply to had to be done differently to other messages.

To make this sort of thing easier I've written a 'match' word for Factor that does pattern matching. I based it on code from Paul Graham's excellent Common Lisp book, On Lisp.

Given a Factor sequence or primitive type you can pattern match and create bindings to symbols with a special format. Any symbol that begins with a '?' character is used as a pattern match variable, and binds to the value in a matching sequence. Here's a simple example:
"match" require
USE: match

SYMBOL: ?a
SYMBOL: ?b
SYMBOL: ?c

{ 1 2 3 } { ?a ?b ?c } match .
=> H{ { ?a 1 } { ?b 2 } { ?c 3 } }

The two sequences match in that they are both sequences with three items. So the result is a hashtable containing the pattern match variables as keys and the values of those variables in the second sequence as the value. If items in the sequence are not pattern match variables then they must match exactly:
{ 1 2 3 } { 1 2 ?a } match .
=> H{ { ?a 3 } }

{ 1 2 3 } { 2 2 ?a } match .
=> f

The second example doesn't match as the first element in both sequences is not the vaue '2'.

Matching works recursively too:
{ 1 2 { 3 4 } { 5 6 } } { 1 2 ?a { ?b ?c } } match .
=> H{ { ?a { 3 4 } } { ?b 5 } { ?c 6 } }

This type of pattern matching is very useful for deconstructing messages sent to distributed factor objects. But even more useful is to be able to have something like 'cond' but working directly with patterns. This is what 'match-cond' does. It takes a sequence and an array of arrays. Each sub-array contains the pattern to match the sequence against, and a quotation to execute if that pattern matched. The quotation is executed with the hashtable result of the pattern match bound in the current namespace scope, allowing easy retrieval of the pattern match variables.

It sounds complex but is actually very easy in practice. Here what an example 'counter' process might look like using 'match-cond':
SYMBOL: ?value
SYMBOL: ?from
SYMBOL: ?tag

: counter ( value -- )
receive {
{ { increment ?value } [ ?value get + counter ] }
{ { decrement ?value } [ ?value get - counter ] }
{ { get ?from } [ dup ?from get send counter ] }
{ _ [ "Unmatched message" print counter ] }
} match-cond ;

This is a process that keeps track of a count value. You can send messages to increment or decrement the count by an amount, or to get the value and send it back to the calling process. So sends/receives look like:
[ 0 counter ] spawn
{ increment 5 } over send
{ decrement 10 } over send
[ get , self , ] { } make swap send receive .
=> -5

This works out to be quite readable and much nicer than my previous attempts. The match and match-cond code is still a work in progress. I plan to add the ability to pattern match on tuples, all the factor sequence types, and to repeat pattern match variables to ensure the same repitition occurs in the target sequence. For example:
T{ person f "chris" "double" } T{ person f ?first ?last } match .
=> H{ { ?first "chris" } { ?last "double" } }

{ "one" "two" "one" "two" } { ?a ?b ?a ?b } match .
=> H{ { ?a "one" } { ?b "two" } }

{ "1" "two" "one" "two" } { ?a ?b ?a ?b } match .
=> f

The match library is split out from concurrency since it's likely to be quite usable for other things in Factor. It's available from my repository (and is likely to be available shortly from the main Factor repository). To get the patches from my repository:
darcs get http://www.bluishcoder.co.nz/repos/factor


Categories:

Monday, September 04, 2006

Factor 0.84 is out