Tuesday, May 30, 2006

Narrative Javascript

Narrative Javascript extends the Javascript language to include a new operator. That operator allows writing procedural looking code that spans asynchronous operations.

Very much like continuations enable writing web applications such that you don't need to break application flow to model the HTTP protocol, Narrative Javascript enables you to write your code in a sequential style and have it automatically produce the callbacks and handlers for events, XMLHttpRequest, etc.

It consists of a compiler that compiles a Javascript plus the new operator into standard Javascript which can be included in the web page.

Narrative JavaScript consists of a compiler and a runtime library. You write your code code as Narrative JavaScript (using the blocking operator) and the compiler will parse and translate the code into regular (asynchronous) JavaScript. You then deploy the compiled code along with a runtime library. To aid in debugging, the compiler keeps line numbering of the source code intact in the compiled code.


An example from the website shows this code:
function startForm() {
displayPage1();
}

function onPage1Submit() {
var errors = validatePage1Input();
if(errors) {
displayPage1(errors);
} else {
displayPage2();
}
}

function onPage2Submit() {
var errors = validatePage2Input();
if(errors) {
displayPage2(errors);
return;
}

var url = buildUrl();
doHttpRequest(url, handleResponse);
}

function handleResponse(r) {
if (r.page1Errors) {
displayPage1(r.page1Errors);
} else if (r.page2Errors) {
displayPage2(r.page2Errors);
} else {
displaySuccess();
}
}

Written in Narrative Javascript as:
 function displayForm() {
var page1Errors = [];
var page2Errors = [];
var success = false;

while (!success) {
while (page1Errors) {
page1Errors = displayPage1->(page1Errors);
}

while (page2Errors) {
page2Errors = displayPage2->(page2Errors);
}

var url = buildUrl();
var response = doHttpRequest->(url);

page1Errors = r.page1Errors;
page2Errors = r.page2Errors;
success = !(page1Errors || page2Errors);
}

displaySuccess();
}

Notice the code in the following line:
var response = doHttpRequest->(url);

This does an XMLHttpRequest which usually requires having a callback function process the request. This is done automatically by Narrative Javascript. From the sounds of it it compiles the code into CPS form and uses the continuation as the callback. This enables the code to continue from where it left off. In this case returning the value of the XMLHttpRequest and continuing the function.
Any operation that requires a callback function is a candidate for blocking. Please note that blocking does not mean that the entire JavaScript runtime locks up waiting for an operation to finish. Instead, Narrative JavaScript makes use of co-operative multitasking. When you use the blocking operator, you are yielding control to the JavaScript runtime to execute other operations until your operation has yielded a return value.

This system looks very nice and I look forward to playing with it.

Categories:

Monday, May 29, 2006

Amazon S3

I've been playing around with Amazon S3 for a project I'm working on and I'm quite impressed with it. Amazon S3 provides unlimited storage for $0.15 per gigabyte storage and $0.20 per gigabyte of bandwidth.

Once you sign up and get an account you can create 'buckets' which are named collections of stored objects. Each bucket acts as a billing point so it provides a way to seperate data stored for different applications so you can track bandwidth, storage utilisation and cost, etc.

Within a bucket you can store and retrieve objects by key. There is no means to update or append an object so it's either create a new one or replace an existing one completely. You can use 'partial get' in HTTP to retrieve partial contents of an object though.

The API to manage the S3 datastore is a REST based HTTP API or using SOAP. Amazon have provided a number of sample client libraries in different languages to get you going fast. This includes Java, Ruby, Perl and Python.

The objects stored can be made publically accessable via an HTTP URL, or private, allowing only the owner to access them. Every object stored has automatic bitorrent support. By appending a torrent suffix onto the URL for the object you get a bitorrent file that allows using the capabilities of the bitorrent protocol to share the bandwidth of downloading large files.

People have already started building interesting libraries on top of it. For example:
  • S3Ajax. This is a Javascript library that lets you call the S3 API from within the web browser. By hosting the Javascript files on your S3 system and providing access to them to publically via the browser you get a simple web site. As it is hosted on the S3 domain the Javascript functions can call the S3 API providing read/write access to the storage. S3Wiki is uses this for example.
  • This thread in the S3 forums discusses a filesystem built on top of S3. It works under Linux and can be mounted like a normal drive. Each block in the filesystem is stored as an object in the S3 datastore. It can only be mounted on one system at a time but once unmounted it can be mounted safely on another machine. Providing unlimited storage that can be accessed from any Linux system.
  • JungleDisk has a similar idea but uses a local WebDAV server and transparently copies and encrypts data to the S3 datastore. It can be used in Windows, Linux or Mac OS X. The main difference between JungleDrive and the S3 filesystem above is that JungleDrive is cross platform. It also stores files into S3 objects on a one-to-one mapping I believe, whereas the S3 filesystem stores the blocks that make up the files into an object. The latter approach makes it easier to support partial access to files and streaming without having to download the entire object first. At the risk of making it harder to reconstruct the file itself if things go wrong I guess.

I'm using S3 for holding uploads of media provided by users for later analysis and classification. I don't know how much data the system will eventually collect so having an unlimited (except by cost) storage capability is useful. It also means I don't have to pay for a storage immediately. Instead I can pay as I go.

So how is S3 different from something like Openomy? My undrstanding is the main intent for Openomy is to provide a place for users to store the data from web applications and the user owns that data. So an Openonmy compliant web application would have access granted to an area of the users Openomy storage and can read/write to it. Should the web application go out of business or become inaccessible the user still has the data in their control.

With S3 the usage model is that the web application uses their own S3 store for storing data. The user does not get S3 storage and provide it to the web application. To do that would require giving up their 'secret key' which can't be revoked. This would be bad. The web application could access all of the users data. With Openomy you can authorise and prevent an application from accessing the data. Both usage models are useful and I think they are complementary services.

Currently I'm using S3 from Javascript on the server. It's very easy to call the S3 Java API from Rhino. To import the basic Java classes into Rhino and create an authenticated connection:

importClass(Packages.com.amazon.s3.AWSAuthConnection);
importClass(Packages.com.amazon.s3.S3Object);

var conn = new AWSAuthConnection(accessKeyId, secretAccessKey);

The 'accessKeyId' and 'secretAccessKey' are the keys supplied by Amazon once you've subscribed to the S3 web service. Once you have a connection you can create buckets and store and retrieve objects:

js> conn.createBucket('mybucket1', null).connection.getResponseMessage()
OK
js> conn.listBucket("mybucket1", null, null, null, null).entries
[]
js> var obj1 = new S3Object(new java.lang.String("Hello!").getBytes(), null);
js> conn.put('mybucket1', 'key1', obj1, null).connection.getResponseMessage();
OK
js> conn.listBucket("mybucket1", null, null, null, null).entries
[key1]
js> new java.lang.String(conn.get('mybucket1', 'key1', null).object.data);
Hello!

Wrapping nicer server side Javascript API would probably be a good idea. For example I can't call the 'delete' method of AWSAuthConnection as 'delete' is a reseved word in Javascript. As Rhino allows serialising any Javascript object, you could even store continuations in S3.

Categories: ,

Saturday, May 27, 2006

Seaside for Dolphin Smalltalk

A preliminary port of Seaside, the continuation based web server framework, for Dolphin Smalltalk has been released.

Categories: ,

Thursday, May 25, 2006

Python to Javascript

Simon posted a comment on my post about the C# to Javascript compiler pointer out that there is a Python to Javascript system.

So far I've come across converters to Javascript from the following languages:

Even the JSScheme system might count, as the JIT essentially compiles Scheme into Javascript. I've been able to run this from Rhino to compile the Scheme code to Javascript and run it in the browser. All it would need is some nice libraries to wrap the DOM and use AJAX.

Categories: ,

Script#: C# to Javascript compiler

Nikhil Kothari has a post on a C# to Javascript compiler. Similar in approach to Google's Web Toolkit it allows you to program your client side web code in C# and compile it to Javascript. From this:
sing System;
using ScriptFX;
using ScriptFX.UI;

namespace HelloWorld {

public class HelloWorldScriptlet : IScriptlet {

private Button _okButton;
private TextBox _nameTextBox;
private Label _helloLabel;

private XMLHttpRequest _request;

public void Start() {
_okButton = new Button(Document.GetElementById("okButton"));
_nameTextBox = new TextBox(Document.GetElementById("nameTextBox"));
_helloLabel = new Label(Document.GetElementById("helloLabel"));

_okButton.Click += new EventHandler(OnOKButtonClick);
}

private void OnOKButtonClick(object sender, EventArgs e) {
Callback completedCallback = new Callback(this.OnRequestComplete);

_request = new XMLHttpRequest();
_request.Onreadystatechange = Delegate.Unwrap(completedCallback);
_request.Open("GET", "Hello.axd?name=" + _nameTextBox.Text, /* async */ true);
_request.Send(null);
}

private void OnRequestComplete() {
if (_request.ReadyState == 4) {
_request.Onreadystatechange = null;

string greeting = _request.ResponseText;
_helloLabel.Text = greeting;
}
}
}
}
It generates:
HelloWorld.HelloWorldScriptlet = function Scenarios_HelloWorldScriptlet() {
}
HelloWorld.HelloWorldScriptlet.prototype = {
_okButton: null,
_nameTextBox: null,
_helloLabel: null,
_request: null,

start: function Scenarios_HelloWorldScriptlet$start() {
this._okButton = new ScriptFX.UI.Button(document.getElementById('okButton'));
this._nameTextBox = new ScriptFX.UI.TextBox(document.getElementById('nameTextBox'));
this._helloLabel = new ScriptFX.UI.Label(document.getElementById('helloLabel'));
this._okButton.add_click(new Delegate(this, this._onOKButtonClick));
},

_onOKButtonClick: function Scenarios_HelloWorldScriptlet$_onOKButtonClick(sender, e) {
var completedCallback = new Delegate(this, this._onRequestComplete);
this._request = new XMLHttpRequest();
this._request.onreadystatechange = Delegate.unwrap(completedCallback);
this._request.open('GET', 'Hello.axd?name=' + this._nameTextBox.get_text(), true);
this._request.send(null);
},

_onRequestComplete: function Scenarios_HelloWorldScriptlet$_onRequestComplete() {
if (this._request.readyState == 4) {
this._request.onreadystatechange = null;
var greeting = this._request.responseText;
this._helloLabel.set_text(greeting);
}
}
}

HelloWorld.HelloWorldScriptlet.registerClass('HelloWorld.HelloWorldScriptlet', null, ScriptFX.IScriptlet);

Generating Javascript from other languages seems to be getting popular. Another system that does this is Morfik. They have some compelling demos and appear to compile C#, Pascal, Basic and Java into Javascript.

Categories: ,

Wednesday, May 24, 2006

Back in Wellington

I arrived back in Wellington this morning. Melbourne's weather was just taking a turn for the worse so it looked like a good time to leave. That and I'd managed to complete enough of the project I was working on not to be there full time. Wellington's weather is not looking much better though! Hopefully I can now churn through some of the projects I'd parked while away.

Categories:

Extending prototypes of built in Javascript objects

Web Graphics has a good post outlining the reasons why it is not a good idea to extend the prototype of Object and Array. Tracking down bugs caused by breaking 'for....in' is a pain...

Categories:

Icing: A Scheme continuation based web framework

Tony from LShift has announced an interesting looking project they've put together called Icing. They put SISCWeb, SSAX and SXPath together, along with their own code, to create a web development system in Scheme.

It seems to cover generation of the user interface, accessing the database as well as using the continuation based SISCWeb framework for application flow. There is a presentation here.

Categories: , ,

Monday, May 22, 2006

Squeak Morphic in the Browser

Diego Gómez Deck is experimenting with Seaside and Comet to display Squeak Morphic graphics in a web browser. From his Seaside mailing list announcement:
I implemented a "Fake" canvas on Squeak side (I mean an Squeak-canvas) that generated JavaScript to push (using Comet) to browsers. The speed is not so bad, and we still have room for more optimizations (reduce the size of the generated JS, compress the socket usint GZip-http, etc).


Categories: ,

Linspire to use Haskell

According to the Haskell Weekly News, the OS team at Linspire are standardising on on Haskell as their preferred language for core OS development. Linspire is a Linux distribution.

The announcement from the Linspire team is here.

Categories:

Continuation based Web Servers

There's a lot of discussion going around at the moment about continuation based web servers. Some of the comments I've seen seem to be based on a misunderstanding of what exactly this type of server provides.

Ian Griffiths has a post about why he thinks continuation based web fraemworks are a bad idea.

Ian writes about 'Abandoned Sessions':
This is very much not analogous to the function returning or throwing an exception. In the world of our chosen abstraction - that of sequential execution of a method - it looks like our thread has hung.

The problem with this is that a lot of the techniques we have learned for resource management stop working. Resource cleanup code may never execute because the function is abandoned mid-flow.

The problem of a user exiting an interaction in 'mid-flow' is not unique to continuation based web servers. If the user is stepping through a multi-form shopping cart checkout process then the data they have entered must be stored someone.

If it's in the database or web session then this data will live for a specified time (usually the session timeout) and if the user doesn't continue the flow then it is deleted.

The same logic occurs in continuation based servers. The continuation, if stored on the server, holds the data that the user has entered during the flow. After a set timeout the continuation is removed. In some systems the continuation data is stored in the web session so this is done automatically.

Like any other web framework the continuation data can be stored in a form field on the clients browser (assuming the framework allows serialisable continuations) so there is no need for this 'garbage collection' to occur.

Ian is right that it is important not to acquire a resource and clean it up after a continuation capture boundary. If the user never returns to continue the flow then it may not be cleaned up....unless your language with continuations also has a 'dynamic-wind' construct. This is like try/finally in Java except you can have code run when a block is entered or exited for any reason. So when a block is exited due to a continuation escape then the resource is automatically released. When it is entered again it is automatically acquired. This approach removes the worry about hanging resources.

It's important to remember that the 'thread' running the web request doesn't hang when the continuation is captured. After capturing it the thread is gracefully exited in the normal manner while the web page data is returned to the user.

Ian continues with Thread Affinity:
With an ordinary sequentially executing function, I can safely assume one thread will run the function from start to finish. But if I’m using continuations to provide the illusion that I’ve got sequential execution spanning multiple user interactions, then I might get a thread switch every time I generate a web page.

You'll get a thread switch every time the user makes a request in a standard framework that supplies threads from a thread pool. I've not had a situation where switching threads has been a problem. Given the 'dynamic-wind' feature mentioned before, anything that requries thread affinity can be released and re-acquired in the new thread when the continuation is resumed.

The same goes for the 'Web Farms' issue. In a system where the continuation can be serialised or sent to another machine then another machine on a web farm can deserialise the continuation and continue. If this is not the case then session affinity can be used to ensure that a request in the same session is processed by the same machine.

Ian has a number of issues with back button and branching. The nice thing about continuation based frameworks is that this is all handled for you.

If the user hits the back button then they go back to a previous continuation. A continuation is a snapshot of a stack frame so has access to a copy of all local variables at the time that it was captured. This means that the back button 'just works'. Ian writes:
Normal functions don’t do that - they only jump back to earlier points if you use flow control constructs such as loops. Giving the user the option to inject goto statements at will is an unusual design choice, but anything that models user journeys as sequential execution of code will have to cope with this kind of rewinding, or it’ll break the back button. And I don’t know about you, but I hate sites that break the back button. (Yes, Windows Live Search, I’m looking at you.)

I completely agree that breaking the back button is a bad thing. With a continuation based framework you get a working back button for free because the execution stack is wound back to the point of the continuation for the page the user went back to.

You can choose not to have state wound back by most systems by storing data in the database, or keeping them globally in memory. Sometimes you don't want the user to go back. For example, if they've just processed a credit card you don't want them to go back and resubmit the form.

The way to do this is to have a way of marking a block as 'run-once'. If continuations captured within that block are attempted to run again then an error occurs, either displaying a message (don't submit twice) or going back to a known safe point. The framework handles this for you. This is easy to implement because it's as simple as checking at the beginning of the continuation entry if we've been there before by reading a flag.

Session cloning or 'bifurcating' as Ian calls it is handled just as easily. By opening a tab or new window on the existing page you get a resumption of the continuation the user is already looking at. Since these are copies of the execution stack the user effectively gets a copy of all the local variables. Modiying these has no effect on the information on the original page. So there is no need for the programmer to guard against concurrency here as Ian seems to think:
This means I have to write my function in such a way that it can cope not only being rewound, but also to being split so that multiple threads execute the function simultaneously, each taking different paths. But of course because I’m using continuations, each of these threads gets to use the same set of local variables. The fact that I enabled users to inject gotos into my code at will is now looking like a walk in the park - now they can add arbitrary concurrency!

You can choose to have state that does not get cloned of course. Just store it in the database (for example, the shopping cart).

Somethings you want to be 'global' and unaffected by the user using the back button, bookmarks and cloning. These can be stored in the database or some other persistent store. The shopping cart is the obvious example. You wouldn't want the user hitting 'back' and losing the last item put in the shopping cart. Or cloning the session to result in items not being added. Other things you may want wound back (Some types of data entered in a form, etc). These can be local variables. Continuation based servers give you the choice.

I highly recommend reading some of the papers mentioned in a previous posting of mine. They describe in much more detail some of the advantages (and disadvantages) of using continuations to model web flow.

Categories: ,

Sunday, May 21, 2006

Avi follows up on the Continuation debate

Avi Bryant has a long post following up on the recent discussions about continuations and continuation based web applications.

Categories:

Continuations with Mono

Tomi Valkeinen has implemented continuations on Mono, the open source implementation of the .NET VM.

On top of that Tomi built light weight threads in C#, semaphores, sockets and channels.

Categories:

Saturday, May 20, 2006

Rhino in Spring

I wrote previously about an option to use Javascript on the server using Rhino. I came across 'Rhino in Spring' today which does this in a much more full featured manner.

It is a continuation based system that integrates with Spring allowing you to use Javascript to write web applications and use continuation based features. From the looks of it you can choose where the continuation data is stored too, either in the database, in the session or in the client browser - no state stored on the server! Frmo the site:
At the moment, continuations can be stored in-memory, bound to a HTTP session, or for more sophisticated applications, they can be persisted to a JDBC data source or even let to be managed by the client's browser! Smart stubbing of shared objects is employed that both minimizes the size of the serialized state and allows continuations to be resumed in a different JVM than the one that originally created them, allowing for clustering and failover scenarios.

Categories: ,

Seaside for Dolphin Smalltalk Coming Soon

According to Avi Bryant on the Seaside mailing list, a 'useful version of Seaside' will be available on Dolphin Smalltalk by the 25th of May.
I haven't forgotten, I'm just slightly behind schedule (so what else is new... sigh). But there's nothing like a public commitment to make things happen, so: I promise that I will release some useful version of Seaside on Dolphin by this time next week - on or before Thursday, May 25.

Categories:

No Continuations for the JVM

Gilad Bracha writes that the JVM shouldn't get continuation support and gives some thoughts to continuation based web servers as well.
Summary: In the short term, continuation based web servers are very nice. Seaside is the nicest one I’ve seen. But ultimately they are just a phase, and we can already see how we will outgrow that phase. Since continuation-based servers won’t be all that significant in the long term, and given the huge lead times and costs of adding continuations to the JVM, it makes little sense to support them.

In a recent post by Avi, author of Seaside, to the Seaside mailing list, he comments that DabbleDB (which is written in Seaside) actually makes very little use of the continuation based elements of the framework:
As a data point, it's certainly interesting to note that Dabble DB
makes very little use of call/answer - the only major uses are when importing and when processing payments. We were intentionally going for a very modeless UI style, so this shouldn't be too surprising.
The question is whether this kind of modeless web app will become more common... it seems like it probably will. If so, then yes, the continuation aspects of Seaside will become less interesting, and the callback aspects will become more interesting...

I've found similar in web applications I've been writing lately. Less use of the return from the continuation but making heavy use of the ability to register functions to be called on button clicks, etc (the callback aspect). But when you need the use of the continuation returning for workflow it's very very useful to have it.

Categories: ,

Friday, May 19, 2006

Associative Arrays and Javascript

Andrew Dupont has a post about Associative Arrays in Javascript. He makes the good point that you shouldn't use the Array object for this functionality, instead use 'Object'.

Unfortunately using 'Object' has its own set of pitfalls. Try the following code in your favourite Javascript implementation:
var x = {}
x["toString"]

This will display details on the toString method on 'Object'. So we can't use a plan Object as an associative array as it has some default entries in it already obtained from its prototype.

This can be worked around by using the Javascript standard function 'hasOwnProperty'. So you could write a simple Hashtable object with a 'get' method that does:
function Hashtable() { }
Hashtable.prototype.get = function(key) {
if(this.hasOwnProperty(key))
return this[key];
else
throw "No such key";
}

Now the following code works:
var x = new Hashtable();
x.get("toString");
=> uncaught exception: No such key
x["blah"] = 5;
x.get("blah");
=> 5

It has a subtle bug too though. This breaks:
x["get"] = 5;
x.get("get");
=> TypeError: x.get is not a function

Oops, we overwrote our 'get' method. So the Hashtable object really needs to store its data in an internal object:
function Hashtable() { this.table = new Object }
Hashtable.prototype.get = function(key) {
if(this.table.hasOwnProperty(key))
return this.table[key];
else
throw "No such key";
}
Hashtable.prototype.put = function(key, value) {
this.table[key] = value;
}

The following code now works as expected:
var x = new Hashtable();
x.put("get", 5);
x.get("get");
=> 5
x.get("toString");
=> uncaught exception: No such key

The trap of using a basic Object as an associative array is seductive but as you can see it has pitfalls. They are easy to work around but it's easy to forget. Dojo's current implementation of Dictionary (as of 0.3) has this problem for example. There's a thread on the mailing list at the moment about fixing it (raised by me).

Categories:

Server side Javascript

[ An update to the code demonstrating E4X support is here ]

Ajaxian is reporting that Sun is releasing Phobos, a Javascript application server. This is a web server that runs Javascript for implementing the server side code. The source has not yet been released, according to Ajaxian, but will be.

A while back I worked on doing server side Javascripting using Rhino as the interpreter and Jetty 6 as the server. I got it to work but never tidied it up for release. Sun's Phobos has motivated me to make it available so I've tidied it up and it can be downloaded from javascript-server.tar.gz. A darcs repository with the code is here:

darcs get http://www.bluishcoder.co.nz/repos/javascript-server

The repository contains the Jetty 6 JAR files and the Rhino interpreter JAR, along with an example Javascript file showing how it works and a readme.

Once the 'example.js' is loaded into the Rhino interpreter you can start a Jetty 6 web server on port 8080 with the following command:
var s = startServer(8080);

A simple 'HelloWorld' style servlet looks like this in Javascript:
HelloWorldServlet = makeServlet({ 
ProcessGet: function (req, resp) {
var text = "Hello World!";
resp.setContentType("text/plain")
resp.setContentLength(text.length)
resp.getOutputStream().print(text)
resp.flushBuffer()
}
});

It has a 'ProcessGet' function which is called when an HTTP GET is made. The 'req' and 'resp' objects and the standard Java HttpServletRequest and HttpServletResponse respectively.

The magic of deriving from HttpServlet is done by Rhino. But it can't actually override HttpServlet methods - it can only override abstract methods. To fix this I created a JavascriptServlet Java class which forwards doGet(...) to an abstract 'ProcessGet', which is the method you see overridden above.

With the Jetty server we started running from the interpreter earlier we can add this servlet dynamically:
addServlet(s, "/", HelloWorldServlet);

Now requests to http://localhost:8080/ will run HelloWorldServlet. When can dynamically add other servlets too:
addServlet(s, "/bye", GoodbyeWorldServlet);

Requests to http://localhost:8080/bye/ will run the GoodbyeWorldServlet.

Using the Jetty API you can work out how to add, remove and otherwise do some very cool stuff. Like play with Jetty 6's Ajax continuation support.

The server can be stopped with:
s.stop();

Running Javascript on the client and server gives some interesting possibilities. Sharing code for example. Or writing validation rules in Javascript which run both on the client and server.

The Dojo javascript toolkit can run client side or server side. This means you could use Dojo's packaging and other nice features for server side development in Javascript.

Rhino has continuation's which are serialisable. A continuation based web server could be implemented which serialises continuations like SISCWeb.

Download javascript-server, play with it, and let me know what you do with it. I hope to follow up on some of the above ideas too.

Categories: ,

Wednesday, May 17, 2006

Google Web Toolkit

This is pretty nifty. Google have released 'Google Web Toolkit'. It's a system for writing ajax applications in Java.

The Java code is compiled into Javascript which then runs in the browser. From the Google site:
GWT applications can be run in two modes:
  • Hosted mode - In hosted mode, your application is run as Java bytecode within the Java Virtual Machine (JVM). You will typically spend most of your development time in hosted mode because running in the JVM means you can take advantage of Java's debugging facilities and remain within an IDE like Eclipse.
  • Web mode - In web mode, your application is run as pure JavaScript and HTML, compiled from your original Java source code with the GWT Java-to-JavaScript compiler. When you deploy your GWT applications to production, you deploy this JavaScript and HTML to your web servers, so end users will only see the web mode version of your application.

Categories:

Stuck in Melbourne

I managed to survive the 20 hours of flights and airports to get from San Francisco, to Auckland, to Melbourne. There's a noticable temperature difference when you go from summer in the US to winter in Auckland and winter in Melbourne! A bit disconcerting when I only packed expecting a summer environment.

Worse, it seems I'm stuck here in Melbourne for awhile. I was booked to fly out tomorrow night but just got a call from my employer telling me they've rearranged my flights and I'll be here longer...whether I like it or not. Is it possible to be kidnapped by the people you work for?

Categories:

Monday, May 15, 2006

Dojo and Spidermonkey

I attended the sessions on Dojo at the Ajax Experience conference to learn a bit about it.

Previously I'd considered Dojo to be a large, hard to understand library but I came away quite impressed with it after attending the seminars related to it. You can use the included package mechanism to make the base package fairly small and have components automatically loaded when required. One other neat factor is that Dojo can be used in a server side or command line Javascript environment.

I used Spidermonkey, the Mozilla C based implementation of Javascript, to try it out. I downloaded a nightly tarball of Dojo to get the source so I could set up a customised Dojo build for Spidermonkey.

Dojo has a 'profile' system that allows you to create a customised 'dojo.js' which contains Dojo packages that you want preloaded. Anything not preloaded will be loaded automatically when required by the package system. One file that is needed by Dojo is 'hostenv' which needs to be tailored for a specific Javascript implementation. There is a hostenv_browser for web browsers, hotenv_spidermonkey for spidermonkey, hostenv_rhino, etc.

To set up a customised dojo.js, which does nothing but makes it specific for Spidermonkey, I created a 'buildscripts/profiles/spidermonkey.profile.js' file with the following contents:
var hostenvType="spidermonkey";
var dependencies=[];
load("getDependencyList.js");

This is exactly the same as the 'minimal.profile.js' but with the hostenvType set to 'spidermonkey'. Once this is done a customised dojo.js system for Spidermonkey can be created with the following Ant command:
cd buildscripts
ant -Dprofile=spidermonkey -Ddocless=true release intern-strings

The custom Dojo build for Spidermonkey is created at 'release/dojo'. You can test it with Spidermonkey with something like (user input in red):
cd release/dojo
js
js> load("dojo.js");
js> dojo.version
0.2.2+ (3802)

Many of Dojo's libraries are usable from Spidermonkey, including the package system and events. Here's an example demonstrating a usage of Dojo events from Spidermonkey:
cd release/dojo
js
js> load("dojo.js");
js> dojo.require("dojo.event.*");
js> var scheme = {}
js> scheme.jit = false
false
js> scheme.setJit = function(value) { this.jit = value }
js> var monitor = {}
js> monitor.watchJit = function() { print("Jit: " + scheme.jit) }
js> dojo.event.connect(scheme,"setJit",monitor,"watchJit")
[object Object]
js> scheme.setJit(true)
Jit: true
js> scheme.setJit(false)
Jit: false

The intent of this is to test decoupling some of the 'Scheme in Javascript' code from the browser DOM so it can be run from the command line. I don't want the code to explicitly check DOM objects so instead it keeps internal variables and browser specific code can watch these for changes to set checkboxes on the page.

In this case I use 'dojo.event.connect' so that when 'setJit' is called then 'watchJit' is called afterwards. As can be seen from above this works.

Categories: ,

Friday, May 12, 2006

Melbourne Bound

The Ajax Experience conference has been very interesting so far. Lots of great talks and technologies. I'll post more on that later when I get some free moments.

The plan after that was for me to fly back to NZ on Saturday night, arriving in Auckland on Monday morning, then from there back to Wellington.

Unfortunately I now need to go to Melbourne for work reasons so I get to arrive in Auckland after a 12 hour flight, hop on a 3 hour flight to Melbourne almost straight away, arriving there Monday morning and jumping straight into work. After 15 hours of flying I'm not sure how productive I'm going to be! Hopefully I'll only be there for a week at most and I can finally get home and play around with some of the new technologies I've been hearing about at the conference. And finish off some of my own projects.

Categories:

Saturday, May 06, 2006

San Francisco Bound

I'm flying out to San Francisco tomorrow to stay a week. I arrive on the 7th of May and leave on the 13th and will be at the Ajax Experience conference while I'm there. It's my first time in the US and I'm looking forward to it!

Categories:

Why the Scheme->Javascript JIT doesn't work on Firefox

After doing some digging I've found a bug in the Scheme-in-Javascript implementation that seems to be the reason the JIT doesn't work on Firefox.

The system uses Javascript objects as associate arrays (hashtables), mapping symbols names to definitions. For example:
TopEnv['eval'] = function(list) { return doEval(list.car); }

There is a problem with using Javascript objects for this type of thing and that is that "TopEnv['eval']" is really syntactic sugar for 'TopEnv.eval' in Javascript. In firefox this actually returns the 'eval' standard function. You can see this by evaluating the following in a Javascript shell in Firefox:
var x = new Object()
x["eval"]
=> function eval() { [native code] }

In Internet Explorer this returns nothing. The JIT process does a lookup on 'eval' at some point and on Firefox it returns the native 'eval' function which breaks the JIT process. On IO it returns the Scheme 'eval' function.

The fix is relatively easy but finding all uses of this type of lookup in the code is time consuming. But JIT will work once I've done this.

Categories: ,

Thursday, May 04, 2006

Port Seaside to VAST?

Posted on the Seaside mailing list is a request for people to port Seaside, the Smalltalk continuation based web system, to VAST - a Smalltalk system. The offer includes the following incentive:
To make it worth your while, we are offering a reward of a full license to VAST and all of our add-on products (a $9,595 value) to the first *five* people who successfully and independently port Seaside to VAST (we want to make sure that anyone who gets it working is rewarded, not just the first to finish).


Categories: ,

Lightweight threads in the browser

I've made some updates to the Scheme in Javascript library. You can run it at the jsscheme page.

I've split out the example scheme source and made it loadable via Ajax. To do this I use jQuery, a very nice Javascript library. I added a Scheme wrapper around the Ajax GET call so you can do code like this:
(ajax-get "example.scm" (lambda (result) (alert result)))

The more interesting change is a simple concurrency library giving lightweight threads that run in the browser. The API loosely follows that of Termite, a concurrent Scheme. It uses '!' for sending a message to a process and '?' to receive.

To use the library click on the 'currency' button. This will load the source via Ajax. Then press 'eval'. Once loaded you can start using it. There are some examples in the concurrency.scm file.

For example, a 'logger' process:
(define logger
(spawn
(lambda ()
(let loop ((msg (?)))
(if (string? msg)
(begin
(process-msg msg)
(loop (?)))))
(process-msg "Logger exiting"))))

This loops waiting for a message in the process queue. When it appears it displays it and then waits for more messages. Any non-string message quits the process. An example of using it:
(define (logger-test)
(spawn
(lambda ()
(! logger "Hello")
(! logger "World")
(! logger 'quit))))

I've also done an example of concurrency using Ajax. An 'ajax-process' waits for requests of the form '(FROM URL). When received it will retrieve the URL using Ajax and send the result to the FROM process:
(define ajax-process
(spawn
(lambda ()
(let loop ((msg (?)))
(let* ((from (car msg))
(url (cadr msg)))
(ajax-get url (lambda (result) (! from result))))
(loop (?))))))

Using it is simple:
(alert (!? ajax-process `(,current-process "example.scm")))

This uses '!?' which sends a message and immediately waits for the result. You don't have to send to the current process. The next example instructs the Ajax process to send it to the log window:
(define (ajax-test2)
(spawn
(lambda ()
(! ajax-process `(,logger "example.scm"))
(! ajax-process `(,logger "concurrency.scm")))))

All this runs within the browser. Unfortunately it's not fast as it's interpreted. The JIT compiler does not work on Firefox and even if it did, it does not work with continuations which the concurrency system is based on. I'll look into improving this over time.

Categories: ,

Wednesday, May 03, 2006

Source for Unify Available

The source for the system presented in the paper 'A Language Based Approach to Unifying Events and Threads' is now available.

Download information is available at the Unify home page.
The source code package can be browsed online here. It contains the application-level thread library, a simple web server and a HTTP load generator. For more information, see the READ_ME file.

To compile and run the source code, you will need the Linux 2.6 kernel, the most recent development snapshot of GHC 6.5, and the Linux asynchronous I/O library (libaio).
(From Lambda the Ultimate).

Categories:

Scheme Implementation in Javascript

A post on Lambda the Ultimate recently was about an Oberon interpreter written in Javascript, allowing Oberon to be embedded in the browser.

This reminded me of an near-R5RS compatible Scheme written in Javascript called jsScheme. It's a very nice implementation to play with but is IE 6 only. I've done some hacking at it and made it compatible with Firefox. I've put the modified version here.

The modified version has some changes to cater for Firefox and some things still don't work. JIT compilation for example, so I've left the checkbox for this disabled by default. This does slow things down quite a bit but I'll enable it again when I find the problem. For IE JIT works fine and you can enable it. If you find any other issues I'd be keen to get feedback. Leave a comment here or email me. I've also put the modified version in a darcs repository available with:
darcs get http://www.bluishcoder.co.nz/repos/javascript-scheme

Please send me any patches you'd like to apply.

What's nice about this system is it has most of R5RS scheme implemented, including tail recursion and first class continuations. A simple FFI for calling into Javascript is also available. For example, here's how to do some Javascript Date manipulation:
(define d  (js-eval "new Date();"))
(display (js-invoke d "toString"))
(newline)

'js-eval' will evaluate Javascript code and return the object. 'js-invoke' will invoke methods on that object. There is also 'js-call' which can call Javascript functions. Unfortunately Firefox had problem using 'js-call' on things like 'alert' and 'setTimeout' so I wrapped these built-ins with Javascript functions which can be called:
(js-call (js-eval "jsAlert") "Hello World!")

'js-call' will a Javascript function object. The 'js-eval' was required to return that function object given the function name.

Lambda the Ultimate also pointed to a Links paper recently. In this paper they describe compiling the Links language to Javascript in continuation passing style and using this to implement user level threading for client side Javascript. This should be possible using this Scheme system as well. We can create continuations using call/cc and use the 'set-timeout!' function I added which wraps 'setTimeout' to do the scheduling. Here's an example of using 'set-timeout!'.
(set-timeout! (lambda () (alert "Hello World")) 1000)

This example waits one second and displays the alert. Any Scheme function can be passed, including continuations:

(define abort #f)

(define (yield)
(call/cc
(lambda (k)
(set-timeout! k 10)
(abort))))

(define (spawn thunk)
(call/cc
(lambda (k)
(set! abort k)
(thunk))))

(define (fib n)
(yield)
(if (= n 0)
0
(if (= n 1)
1
(+ (fib (- n 1)) (fib (- n 2))))))

This code allows you to spawn a single thread which will run in the background of the browser. That thread can call 'yield' to return control back to the browser for a brief time. A simple fibonacci function is provided which calls 'yield'. For example:
(spawn (lambda () (display (fib 10))))

This will return control immediately. You can carry on using the browser and the result will be displayed when the thread completes. A simple thread scheduler implementation would be an interesting exercise.

Categories: ,

Tuesday, May 02, 2006

Links: Web Programming Without Tiers

From Lambda the Ultimate:
Links is a programming language for web applications. Links generates code for all three tiers of a web application from a single source, compiling into JavaScript to run on the client and into SQL to run on the database. Links provides support for rich clients running in what has been dubbed `Ajax' style. Links programs are scalable in the sense that session state is preserved in the client rather than the server, in contrast to other approaches such as Java Servlets or PLT Scheme.

It looks like Links is making some great progress. A couple of interesting points from the paper:
  • Links supports web continuations. The continuation state is stored on the client, not the server. They defunctionalize the continuation object, replacing functions with unique identifiers. In this way they reduce the size of the continuations to be manageable for client side storage.
  • Client side concurrency! The client side code is compiled to Javascript in continuation passing style. This gives lightweight threads in Javascript on the browser.


Categories: ,

Space Invaders Emulator in Haskell

I been tinkering with Haskell lately to get familiar with a purely functional programming language. As a project to work on I'm trying to port the Space Invaders emulator I wrote for Factor to Haskell.

I'm pretty much a complete newbie to Haskell and have no idea on the best ways to implement this type of thing. I'm starting simple, seeing where the bottlenecks are, and will improve the code from there.

For representing the memory I'm using a standard Haskell Array type. Being a purely functional programming language this means that any updates to the array result in it being copied and the copy being returned. If this turns out to be too inefficient there are some variations on this array type, including UArray for unboxed storage of values.

My first cut at an 8080 CPU abstraction was:
data Cpu = 
Cpu { a :: Word8,
b :: Word8,
c :: Word8,
d :: Word8,
e :: Word8,
f :: Word8,
h :: Word8,
l :: Word8,
pc :: Word16,
sp :: Word16,
doInterrupts :: Bool,
mem :: Memory }

Functions that set register values in the Cpu type actually return a new Cpu with the old values copied across and the new value set. 'Memory' is the array type:
type Memory = Array Word16 Word8

This represents a mapping from 16 bit numbers (the memory address) to 8 bit values. To make the code easier to read I also created some data types to refer to the 8080 registers:
data Register16 = BC | DE | HL | SP | PC deriving (Show)
data Register8 = A | B | C | D | E | F | H | L deriving (Show)

To implement the instruction set I created an 'Instruction' datatype with constructors for each possible type of 8080 instruction. I got this idea from the OmegaGB Gameboy Emulator in Haskell. Much thanks to the author of that for making their developer log and design ideas available. As an example, part of my 'Instruction' type is:
data Instruction =
MoveRegister Register8 Register8 -- MOV r1,r2
| MoveFromMemory Register8 -- MOV r,M
| MoveToMemory Register8 -- MOV M,r
| MoveImmediate Register8 Word8 -- MVI r,data
| MoveToMemoryImmediate Word8 -- MVI M,data
| LoadRegisterPairImmediate Register16 Word16 -- LXI rp,data16
| LoadAccumulatorDirect Word16 -- LDA addr
| StoreAccumulatorDirect Word16 -- STA addr
| LoadHAndLDirect Word16 -- LHLD addr
| StoreHAndLDirect Word16 -- SHLD addr
| ...

My usual approach for decoding the machine code opcodes into these Instruction types would be to create a function that maps each opcode from 0 through to 255 to one of the instructions. Instead of doing this I went to the 8080 instruction set reference and created Haskell overloaded functions that work on the binary of the opcode and decode the registers,etc to operate on from that. So some of these functions look like:
opcodeToInstruction (Binary 0 1 d3 d2 d1 1 1 0) _ _ = 
let r = register8Lookup (d3,d2,d1) in
MoveFromMemory r

opcodeToInstruction (Binary 0 1 1 1 0 s3 s2 s1) _ _ =
let r = register8Lookup (s3,s2,s1) in
MoveToMemory r

opcodeToInstruction (Binary 0 1 d3 d2 d1 s3 s2 s1) _ _ =
let r2 = register8Lookup (s3,s2,s1)
r1 = register8Lookup (d3,d2,d1) in
MoveRegister r1 r2

opcodeToInstruction (Binary 0 0 1 1 0 1 1 0) b2 _ =
MoveToMemoryImmediate b2
The 'register8Lookup' uses the binary string for the register as it is encoded in the opcode to return the actual register type:
register8Lookup :: (Integer,Integer,Integer) -> Register8
register8Lookup (1,1,1) = A
register8Lookup (0,0,0) = B
register8Lookup (0,0,1) = C
register8Lookup (0,1,0) = D
register8Lookup (0,1,1) = E
register8Lookup (1,0,0) = H
register8Lookup (1,0,1) = L

This had the advantage of letting the Haskell compiler work out exactly what instruction to decode the opcode value too. It's made the code quite readable.

A 'runInstruction' function is used to execute the instruction and modify the Cpu type. Some examples of implementations are:
runInstruction :: Cpu -> Instruction -> Cpu
-- Data transfer group
runInstruction cpu (MoveRegister r1 r2) =
setRegister8 cpu r1 (getRegister8 cpu r2)

runInstruction cpu (MoveFromMemory r) =
let hl = getRegister16 cpu HL
v = readMemory8 cpu hl in
setRegister8 cpu r v

runInstruction cpu (MoveToMemory r) =
let hl = getRegister16 cpu HL
v = getRegister8 cpu r in
writeMemory8 cpu hl v

runInstruction cpu (MoveImmediate r v) =
setRegister8 cpu r v

runInstruction cpu (MoveToMemoryImmediate v) =
let hl = getRegister16 cpu HL in
writeMemory8 cpu hl v

-- Branch Group
runInstruction cpu (Jump addr) =
setRegister16 cpu PC addr

runInstruction cpu (ConditionalJump condition addr) =
let c = isConditionTrue cpu condition in
if c then setRegister16 cpu PC addr else cpu

The code that looks up the machine code byte from memory converts it to the 'Binary' datatype and calls 'opcodeToInstruction' to get the instruction out of it:
toBinary :: Word8 -> Binary
opcodeToInstruction :: Binary -> Word8 -> Word8 -> Instruction
readMemory8 :: Cpu -> Word16 -> Word8

step1 :: Cpu -> Cpu
step1 cpu =
let addr = pc cpu
opcode = readMemory8 cpu addr
b2 = readMemory8 cpu (addr+1)
b3 = readMemory8 cpu (addr+2)
instruction = opcodeToInstruction (toBinary opcode) b2 b3 in
runInstruction cpu instruction

With all the instructions implemented I ran a simple benchmark of 1,000,000 instructions. Using '-O2' with GHC I got approximately 20 seconds on my machine (an Athlon 2500+) which is a tad slow.

As a comparison, my Factor implementation runs 1,000,000 instructions in about 1.7 seconds.

An easy change to make was to change the memory type to use a UArray which is an unboxed array. This immediately change the timing to 1.6 seconds. Which is much better than I expected. Remember that this is unoptimised code and purely functional. The only impure code is the I/O printing the CPU registers after each instruction (which I piped to /dev/null for the timings).

The implementation currently emulates about 90% of the 8080 CPU instruction set. I still need to handle interrupts and then add a simple GUI. Once that's done I'll make the code available.

Overall I found Haskell quite easy to develop with. I used 'ghci', the interpreter that is part of GHC, for most of the testing and iterative development. Using that was a lot like developing with a Lisp REPL. The static typing didn't get in my way and in fact it pointed to errors I'd made at compile time instead of run time in a lot of cases. I'll see if that remains the case when I tackle the GUI though which is likely to be a bit more difficult.

Categories: