New New Zealand BJJ Brown Belt!

Big congratulations to Geoff Grant of GSW Martial Arts in Wellington, who was awarded his brown belt in Brazilian Jiu Jitsu by BJJ Black Belt John Will tonight. More great news was Ko getting his purple belt and Alex getting his blue belt.

The excitement happened at the seminar that John held in Wellington tonight. The seminar was great with a very full mat and lots of great information. We weren't just given techniques but lots of food for thought to apply to our existing grappling moves to improve them. I've been to a few martial arts seminars and John would have to be hands down one of the best teachers out there. Now to work on applying what I've learnt...

Categories: martialarts, bjj

Dynamically adding nodes to mnesia

Mnesia is very flexible in the different ways it allows you to distribute the database. For my purposes I'm keeping it simple and want a master database and allow other mnesia nodes to dynamically start up and access or get copies of the master database tables. This turns out to be easy.

The first step is to have the database working as I outlined in my previous post on mnesia. This runs on my main server. To bring up another node on a different machine I start that node using an additional command line option:

erl -name newnode -mnesia extra_db_nodes "[mainserver@servername]"

The parameter passed to 'extra_db_nodes' is a list of the nodes running the master mnesia database schema. In this case there's only one.

From the 'newnode', start mnesia using 'mnesia:start()' as normal. This will start mnesia and connect to the 'mainserver'. You can now run queries and transactions on the remote database tables. It's also possible to set up the tables to 'replicate' to the 'newnode' so that queries are run against the data locally.

Categories: erlang

Distributed Erlang and Firewalls

I have two Erlang nodes communicating with each other, one of which is behind a firewall. The Erlang distribution mechanism uses a couple of TCP ports that need to be opened in the firewall. The first is port 4369, used by 'epmd'. The second port is dynamically assigned which makes it difficult to configure the firewall.

You can pass some kernel configuration parameters to erlang to tell it what range of ports to use. A 'min' and 'max' port value can be configured. By setting these to the same value you can ensure only a single port is used. This can be configured like this:

erl -name "mynode" -kernel inet_dist_listen_min 4000 inet_dist_listen_max 4000

This would ensure the port is number 4000. If you are using yaws then you can pass this information to yaws with the 'erlarg' argument:

yaws -i -name yaws -erlarg "-kernel inet_dist_listen_min 4000 inet_dist_listen_max 4000"

Categories: erlang

reddit review

reddit gets a very positive review from Solution Watch. Reddit is a web application written in Common Lisp, developed as part of Paul Graham's startup initiative.

Categories: commonlisp

mnesia

mnesia is the database written in Erlang that is part of the open source Erlang release. It allows storage of any Erlang type, including complex tuples and records, and various methods of querying and selecting these items.

Continuing the saga of my server push project, I need to store the data that is generated by the server events so that web clients can browse the historical information. This gave me the opportunity to work out how to use mnesia - it turns out to be quite easy.

First steps are the following functions that need to be called to initially setup the mnesia system:

mnesia:create_schema([node()]).
mnesia:start().

The easiest way of using mnesia seems to be to store and retrieve Erlang records. Defining a record for my data is done with '-record':

-record(odds, {key, date, meeting, race, update_time, close_time, pools, dividends})

This data is dividend information for a particular race produced by a Totalizator agency. When a race runs, at various intervals an update of the amount of money bet on each runner in the race is provided. This 'odds' record holds that information. The data in the record also has complex information. For example, 'dividends' is an Erlang list of tuples, one for each entrant in the race, each tuple containing the runner number, current dividend if that runner wins and current dividend if it comes 1st,2nd or 3rd, etc).

To create a table to hold instances of this record type, the mnesia function is 'create_table':

mnesia:create_table(odds,
                    [{attributes, record_info(fields, odds)},
                     {disc_copies, [node()]},
                     {type, bag}]).

First we pass the name of the table, odds, and then a list of key/value pairs holding configuration information about the table. The first, attributes, lists the field names in the table. The function 'record_info' is used to return all field names from the 'odds' record. This saves us from manually entering [key, date, meeting, race, update_time, close_time, pools, dividends] and protects us from changes to the record structure.

The 'disc_copies' attribute lists the nodes that will hold disc based copies of this table. In this case, only our node. The type of 'bag' means that multiple entries can exist in the table for the same key. One of the other types, 'set', is used for unique keys. Mnesia automatically assumes that the first element in the record ('key' in our 'odds' record) is the key field. For the 'odds' structure this will actually be a tuple of {date,meeting,race} so I can pull all odds event information for a particular race quite efficiently.

My server event producer sends a notification to a 'notifier' process as outlined in my previous weblog posting. The event it provides is actually an instance of the 'odds' record. By creating a process to handle these 'odds' events and I can easily store them in the database:

odds_monitor() ->
    receive
 {odds_change, {Date,Meeting,Race,UpdateTime,CloseTime,Pools,Dividends}} ->
     mnesia:transaction(
       fun() ->
        mnesia:write(#odds{key={Date,Meeting,Race},
      date=Date,
      meeting=Meeting,
      race=Race,
      update_time=UpdateTime,
      close_time=CloseTime,
      pools=Pools,
      dividends=Dividends})
       end),
     odds_monitor()
    end.

All updates to mnesia need to be done as part of a transaction. The 'mnesia:transaction' function takes a function as an argument. It is inside that function that all the database transactions should occur. In this case a simple write of an 'odds' record. The syntax '#odds{..}' creates a new odds instance.

With this process running and registered with the event notifier I now get the data stored in the database as well as sent to the web clients. Getting all the odds data for a given race is as simple as:

mnesia:transaction(
    fun() ->
        mnesia:read({odds, {{2005,11,25},"MEETING1",1}})
    end).

'read' returns the object that has the specific key. Or in the case of a 'bag' table type, all the objects with that key. It is also possible to do quite complex queries across all fields in the table but for my purposes this call is enough. An example of the data returned is:

{atomic,{odds,{{2005,11,25},"MEETING1",1},
               {2005,11,25},
               "MEETING1",
               1,
               {21,3,0},
               finished,
               [{win,14831},{place,9579},{trifecta,12400}],
               [{1,3.20000,1.35000},
                {2,2.20000,1.25000},
                {3,8.30000,2.15000},
                {4,34.2000,4.70000},
                {5,scratched,scratched},
                {6,27.3500,5.45000},
                {7,scratched,scratched},
                {8,20.0500,3.65000},
                {9,34.1000,5.75000},
                {10,9.25000,1.80000},
                {11,20.5000,4.70000}]}}

The documentation for mnesia has a good tutorial, as well as the reference of all the functions.

Categories: erlang

learning Lisp with dmnd

A new Common Lisp blogger, dmnd, is writing about their attempts to learn Lisp and working through Peter Siebel's book, Practical Common Lisp.

Categories: commonlisp

Berimbau Lessons

While I'm on the topic of music, and am unlikely to ever get on it again given my lack of ability, I'd like to plug some great online Berimbau Lessons I used a while back to improve my Berimbau playing. I purchased the downloadable versions of Berimbau Lessons 1 and 2 and they were excellent. Good quality, good music, easy to understand.

I knew how to play the standard Angola rhythm from when I played Capoeira and wanted to learn some others. Both video lessons had a lot of different rhythms and inspired me to play the Berimbau more, as I had more things to try out. I've not seen much on the web with regard to learning this unusual instrument which is a shame. I can definitely recommend Bridges To.

For those unfamiliar with the berimbau, it's a single stringed musical instrument used as the main instrument for the music in the martial art Capoeira. In Capoeira, the 'players', perform the Capoeira techniques against each other, within a circular ring of people who play musical instruments and sing traditional Capoeira songs (usually in Portugese). The rhythm of the music sets the rhythm of the dance/fight/game.

Categories: misc, martialarts

Jamming via the Net

What a neat idea! NinJam, allows people to collaborate over the internet to create music. There's no need to register or provide information, you can just get an anonymous group of people together playing instruments and create music.

NINJAM uses compressed audio which allows it to work with any instrument or combination of instruments. You can sing, play a real piano, play a real saxophone, play a real guitar with whatever effects and guitar amplifier you want, anything. If your computer can record it, then you can jam with it (as opposed to MIDI-only systems that automatically preclude any kind of natural audio collaboration1).

Since the inherent latency of the Internet prevents true realtime synchronization of the jam2, and playing with latency is weird (and often uncomfortable), NINJAM provides a solution by making latency (and the weirdness) much longer.

Latency in NINJAM is measured in measures, and that's what makes it interesting.

The NINJAM client records and streams synchronized intervals of music between participants. Just as the interval finishes recording, it begins playing on everyone else's client. So when you play through an interval, you're playing along with the previous interval of everybody else, and they're playing along with your previous interval. If this sounds pretty bizarre, it sort of is, until you get used to it, then it becomes pretty natural. In many ways, it can be more forgiving than a normal jam, because mistakes propagate differently.

Link found from Mike Austin's blog.

Categories: misc

Observers, Erlang and Server Push

As I've mentioned previously, I'm working on a framework for pushing events from an Erlang based web server to browser based clients using AJAX. My goal being to make it look like normal Erlang message sends to 'processes' running on the browser.

I have a server process that needs to broadcast events to interested parties whenever something of interest happens. My first thought was that this is the classic Observer pattern in OO style frameworks. I wanted other processes to be able to register with the server and they would then receive the notification. They can then later unregister when no longer interested in the data.

This turns out to be very easy in Erlang. A simple process to handle this looks something like:

notifier(Observers) ->
    receive
        {notify, Message} ->
            lists:foreach(fun(To) ->
                            To ! Message
                          end,
                          Observers),
            notifier(Observers);
        {register, Observer} ->
            notifier([Observer|Observers]);
        {unregister, Observer} ->
            notifier(lists:delete(Observer, Observers))
    end.

Given a notifier, observers can register their interest in events. A server then sends a 'notify' message to the notifier, passing it the required data. This is in turn sent to all currently registered observers. So something like:

test() ->
    receive
        X ->
            io:format("Received ~p~n", [X]),
            test()
    end.

Changes = spawn(module, notifier, [[]]).
Observer1 = spawn(module, test, []). 
Changes ! {register, Observer1}.
Changes ! {notify, "Hello Message!"}.
...observer1 will output 'Received Hello Message!'...
Changes! {unregister, Observer1}.

Using code similar to this I created a notifier for my data source, and the observers are my HTML web pages. This is easy since I'm modelling each active page as a process. When an event occurs, the HTML on all displayed web pages are updated immediately using the AJAX based server push.

Categories: erlang, ajax

Seaside Praise

Chris Petrilli is using Seaside and seems to be enjoying it:

Rails is brilliant—for what it is. It takes the historical model of page interaction and data storage to new heights of simplicity. It doesn’t, however, change how you view the web. Seaside does. Whether you use it for your next project, or not, it’s worth looking at, going through the tutorials, and allowing your mind to conceive of a web that simply behaves more naturally.

(From James Robertson).

Categories: smalltalk

Frag: Haskell First Person Shooter

Frag is a 3D first person shooter game written in Haskell. Screenshots and instructions available at the Haskell Wiki. The announcement and a discussion about it here.

Features:

• Yampa, a domain-specific embedded language for the programming of hybrid systems that using the concepts of Functional Reactive Programming (FRP) was used to program the game entities.
• The Quake 3 BSP level format, Q3Map2, and the MD3 format for models and animations are used in this game.
• Sven Panne's OpenGL binding, HOpenGL is used to render graphics.

Categories: haskell

Paul Graham on Web 2

Paul Graham has a great article on 'Web 2'.

As you read this, a whole new generation of software is being written to take advantage of Ajax. There hasn't been such a wave of new applications since microcomputers first appeared. Even Microsoft sees it, but it's too late for them to do anything more than leak "internal" documents designed to give the impression they're on top of this new trend.

...

The proof that Ajax is the next hot platform is that thousands of hackers have spontaneously started building things on top of it. Mikey likes it.

Categories: web

Real World Haskell

Joel Reymont is quite well known for using languages like Erlang and Common Lisp for building real world production applications to gain competitive advantage. On his weblog he's been quite open about how he has used these languages for things like his Open Poker server, and 'uptick' trading system. Now it seems he's using Haskell for some of his work. Hopefully he'll start blogging about what he finds the strengths of Haskell to be. His post is very positive about the language, although there seem to be some rough spots:

Overall, the experience with Haskell has been exhilarating and I'm already preparing to use it on my next projects like detecting collusion in poker as well as rake optimization (Dazzle paper very helpful here!). Still, I think that GHC can be a bit rough around the edges and I would think twice about writing high-performance network apps with it.

Categories: haskell

Monads in Ruby

A note to myself to read this Monads in Ruby article. It looks like a nice, understandable, introduction to what Mondads are and how they work. From Lambda the Ultimate.

Categories: ruby

Update to Backbase and Forms

I've made some small updates to the 'Backbase and Forms' post based on feedback from Grauw.

The changes were:

• I now use 'ancestor::forms[1]' as the XPath expression to identify the parent form. This works for any level of nesting and is more robust than '../../form'.


• The s:execute example used a 'div' as the container for the s:execute. This causes an uneeded block to be rendered in the HTML. I replaced the div with an empty s:execute, which gets replaced with an s:execute containing the s:task when the form is submitted.


• I cleared up the text about which is the default type for an input. At least, I hope I cleared it up!

Thanks for the feedback!

Categories: backbase

Observing field changes using Backbase

In a web application I'm writing I want to have fields on the page updated when the user changes other fields. This should happen client side or server side if server data is needed. Using Backbase this turns out to be quite easy.

There is an event called 'change' that is fired when the data in a field is changed and the user exits the field. When the change event is fired I can then perform some actions.

The other option is to 'observe' another element. When an element is observed, the observer also receives events that the observed item has. The event received by the observer is called 'observe-X', where 'X' is the name of the event fired by the observed item. If A has the 'change' event and B is observing A, then B will receive the 'observe-change' event. There is a limitation in that an observer can only observe one thing.

I wrote a simple demonstation of this, form3.html. This displays two input fields. When a number is entered into either field, and that field is exited, the sum of the two numbers is immediately displayed below then.

My first thought was to have the 'sum' element observe the two input fields and display the result when either of the two change. Unfortunately you can't observe two fields in the current version of Backbase so instead each field process its own 'change' event and sets 'sum' when it changes.

The 'sum' itself is just a span:

<span id="result">0</span>

The 'input' fields have an event handler:

<input id="number1" type="text" size="20" value="0">
      <s:event b:on="change">
            <s:task b:action="set" 
                    b:target="id('result')/text()" 
                    b:value="{(id('number1')/@value)+(id('number2')/@value)}"/>
      </s:event>
</input>

The handler is for the 'change' event. When this occurs we use the 'set' action to set the value of a DOM node. The 'target' and 'value' of the 'set' action is defined using XPath. For the value I take the 'value' attribute of the number1 and number2 nodes. These are input fields, where 'value' holds the data in the field. The target is the 'result' span, choosing to replace the text of that span.

The same code exists for the 'number2' input field. Any event could be run during the 'change' handler. I could have done an Ajax request to the server to do the calculation for example.

Categories: backbase

The Rev's Seminar

Geoff has a summary, including photos, of the seminar held at GSW by John 'The Rev' Jensen, from Dominance Jiu Jitsu. Sounds like it was a fantastic seminar. I was unfortunately not able to make it myself due to having to work that weekend, but I'll definitely be making the John Will seminar in Auckland and Wellington. It'll be a busy time in Auckland for me, with the two seminars, the Common Lisp Users Group meeting, and hopefully catching up with Massive on the Sunday.

Categories: martialarts, bjj

Blackdog review

A nice review of the Blackdog with screenshots. The forum comments are also worth reading as it includes input from a Blackdog engineer with advice on how you can use it.

Once helpful comment was how to avoid keylogging software. Blackdog literature puts forward one use for the product is in untrusted environments to access secure web sites, etc. But it's been pointed out that you are still vulnerable to key logging on the host. By having firefox on the blackdog remember all the passwords and 'autofill' the fields in you can avoid this. Because the passwords are stored on the Blackdog - which can be set to be only accessable via fingerprint authentication - they can't be retrieved by the host. And if they are autofilled by firefox they can't be captured by a keylogger.

Categories: blackdog

Ajax for Mobiles: Opera Platform SDK

This is a neat idea. Opera have released the Opera Platform SDK for mobile phones. From the web page it appears to be a framework written in Javascript that runs on the Opera browser on the phone. Installed locally on the phone are webservices for access phone functions (contacts, etc). You write your applications in Javascript, using Ajax type techniques to talk to webservices on the phone or via a data connection. Link and more details from Ajaxian.

Categories: ajax, mobile

Backbase and Forms

There was a discussion on the Backbase developer forums recently about how forms work. I found the manual a bit confusing and Laurens Holst from Backbase cleared things up. This post outlines what I learnt - hopefully I got it right.

Backbase can handle normal HTML forms where the submit causes the browser to go to another page with the form data is passed as query parameters or POST data depending on the form method. In this case the form looks and works exactly like a normal HTML form (See form1.html for a live example):

<form name="main" action="action1.yaws" method="post">
  <table>
    <tr>
      <td>Name</td>
      <td><input name="name" type="text" size="40"/></td>
    </tr>
    <tr>
      <td>Data</td>
      <td><textarea name="data"/></td>
    </tr>
  </table>
  <input type="submit" value="Send"/>
</form>

For the submit button you can use 'input' with a type of 'submit', or the HTML 'button' element. They work the same way (form1a.html uses the button element).

The 'type' of an input defaults 'text' and the type of a button is supposed to default to 'submit' but this is not always the case cross browser. Laurens advised on the backbase forums to always specify the type explicitly if you are using the standard HTML elements. For example, some versions of Internet Explorer do not default a 'button' to be 'submit.

Backbase also provides an extension element called 'b:button'. This produces a button that is styled to look the same no matter what browser is used. The standard HTML buttons can look different across browser. Using 'b:button' requires slightly different attributes to the standard input or button type.

It uses two attributes. The first, b:action, is used to define what should happen when the button is pressed. For form submission we'd use 'submit'. You could load another file with 'load' or display an alert with 'alert'. There are many actions that can be used and they are covered in the BXML reference manual.

When the action is 'submit', a second attribute is required. This is b:target. It should be set to an XPath expression identifying which form will be submitted. Usually it's the parent form. You can use a relative path expression (like "../../form") or one indexed by the elements id (for example, "id('mainform')"). Another option is the XPath expression "ancestor::form[1]". This is better than "../../form" as it ensures you get the parent form no matter how many elements are in-between. Using b:button our form above would look like (See form1b.html):

<form name="main" action="action1.yaws" method="post">
  <table>
    <tr>
      <td>Name</td>
      <td><input name="name" type="text" size="40"/></td>
    </tr>
    <tr>
      <td>Data</td>
      <td><textarea name="data"/></td>
    </tr>
  </table>
  <b:button b:action="submit" b:target="ancestor::form[1]">Send</b:button>
</form>

The b:action and b:target tags can also be used with the standard 'input' and 'button' elements. Which you use depends on whether you want normal HTML submission or backbase enhanced submission. You should never put all three tags in:

<input type="submit" b:action="submit" b:target="../../form" value="Send"/>

This would result in the form being submitted twice. Once via the normal HTML mechanism and the second time by backbase. You should also never just include the b:action attribute:

<input type="submit" b:action="submit" value="Send"/>

Note the absence of b:target. This would result in the standard HTML form submission being used. If you use any of the special backbase form facilities like b:destination (see later) on the form they'll be ignored.

None of this really shows the power of backbase forms though. You can very easily do the form submission via an Ajax request - and have the result returned by the form injected into the current pages HTML rather than doing a full page refresh. This is done by adding the b:destination and b:mode tags to the FORM element (See form2.html):

<xmp b:backbase="true">
  <form name="main" 
        b:destination="id('replaceme')"
        b:mode="replace"
        action="action2.yaws" 
        method="post">
    <table>
      <tr>
        <td>Name</td>
        <td><input name="name" type="text" size="40"/></td>
      </tr>
      <tr>
        <td>Data</td>
        <td><textarea name="data"/></td>
      </tr>
    </table>
    <b:button b:action="submit" b:target="ancestor::form[1]">Send</b:button>
  </form>
  <div id="replaceme">
    <p>This paragraph will be replaced with the results of
       the form submission.</p>
  </div>
</xmp>

The b:destination tag is an XPath expression that identifies where the results of the form request will go. The submission will use Ajax to access the normal form action URL, but the data returned from that URL will be placed at the location of the destination. The b:mode indicates whether the data from the request will replace (as in this example) that node completely, or some other behavior ('replacechildren' for example will only replace the children of the destination node).

So entering the form data and pressing 'send' now will result in no page refresh and the result appearing in place of the div with the 'replaceme' id. The result returned from the form submit should be XHTML and can contain BXML. For example, it might look something like:

<?xml version="1.0" ?>
<div id="replaceme"
     xmlns="http://www.w3.org/1999/xhtml"
     xmlns:b="http://www.backbase.com/b"
     xmlns:s="http://www.backbase.com/s">
  <p>Some paragraph</p>
</div>

Notice the div has some backbase namespaces. This is to allow BXML to be placed in the div, although the example doesn't use it here. Example form2a.html does demonstrate some BXML. It is functionally similar to the previous example but instead of inserting a paragraph into the div, it inserts an 's:execute' element to execute a task to display an alert. The effect is to display an alert box sent by the server as soon as the button is pressed:

<xmp b:backbase="true">
  <form name="main" 
        b:destination="id('replaceme')"
        b:mode="replace"
        action="action2.yaws" 
        method="post">
    <table>
      <tr>
        <td>Name</td>
        <td><input name="name" type="text" size="40"/></td>
      </tr>
      <tr>
        <td>Data</td>
        <td><input name="data" type="text" size="40"/></td>
      </tr>
    </table>
    <b:button b:action="submit" b:target="ancestor::form[1]">Send</b:button>
  </form>
  <s:execute id="replaceme"/>
</xmp>

In this case the data returned by the server as a result of the form request is something like:

<?xml version="1.0" ?>
<s:execute id="replaceme"
     xmlns="http://www.w3.org/1999/xhtml"
     xmlns:b="http://www.backbase.com/b"
     xmlns:s="http://www.backbase.com/s">
    <task b:action="alert" b:value="Text of the alert"/>
</s:execute>

There is much more that can be done with form submission. When the submit event is sent as a result of the button press you can intercept it, check the validity of fields, and choose not to submit it, or allow it to continue. Multi-page wizard style forms are available, etc. I'll save those for another post.

Categories: backbase, ajax

AFLAX makes server push easy

Paul Colton, the author of AFLAX, read my post about server push and has added persistent socket connections to the framework. There is a demo of a chat client application here.

This is very nice and is sure to make server push more reliable. Hopefully a Linux client for Flash 8 will appear in the near future.

Categories: aflax, ajax

Blogger Categories

Blogger doesn't support categories so I use del.icio.us for this purpose. Everytime I post I add the categories text at the bottom of the post and when its published I add it to my del.icio.us tags. Up until now I haven't been able to have a list of categories but I've just noticed that del.icio.us has a nifty tagroll so I've added it to my sidebar to act as a categories list.

Categories: misc

More on Ajax and server push

So far I've looked at two ways of doing server push. The first used a hidden IFRAME which had the SRC attribute set to a persistent connection on the server. The server would push Javascript code along this connection whenever it wanted the client to do something. This had the downside of the browser always displaying a 'loading' indicator as the page never completed loading. From a UI perspective this makes it unusable except as a fallback method of last resort.

The second was outlined in my previous post and used XMLHttpRequest. Like the IFRAME, the request was for a persistent connection. This time the browser wouldn't display a 'loading' indicator and was pretty much exactly what I wanted. I'd prefer it if I didn't have to close the connection after each event to prevent the need to re-establish a connection as this limits scalability a little.

The problem with not closing the connection is you don't know when an event is completely sent back to the client so there is no real guarantee that the client has received the entire event data at one time.

A way of getting around this is to use a mime type of multipart/x-mixed-replace. This allows you to define the 'chunks' in your page. So the browser knows when to process each chunk. Not all browsers support this for XMLHttpRequest initiated transactions though. Mozilla appears to have recently got support for it but IE does not. Alex Russell points to a Microsoft Support article that provides another means to do the same sort of thing but then you have to have two data formats depending on which method you are using.

Even if multipart/x-mixed-replace was the tool of choice there is one further problem with persistent connections. Most (if not all) web browsers only have a limited number of connections it uses for HTTP requests. Internet Explorer only uses two for example. This means if you use one for a persistent connection then all other requests only go through the single connection. If you load another page that uses a persistent connection then you're stuck. No more HTTP requests can be made. This pretty much kills the idea of keeping the connection persistent for general use, and is another reason why my previous post closed the connection between events.

In fact, I've since changed the example (not yet available for download) so that it closes the connection after a set timeout as well, to allow other HTTP requests to be processed. I was having problems with using Backbase whereby some of its server requests were blocking due to the persistent connection being around.

I've done some research over the last couple of days and found a third possible option. Flash allows persistent connections to the server via its XMLSocket class. It is also possible to communicate between the browser and an embedded flash application via Javascript. This would allow writing a small flash application that did nothing but make the connection to the server and then Javascript is used for sending and receiving events across that connection.

This has the advantage of using a different connection pool that the browsers, and you can use any protocol. The disadvantage is that XMLSocket can only connect to ports numbered greater than 1024 and it is not tunneled over HTTP so could be blocked by a firewall. But if the firewall issue is not a problem then it would be possible to write a small Erlang tcp server that communicated across that socket and to the browser.

Backbase actually does something very similar for its charting controls. These controls are embedded flash with the data sent to the control using Javascript from the browser. It does not use a socket connection to the server however.

Another library that does similar is the Canvas object in Nevow. That library embeds a flash object in the page which is used for displaying vector graphics. It makes a connection to the server and allows the server to update the canvas as desired via communication over that socket.

The best general library for doing this sort of thing would seem to be the AFLAX library. This amazing library allows calling any Flash library call from Javascript. Unfortunately it's Flash 8 only and that version of Flash is not yet available for Linux as far as I'm aware.

What I plan to do is write a small flash application to embed to do the server connection and explore using that for the server push events. As a fallback I can have the library use XMLHttpRequest if there is a problem due to firewall restrictions. The final fallback could be an IFRAME if the browser doesn't support XMLHttpRequest.

Categories: erlang, backbase, ajax

Pushing events to the browser via Ajax

I mentioned previously that I was exploring pushing events to HTML clients using Yaws and Erlang. I started off using Ajax, changed my mind and went to using a hidden IFRAME and now I've gone back to Ajax again.

Keeping a persistent connection open turns out to be a bit of a problem with browsers that want to limit the number of connections available. But I have some example code working and am making it available now in case someone wants to play with it. The code is availble in html_rpc.tar.gz and requires Erlang and Yaws.

The simple demo shows a single web page and some example commands that you can run from the Erlang shell to send alert messages to the client. I have a partially written messenger application using the same code (just a different Ajax handler) but it's not quite complete enough to make available yet.

To run this simple demo, start Yaws serving the pages in the archive. Make sure you have compiled the html_rpc.erl file with something like the following from the Erlang shell:

1> compile:file("www/html_rpc/html_rpc.erl").
{ok,html_rpc}

You'll also need to start the processes in that file:

2> html_rpc:start().
true

Once that's done load the test1.yaws file in the browser. It'll display a page with instructions showing want to run at the Erlang shell to demonstrate things.

The implementation is actually pretty simple. A page that wants to receive messages needs to include the 'html_rpc.js' file using the SCRIPT HTML tag. This provides a function to use Ajax to call call a server routine, process a message, then recall that routine. The server routine is 'long lived'. That is, it doesn't return until it has a message from the server to deliver to the client. It sends 'no-op' messages every now and then to keep the connection alive.

When the server has a message to send the Ajax request is completed. The client processes it then re-establishes the connection with the server, where things start again.

The Yaws handler to manage the connection is pretty simple:

<erl>
out(A) ->    
  {ok,Page} = queryvar(A, "page"),
    html_rpc:start_ajax_handler(Page, fun(A,B,C) -> handler(A,B,C) end).
    
handler(Yaws_Pid, Page, Message) ->
    case Message of 
 {alert, Value} ->
     "function() { alert('" ++ Value ++ "'); }"
    end.
</erl>

The 'start_ajax_handler' function does all the work. Just pass a 'handler' to handle messages specific for your application. In this case I have an alert message. The handler returns a string that is 'evaled' on the client to become javascript, and that javascript object is passed to a handler in the HTML page. In this example I'm making my message object functions that will be called by my handler in the HTML:

function process_message(message) {
  message()
}
html_rpc_receive("ajax1.yaws?page=1234", process_message);

This code from the client uses 'html_rpc_receive' to establish the Ajax connection. The URL it uses to manage the connection is the Yaws code seen above. I pass a unique page identifier along with it so the server knows which client to send the events to. In this code I've hard coded '1234' but in reality I generate a large random name. 'html_rpc_receive' also takes a handler to process the received messages (the evaled string returned from the Erlang code). As I mentioned before I'm returning a function so I just call it as a function.

From the server I can now run code like:

5> html_rpc:send("1234", {alert, "Hello World"}).
{alert,"Hello World"}

This will immediately display an alert on the client with the text 'Hello World'. You'd want to write handlers that do real stuff though like display messages, animation, etc.

It's quite easy to use but could be easier. I'll be looking at how to improve things and put some more examples up in future posts.

Categories: erlang, yaws, ajax

Seaside: A radically productive framework

IBM Developerworks has an article on Seaside.

Continuations, or high-level programming abstractions, are often seen as academic toys, but a new breed of Web server can make the average developer much more productive. Continuation servers let you support the Back button in Web browsers and code with a more consistent programming style. Developers the world over recognize Seaside as the top existing continuation server, but Seaside does much more than continuations.

Categories: seaside

Pushing events to web clients

I'm playing around with pushing events from the server to web based clients. This isn't new stuff of course, it's been around for awhile in the form of things like Pushlets, mod_pubsub and KnowNow.

An interesting approach was raised by Joe Armstrong on the Erlang mailing list. He suggests viewing HTML pages as processes and being able to send messages to them. Followup posts to that message have some interesting ideas as well.

Because I'm lazy and I didn't want to write a lot of client side GUI and Javascript stuff I'm using Backbase for the client side and generating requests to the server. Backbase has some nice abstractions which make this easy to do in a cross-browser friendly manner. For the server side I'm using Yaws and Erlang.

Currently I've got things working whereby you have a web page running on the client, and you can send messages to it from the Erlang server. When a page is displayed in the browser it spawns an Erlang process for that page. This is stored in a lookup table indexed by a random name. Given this name you can get the process id and send it messages. These messages are sent to the browser immediately via a long running connection that is held open between the browser and server.

So assuming a page with the name '12345678' associated with its process, the following code run on the Erlang server causes an alert box to be displayed on the page:

process_for('12345678') ! {alert, "Hello"}.

This would allow building an html user interface interactively by executing commands on the server. Any backbase BXML can be sent, or as in the case of the alert example, some of them are abstracted out into messages of their own.

I'm putting together a simple messaging example to put online to demonstrate how it works. It's the usual 'server push' example allowing clients to send messages to other clients and have them displayed immediately.

What other ideas could come from treating HTML pages as processes? Writing clients to Erlang programs could become easier, especially if something like Distel was implemented.

Categories: erlang, backbase, web, yaws

Erlang Cookbook

Being pretty new to Erlang I'm not familiar with the functionality contained within the standard library. Searching for ways to convert numbers to strings I came across the Erlang Cookbook which looks to be a very useful resource. For the record, converting a number to a string can be done using the 'format' function in the 'io_lib' module:

hd(io_lib:format("~B", [1000])).
  => "1000"

io_lib:format returns a list so I get the head of the list using 'hd'.

Categories: erlang

Why Bluish Coder?

I've been asked a couple of times why I named the blog Bluish Coder. The reason is pretty simple...I stuck my name in an anagram engine and it was one of the options that came out. Calling a mostly programming oriented weblog 'Subchloride', or 'Bodice Hurls' didn't seem quite as good a choice!

Categories: misc

Massive news

3D World Issue 71 puts Massive through its paces. The 'Latest News' page of the Massive website has a PDF of the article available for download. 3D World rated it 9/10.
In other Massive news, they have an education course soon to be available from educational institutes around the world.

Categories: massive

Factor browser updated

I've updated the online Factor source code browser to run Factor 0.79. There were a lot of changes from Factor 0.78 to 0.79 and I haven't had the chance to merge a lot of my code over. As a result I've keep my factor darcs repository at the old version level and created a new one from 0.79 onwards. This repository can be obtained with:

darcs get http://factor.modalwebserver.co.nz:/factor_79

Don't pull from this into my old factor darcs repository as this one starts from scratch.

Categories: factor

Web API's

Ajaxian have an article about 'Mashups' - a web site that mixes data from various other Web applications. They link to a great list of applications that provide API's allowing them to be integrated into your own web applications: http://www.programmableweb.com/apis

Categories: web

GSW is the place to be!

GSW, the specialist Grappling Studio in Wellington, is rocking along this month. Geoff posts about Steve Silva turning up at class this week. On the weekend there's a seminar held by The Reverend John Jensen and a couple of weeks after that BJJ Black Belt John Will is doing a seminar.

Categories: martialarts, bjj

Pushing Mnesia further

Luke Gorrie writes on Lambda The Ultimate about using Mnesia - the open source database written in Erlang - on large volumes of data:

We have pushed it to >60M records using a custom backend of BerkeleyDB stored on a cheap 12-disk SCSI array. Mnesia scalability is everything in our telecom system since we store one record per subscriber and lots of operators have tens of millions of subs.

With the disk arrays we do blow out to 8U of rack space for a full cluster but it still looks comical compared with the racks and racks that make up Java/Oracle systems at these sites :-)

Categories: erlang

NZ Common Lisp Users Group Meetup

The next meetup of the New Zealand Common Lisp Users Group is in Auckland on the 26th of November starting at 12pm. I'll be in Auckland that day but at a Martial Arts seminar for most of the day. Hopefully I can make the tail end of it or at least make the dinner.

Categories: commonlisp

Io 1.0 Released Date Set

Some interesting news from the Io Programming Language Blog. The release date of the 1.0 version of Io is set to be the 1st of January 2006 and includes some refactorings:

This work involves a reorganization of the source tree, moving much of the implementation from C into Io, and extending the coverage of the unit test. Coroutines will be exposed within the language and actors, expcetions and the asynchronous networking control will be implemented in Io.

I like that parts of the C implementation will be rewritten in Io itself.

Categories: io

Mnesia and millions of processes

Ulf Wiger has being doing some tests putting the Erlang database Mnesia through its paces on a 16GB Sparc. This post to the Erlang mailing list describes some of the findings. Here's a summary:

On a 16 GB machine, you can:

• run 6 million simultaneous processes
  (through use of erlang:hibernate, I was actually
  able to run 20 million - spawn time: 6.3 us,
  message passing time: 5.3 us, and I had
  1.8 GB to spare.)


• populate mnesia with at least 12 GB of data, but
  think through how you want to represent it, since
  the 64-bit word size blows things up a bit.


• keep a 10 GB+ disc_copy table in mnesia. The
  load times and log dump cost seem acceptable
  (10 minutes to load, dumping takes a while but
  runs in the background quite nicely.)

6 million running processes on a single machine is very impressive!

Categories: erlang

The 'Help get Chris out of debt' sale

I'm selling my Nokia 7610 Symbian phone and Bluetooth GPS on TradeMe, an NZ auction site. If you want to play with developing GPS software on the Symbian platform, here's your chance.

Categories: misc

Dylan Inertia

Mike Austin is doing some interesting GUI stuff with Dylan. It looks very nice. A later post shows building GUI's using a declarative format from a file.

Categories: dylan

Factor 0.79

Slava has released Factor 0.79. This version uses OpenGL and freetype for the GUI rendering instead of the SDL_gfx and SDL_ttf libraries.

Categories: factor

Erlang on the Gumstix

Pascal Brisset posts on the Erlang mailing list instructions on compiling and running Erlang on the gumstix. The gumstix is a very small mini-computer.

Categories: erlang

ex11 on the Blackdog

There is a neat little X11 library for Erlang called ex11 written by Joe Armstrong. It's not just a wrapper around the Xlib C library - it talks the directly to the X server using the underlying X protocol. It is very fast and efficient.

With a few tweaks I got the latest working copy of the code (from here) working on the Blackdog. The archive for installing on the Blackdog device is ex11.tar.bz2 and is around 1MB in size. Just untar this file from the root home directory:

  cd /root
  tar jxvf ex11.tar.bz2

You will need to have set up Erlang on the Blackdog first. A 'demo.sh' shell script runs a number of examples:

  cd /root/ex11-latest-snapshot-2004-09-09
  sh demo.sh

The 'tweak' I had to make to the ex11 source was to stop it from looking for the .Xauthenticate file and instead connect to host:0.0, which is what the Blackdog uses to identify the host X server. For some reason the BD doesn't have .Xauthenticate or even the Xauth command. ex11 runs very snappy on the Blackdog.

Categories: erlang, blackdog

Mickaël Rémond reviews the Blackdog

Mickaël Rémond has a review of the Blackdog computer. It was a post from Mickaël to the Blackdog forums that originally got me interested in running Erlang on it.

Categories: blackdog

Backbase and Yaws

The other night I was trying to get Backbase to work using the Yaws webserver. This in itself is trivial since the Backbase community edition is just a set of files that are served up by any webserver. The main difficulty I was striking was using the dynamic HTML generation of Yaws to generate the Backbase XHTML code - which uses elements and attributes with a special namespace prefixed to them.

Yaws has an 'ehtml' facility that works a bit like the various Lisp based S-Expression HTML libraries. The following code in a Yaws dynamic file:

out(A) ->
  {ehtml, {p, [], "hello"}}.

generates:

<p>hello</p>

I struck a problem using DIV's though:

  {ehtml, {div, [], "hello"}}.
** 1: syntax error before: 'div' **

This is because 'div' is an infix operator in Erlang:

 1000 div 10.
   => 100

Similar problems occurred if I tried to use namespace prefixed elements and attributes. The solution for both cases, thanks to the Yaws mailing list, was to place these problematic symbols in single quotes:

  {ehtml, {'div', [], "hello"}}.
  {ehtml, {'b:form', [{'b:destination', "URL"}], "Form Stuff"}}.

Categories: backbase, yaws, erlang

Erlang and Yaws on the Blackdog

Further to my post about the Blackdog, here are the binaries and setup instructions to get Erlang and Yaws running on it.

• otp_R10B-8.tar.bz2 (32Mb)
• yaws.tar.bz2 (800Kb)
• stow.tar.bz2 (4Kb)

Copy these files to the blackdog then, assuming you don't have stow already installed, perform the following to install Erlang and Stow from within a blackdog xterm:

mkdir /usr/local/stow
cd /usr/local/stow
tar jxvf /root/otp_R10B-8.tar.bz2
tar jxvf /root/stow.tar.bz2
stow-1.3.3/bin/stow stow-1.3.3
stow otp_R10B-8

I compiled Yaws to be installed locally so the steps for that are slightly different:

cd /root
tar jxvf yaws.tar.bz2

Edit the 'yaws.conf' file to reflect what you want or you can leave it at the default to test the installation. To run:

export PATH=/usr/local/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/lib:$PATH
ldconfig
cd /root
bin/yaws -i -c yaws.conf

This runs Yaws in 'interactive' mode. Visiting http://localhost:8000 should confirm things worked by showing the yaws standard install. To exit interactive mode and shut down Yaws press CTRL+G and press 'q' and enter.

Erlang can be run as usual with 'erl'.

Categories: erlang, blackdog

Blackdog

I recently got a Blackdog device to try some ideas out. The Blackdog is a very small Linux server with no screen or keyboard.

You plug it into a PC via USB and it emulates a USB CD device. The host machine then autorun's this CD. If the host is a Windows machine it starts an X client. If it's a linux machine it does nothing.

Once that is done the Blackdog then stops pretending to be a CD and pretends to be a USB ethernet device and sets a private network between the host and the Blackdog. The running X client on the host machine then connects over that network to the X server on the Blackdog. Through this you can run programs like xterm and have them display on the host. The Blackdog also does fingerprint authentication during this process to ensure the user is authenticated.

It's a pretty neat little device and has some interesting possibilities. It works flawlessly connecting to Windows XP machines. For Linux machines there are a few issues as newer kernels need a patch applied to it. But with a little tweaking I've got it working on a number of machines.

The blackdog has a PPC based processor. To develop for it you use the Blackdog SDK. This includes a QEMU based system which emulates the Blackdog. You compile the software from in that and then scp it to the Blackdog. Alternatively you could apt-get the development tools on the Blackdog itself.

I've successfully got a number of programs running on it including Factor, Erlang and Yaws. These run very nicely in fact. Hopefully I can develop some of my ideas so it becomes more than an interesting gadget and something actually useful to me.

Categories: blackdog

Trap Exit

Trap Exit is a great Erlang resource that I came across recently. It mirrors the Erlang mailing list as a forum, and has a number of great HOWTO's on using Erlang.

Categories: erlang

Ivy Programming Language

From PLNews: The Ivy programming language has been announced. It is an extensible, late-binding, bytecode-interpreted scripting language. Ivy includes support for closures, lambda functions, named argument passing, default arguments, garbage collection, C-like operators, built-in string slicing, plus other features.

From the announcement the author also plans to look at adding coroutines or closures.

Categories: programming, ivy

Goodbye Radio Userland, Hello Blogger

I've decided to move from my old Radio Userland weblog and try out a blogger weblog hosted under my own domain. I've been using mostly Linux for the last few years and it's just too painful running Radio Userland under Wine.

Categories: misc