Distributed Concurrency in Factor

One of the advantages (and pretty much the only one) of being down with the flu recently is I've been able to hack away at Factor code, like the serialization library mentioned previously.

I've just added simple distributed message passing support to the concurrency library. Processes now belong to 'nodes'. These are individual Factor instances running on a machine. Messages can be sent between Factor nodes using the same foramt as sending from processes within a Factor instance.

So far the support I've added is very basic and not at all optimized but it works. Messages can be any Factor type supported by the serialization library.

I've extended the serialization library by added a serializer for local processes that serializes it as a 'remote-process'. This holds the node details (hostname and port) and the process identifier (known as the pid). This allows you to send a local process to a remote process, and that remote process can send a reply back to the local one by sending a message to the remote-process object it receives.

A possible future extension to this might be to serialize proxies for other types. For example, sending a stream in a message can serialize it as a proxy stream so that writes to it from the remote system are sent back to the stream on the local system.

The current state of the system is in my repository:

darcs get http://www.bluishcoder.co.nz/repos/factor

The 'start-node' word is used to start up a TCP listener to handle requests from remote processes. This is required for distributed message passing to work. If you don't use 'start-node' only local message sends will be enabled. Here's a simple example that sends a message to a remote process, and it sends a reply back to the caller:

#! On Machine 1
"concurrency" require
USE: concurrency

"machine1.com" 9000 start-node

: process1 ( -- )
  receive "Message Received!" reply process1 ;

[ process1 ] spawn "process1" swap register-process

This starts the node up with hostname 'machine1.com' on port 9000. A word called 'process1' is defined that blocks until it receives a message (either from another local process or a remote process) and replies to the message sender with the string "Message Received!". It then calls itself to restart the blocking on message receive.

This word is spawned as a process and registered in that nodes global register of named processes as 'process1'.

#! On Machine 2
"concurrency" require
USE: concurrency

"machine2.com" 9000 start-node

[ "machine1.com" 9000  "process1"  
  "test-message" swap send-synchronous .
] spawn

This code, run in a Factor instance on Machine 2, starts a node with hostname 'machine2.com' on port 9000. It spawns a process which sends a message to the process named 'process1' on the node at hostname machine1.com, port 9000.

This example sends a synchronous message. It is the equivalent of Termite Scheme's '!?' operator or Joe Armstrongs '!!' proposed Erlang extension - basically an RPC call. The message is sent to process one and blocks waiting for a reply to that specific message. On the reply it displays it (using '.') which results in 'Message Received!' being printed.

Asynchronous sends work too. For example, a logger process on machine 1:

#! On Machine 1

: logger ( -- )
  receive print logger ;

[ logger ] spawn "logger" swap register-process

Messages can be sent to this process from any machine with:

#! On Machine 2

[ "machine1.com" 9000  "logger"  
  "Log Message!" swap send
] spawn

After this message send 'Log Message!' will be printed on machine 1.

Messages can be sent to named processes, registered in the nodes global registry, as these examples show, or they can be sent to any process given the pid - a unique identifier for that process. They can also be sent to remote processes given a deserialized reference to the process object. You could store on a file system or web server deserialized references to important processes that clients can send messages to.

I'm still working on the public API and making the performance better. Currently all message sends open and close the TCP socket to the remote server per message. There is also no security. The server connection for the node is accessable by anyone. Initially I may follow the Erlang 'magic cookie' approach to prevent unauthorised message sends but keen to look at better options.

My motivation for working on this is to add to my in-progress web framework the ability to have the server side processes distributed across Factor instances or machines. This is one way to enable Factor to use multiple processors in a machine for example.

Categories: factor, concurrency

Object Serialization with Factor

As Slava noted in his weblog, I've tidied up a previous serialization library to get it working with current Factor versions in the hopes of using it to get distributed concurrency working.

The serialisation supports most Factor types and is extensible by adding methods to generic functions. You can even serialize quotations containing words, and so long as those words exist in the target system, the deserialization will link them correctly and the quotation is callable. Actual code doesn't serialize at this stage though.

An example of usage:

"serialize" require
USE: serialize

[
  [ "Hello World!" serialize ] with-serialized
] string-out
  => ...serialized format as a string...

[
  [ deserialize ] with-serialized
] string-in 
  => "Hello World!"

The 'string-out' and 'string-in' shown above are standard words to direct all output to go to and from strings respectively. You can also serialize to files and deserialize them on any other Factor system.

Categories: factor

Transterpreter source available

The source for the Transterpreter, a portable Occam implementation, has been made available.

The Transterpreter is a small (2000 lines of code), portable (strict ANSI C), open-source runtime for a growing family of massively concurrent programming languages. Capable of supporting thousands of threads on small devices, it is well suited for embedded and ubiquitous systems development.

Categories: occam

Cells style gadget updating in Factor

Factor has a Cells like mechanism for propogating changes in models to their GUI representation (called Gadgets). It's like a lightweight functional reactive programming system. Using the latest darcs Factor code, which includes a Calendar library, you can quite easily get a gadget displaying with a dynamically updated date.

To have a gadget dynamically updated it must reference a 'model' which wraps the value to be displayed. Whenever the data wrapped by the model is changed, the gadget displaying it automatically updates.

In this example I create a global value called 'time' that holds a model wrapping the current date and time:

SYMBOL: time
f <model> time set-global

The model here wraps the value 'f' which is false indicating no value currently set. To have the value change regularly I start a background thread which sets the value every second:

: update-time ( -- )
  now time get set-model 1000 sleep update-time ;

[ update-time ] in-thread

The 'update-time' words gets the current date and time (using 'now') and sets the model's value to it. It does this every 1,000 milliseconds.

We can confirm that this value is updating by doing a 'get' of the 'time' variable and see that it changes between calls:

time get model-value . 
  => T{ timestamp f 2006 8 7 12 52 8.98 -12 }

time get model-value . 
  => T{ timestamp f 2006 8 7 12 52 49.14 -12 }

A label gadget that displays this value, and is updated as it changes, can be created and dislpayed with:

time get [ timestamp>http-string ] <filter> <label-control> gadget.

The <filter> object takes a model (retrieved with 'time get') and quotation that it will call on the model when its value changes to get a string to display. In this case I'm using the calendar libraries 'timestamp>http-string' word to convert the timestamp held by the model into a string representation. A label gadget is created using this filter as the displayed text.

The 'gadget.' word is what displays the gadget in the GUI. There should appear a date that is updated every second. You can display multiple gadgets on the same model, scroll the window, etc and it will be updated. You can see a screenshot here. You'll have to imagine the seconds value in the screenshot updating every second!

Categories: factor

JSON parser for Factor

I've implemented a simple JSON parser for Factor following RFC 4627. It will be available soon from the standard Factor repository and is also available from my darcs repository:

darcs get http://www.bluishcoder.co.nz/repos/factor

The three main words in the 'json' vocabulary are '>json', 'json-print' and 'json>'. An example of use is:

"json" require
USE: json
TUPLE: foo a b ;
"hi" "there"  >json print
  => "{"a" : "hi", "b" : "there"}
"[ 10, 20, \"foo\", { \"value\" : 1e2 } ]" json> .
  => { 10 20 "foo" H{ { "value" 100 } } }

Factor tuples are converted to Javascript objects. Going the other way, Javascript objects are converted to Factor hashtables.

Categories: factor, javascript

Factor web framework

I'm working on a new web framework for Factor. My plan is to have web applications built as a collection of concurrent processes running on the server and the client browser. Processes on the server can send messages to the browser (using a 'comet' style connection) and vice versa.

Client processes are implemented using the lightweight concurrency system I built using Narrative Javascript. An example process that runs on the browser looks like:

function alerter() {
  while(true) { 
    var msg = receive->();
    document.getElementById("messages").value += (msg + "\n")
  }
}
register_process("p1", spawn(alerter));

This spawns a process named in the registry as 'p1'. It blocks waiting for a message. When one is received it will add the message string to an text area with the Id 'messages' and then resume waiting.

The message can come from other client side processes or from the server if a comet connection is established. From the Factor server you can get access to the client side process with 'get-process-on-page'. Each page connected to the server has an 'id'. 'get-process-on-page' takes this id and returns a process which can be used to send messages to the client process:

"1234" "p1" get-process-on-page "Hello" over send "World" over send

This example gets the 'p1' process on the client with the id '1234' and sends it two messages. One 'hello' and then 'world'. These will result in the 'p1' process shown above running on the browser waking up and adding the text to the textarea.

Most Factor objects can be sent as messages and they are serialised to JSON when sent to the client. I'm currently writing a JSON parser for Factor which will allow client processes to send messages to server processes.

The framework so far works up to this point and runs on Opera, Safari, Firefox and Internet Explorer. Once the JSON parser is written and working I'll make the code available.

I'm keen to see how this style of web programming compares to continuation based web servers, and how they can be combined (using cont-responder in Factor). I hope to also write other back ends (Scala, Erlang, etc) to compare the different technologies.

Categories: factor, javascript, concurrency

Factor libraries updated

I've updated a number of the libraries I contributed to Factor so they run with the latest version. These include:

• cont-responder, the continuation based web server
• Parser Combinators (the lazy list library I wrote was kindly updated by 'Yuuki' from the #concatenative IRC channel)
• sqlite and tuple-db, the database libraries
• concurrency (Erlang style messaging)
• Space Invaders Emulator

The Factor module system in the latest release is very nice. It makes loading space invaders as simple as:

"space-invaders" require

There have been a number of great features added to Factor recently, so it's worth a look if you haven't tried it out for a while.

Categories: factor