Bluish Coder: mozilla

Experimental Playback Statistics for HTML Video and Audio

2010-08-24T17:30:00+12:00

Now that HTML video is getting more usage there have been requests for statistics during playback so performance can be measured from JavaScript. This has come up a few times on the WHATWG mailing list.

I raised bug 580531 to add some additional data to media and video DOM elements to provide this information. The patch in that bug adds two attributes to the DOM HTMLVideoElement object, both prefixed by ‘moz’ to prevent name clashes as they are experimental:

interface nsIDOMHTMLVideoElement : nsIDOMHTMLMediaElement
{
   ...
   // A count of the number of frames that have been decoded and ready
   // to be displayed. This can be used for computing the decoding framerate
   readonly attribute unsigned long mozDecodedFrames;
 
   // A count of the number of frames that have been dropped for performance
   // reasons during playback.
   readonly attribute unsigned long mozDroppedFrames;
};

mozDecodedFrames is an incrementing count each time a frame is decoded and ready to be displayed. mozDroppedFrames is incremented each to a frame is dropped to keep playback going at the correct frame rate. These can be used to compute the current framerate that the video is being decoded at, and the framerate that it should be decoding at by combining the two counts. This will give client JavaScript code an indication if the hardware is able to play the video.

Another requested feature is information on the download rate. Currently the Firefox implementation of HTML video and audio uses a non-standard extension to ‘progress’ events. We provide information attached to the event that gives the current byte position in the media file as it downloads, and the total bytes available. This was actually required by the WHATWG spec at one point but it changed a while back and I don’t think any other browser implements it.

This has been used in experimental pages to compute things like download rate and we use it in our own controls implementation to display a progress bar as we didn’t implement the ‘buffered’ attribute. Support for ‘buffered’ has now landed so we want to remove the non-standard ‘progress’ information to be spec compliant.

To address the needs of those using the progress data for bandwidth calculation I’ve added two attributes to HTMLMediaElement:

interface nsIDOMHTMLMediaElement : nsIDOMHTMLElement
{
   ...
   // The approximate rate at which the media resource is being downloaded in
   // bytes per second. If the rate is not available (possibly due
   // to not having downloaded enough data to compute a consistent value)
   // this will be NaN.
   readonly attribute float mozDownloadRate;
 
   // The approximate rate at which the media resource is being decoded in
   // bytes per second. If the rate is not available this will be
   // NaN.
   readonly attribute float mozDecodeRate;
};

mozDownloadRate is an estimate, in bytes per second, of the current download rate of the media. If not enough information has been downloaded for a reliable estimate this will be NaN. mozDecodeRate provides the rate at which decoding is occurring. In the patch this is the length of the video divided by the duration and remains constant.

Whether this gets landed is yet to be determined but I think the information is useful and I’d be interested in any feedback on the data I’ve decided to include. There is some feedback in the bug from the patch review already and there’s plenty of time between now and when the Firefox 4 rush is over to look over ideas of what could be included, removed or changed. I posted about the proposed additions in the WHATWG mailing list and there are some responses in that thread.

Safer C Code Using ATS

2010-06-02T18:00:00+12:00

When developing the original Ogg backend for Firefox we had to integrate and use a number of C libraries. Over the months after landing a number of bugs were raised for various issues in the backend. A fair number of these were the result of common coding errors. Things like not checking return values of the C API calls and not releasing resources correctly. This was internal to the third party libraries we used as well as in our own code. This has made me interested in looking at programming languges that make these sorts of errors easier to find.

I’ve previously written about ATS, a programming language with dependent and linear types. With ATS you can use C code directly and annotate the code with types to make some usage errors detectable at compile time. Some papers that cover this usage of ATS are:

In this post I look at ways of using a C library from ATS that shows dealing with static checking of return values and resource releasing. I start from a simple wrapper and then explore how I can use ATS to make usage of the API safer from common programmer errors. The library I’m using to do this is libcurl.

A simple usage of libcurl to retrieve the HTML from a URL and print it to stdout is (simple.c):

#include <curl/curl.h>

int main(void)
{
  CURL *curl;
  CURLcode res;

  curl = curl_easy_init();
  curl_easy_setopt(curl, CURLOPT_URL, "bluishcoder.co.nz");
  res = curl_easy_perform(curl);
  curl_easy_cleanup(curl);
  return 0;
}

This can be compiled and run with:

$ gcc -o simple simple.c `curl-config --libs`
$ ./simple
...

Ensuring release of resources

My first attempt at translating this into ATS is in simple1.dats (pretty-printed version):

%{^
#include <curl/curl.h>
%}

absviewtype CURLptr (l:addr) // CURL*

abst@ype CURLoption = $extype "CURLoption"
macdef CURLOPT_URL = $extval(CURLoption, "CURLOPT_URL")

extern fun curl_easy_init {l:addr}
  ()
  : CURLptr l = "#curl_easy_init"
extern fun curl_easy_setopt {l:addr} {p:type}
  (handle: !CURLptr l, option: CURLoption, parameter: p)
  : int = "#curl_easy_setopt"
extern fun curl_easy_perform {l:addr}
  (handle: !CURLptr l)
  : int = "#curl_easy_perform"
extern fun curl_easy_cleanup {l:addr}
  (handle: CURLptr l)
  : void = "#curl_easy_cleanup"

implement main() = let
  val curl  = curl_easy_init();
  val res = curl_easy_setopt(curl, CURLOPT_URL, "www.bluishcoder.co.nz");
  val res = curl_easy_perform(curl);
  val () = curl_easy_cleanup(curl);
in
  ()
end;

The ‘main’ function looks very much like the C code. The definitions before that make the libcurl C functions, types and enums available to ATS. To compile and run this code:

$ atscc -o simple1 simple1.dats `curl-config --libs`
$ ./simple1
..

The libcurl type CURLoption is a C enum. To wrap this in ATS I use $extype to refer to the C name directly:

abst@ype CURLoption = $extype "CURLoption"

I only reference one value of these types. That is CURLOPT_URL and again I reference the C value directly. Since this is a value rather than a type I use $extval:

macdef CURLOPT_URL = $extval(CURLoption, "CURLOPT_URL")

The CURL type is an abstract type in C - we don’t care what it is actually composed of. A CURL object is always refered to via a pointer. This is modeled in ATS with:

absviewtype CURLptr (l:addr) // CURL*

Think of ‘CURLptr’ as being ‘a reference to a CURL object’. It’s basically a C ‘CURL*’. The definition above states that a CURLptr is an object of type ‘CURLptr’ located at a memory address ‘l’.

The C definition of curl_easy_init looks like:

CURL *curl_easy_init(void);

The ATS definition is:

extern fun curl_easy_init {l:addr} () : CURLptr l = "#curl_easy_init"

The ‘extern’ at the beginning and the ‘#curl_easy_init’ at the end means that the implementation of this function is a C function called ‘curl_easy_init’ in an external library. The ‘{l:addr}’ following the function name gives the type of the ‘l’ used in the return type - that being a pointer. The return type is ‘CURLptr l’ which I described previously. The function returns a CURLptr object located at pointer address ‘l’.

The C definition of curl_easy_setopt looks like:

CURLcode curl_easy_setopt(CURL *curl, CURLoption option, ...);

It is a C function that takes a variable number of arguments. For this example I’m only using one additional parameter so I cheat and declare it in ATS as accepting one. I’ll cover how to do variable argument functions later. The ATS definition is:

extern fun curl_easy_setopt {l:addr} {p:type}
  (handle: !CURLptr l, option: CURLoption, parameter: p)
  : int = "#curl_easy_setopt"

The function takes three arguments and returns one. The arguments it takes are:

a ‘!CURLptr l’. As described before this is a CURLptr object located at address ‘l’. The ’!’ means curl_easy_setopt does not consume the object - we can continue to use it later. The ‘l’ is declared to be of type ‘addr’ earlier in the definition using {l:addr}.
a ‘CURLoption’
a parameter value of type ‘p’. This is of type ‘type’ (from the {p:type} earlier in the definition). This is the type of all ATS objects that fit in a machine word (pointers, integers, etc).

The remaining two CURL functions used in the example are variants of these. The main difference is in the ATS definition of ‘curl_easy_cleanup’:

extern fun curl_easy_cleanup {l:addr}
  (handle: CURLptr l)
  : void = "#curl_easy_cleanup"

Here the ‘handle’ parameter is a ‘CURLptr l’. There is no ’!’ before it. This means the function consumes the argument and it cannot be used after this call. The type ‘CURLptr l’ is a linear type. It must be destroyed at some point in the program and once destroyed it cannot be re-used.

By defining ‘curl_easy_cleanup’ in this way we enforce a contract which states that the programmer must call this function if they have a live ‘CURLptr l’ object. You can see this in practice if you remove the cleanup call and try to compile. A compile error will result. A compile error will also occur if you try to use the curl handle after the cleanup call.

Because of the usage of the linear type we’re already safe from one class of common programmer error. That of not calling cleanup functions for allocated objects.

Handling NULL pointers

There is a bug lurking in this version (and the C program). ’curl_easy_init’ can return NULL and if it does you should not call any other curl functions. Using ATS we can define things in such a way that it is a requirement to check for NULL to successfully compile.

A modified version of the example with the changes to support this is in simple2.dats (pretty-printed version).

In this version we expand on the type CURLptr by adding two typedefs. One for pointers which can be NULL (CURLptr0) and one for pointers which cannot be NULL (CURLptr1):

absviewtype CURLptr (l:addr) // CURL*
viewtypedef CURLptr0 = [l:addr | l >= null] CURLptr l
viewtypedef CURLptr1 = [l:addr | l >  null] CURLptr l

The definitions of the Curl functions that are used are changed to use these types in the appropriate places:

extern fun curl_easy_init
  ()
  : CURLptr0 = "#curl_easy_init"
extern fun curl_easy_setopt {p:type}
  (handle: !CURLptr1, option: CURLoption, parameter: p)
  : int = "#curl_easy_setopt"
extern fun curl_easy_perform
  ( handle: !CURLptr1)
  : int = "#curl_easy_perform"
extern fun curl_easy_cleanup
  (handle: CURLptr1)
  : void = "#curl_easy_cleanup"

‘curl_easy_init’ is changed to return a CURLptr0 since it can return NULL. The other functions use CURLptr1 to signify they must not be NULL. Now if we keep the ‘main’ function as before without checking the result of ‘curl_easy_init’ we get a compile time error saying the type of the ‘curl’ variable does not match that required by ‘curl_easy_opt’.

The fix is to add a check for NULL:

implement main() = let
  val curl = curl_easy_init();
  val () = assert_errmsg(CURLptr_isnot_null curl, "curl_easy_init failed");
  val res = curl_easy_setopt(curl, CURLOPT_URL, "www.bluishcoder.co.nz");
  val res = curl_easy_perform(curl);
  val ()  = curl_easy_cleanup(curl);
in
  ()
end;

The compiler knows that after the assert check that ‘curl’ must not be NULL. This allows the remaining parts of the program to typecheck.

Enforce checking of return values

Another common error is that of not checking the return values of C api calls for errors. This proved to be an issue in the first versions of the Firefox Ogg video backend where return values were not checked in some of the third party libraries we were using. Types can be defined in ATS to give compile time errors if return values are not checked for errors.

‘curl_easy_setopt’ and ‘curl_easy_perform’ both return a value indicating success or failure. A non-zero value indicates an error.

See simple3.dats (pretty-printed version) for the changes required to enforce checking of the values. The definition of ‘curl_easy_setopt’ has changed to:

extern fun curl_easy_setopt {p:type}
  (handle: !CURLptr1 >> opt(CURLptr1, err == 0),
   option: CURLoption, parameter: p)
  : #[err:int] int err = "#curl_easy_setopt"

The same change was made to ‘curl_easy_perform’. The parameter ‘handle’ has changed to have a type of ‘!CURLptr1 » opt(CURLptr1, err == 0)’. The type definition before the ’»’ is what the function accepts as input. After the ’»’ is the type of the parameter after the function returns.

This says that after the function returns the type of ‘handle’ is a ‘CURLptr1’ if ‘err’ is zero. ‘err’ is defined later in the definition as an integer. This forces calling code to check the return value to see if it is zero. That code can then continue to use the CURLptr1 type by extracting it from the ‘opt’. This is the adjusted ‘main’ function showing how this works:

implement main() = let
  val curl = curl_easy_init();
  val () = assert_errmsg(CURLptr_isnot_null curl, "curl_easy_init failed");
  val res = curl_easy_setopt(curl, CURLOPT_URL, "www.bluishcoder.co.nz");
  val () = assert_errmsg(res = 0, "curl_easy_setopt failed");
  prval () = opt_unsome(curl);
  val res = curl_easy_perform(curl);
  val () = assert_errmsg(res = 0, "curl_easy_perform failed");
  prval () = opt_unsome(curl);
  val ()  = curl_easy_cleanup(curl);
in
 ()
end;

Notice the assertion check for the return value of the functions. This is followed by an ‘opt_unsome’ call to ‘un-opt’ the type and continue using it as a ‘CURLptr1’. If either the assert check, or the ‘opt_unsome’ is commented out the code won’t compile. If the assert check is done for a value other than zero it won’t compile. The code could also check the result using an ‘if’ statement - I use assert here for brevity.

Building Firefox with WebM support

2010-05-20T21:20:00+12:00

All the patches to get Firefox built with WebM support have now been attached to their relevant bugs. These patches aren’t ready for review yet but can be used to get a build that has the functionality of the special Firefox WebM builds.

To build use Mozilla Central as the base with the patches from the following bugs applied:

Bug 566241 - Video playback stops intermittently - This has now landed in mozilla-central.
Bug 566498 - Move generic video duration computing into Ogg decoder - This has now landed in mozilla-central.
Bug 566501 - Fix seeks needed only by Ogg backend - This has now landed in mozilla-central.
Bug 567056 - Rename some variables - This has now landed in mozilla-central.
Bug 566246 - nestegg, the webm container parser we use
Bug 566247 - libvpx, The VP8 SDK and associated build changes
Bug 566245 - nsWebMDecoder implementation

Hopefully these will be finished, reviewed and landed in short order.

VP8, WebM and Firefox

2010-05-20T04:40:00+12:00

For all the details on the recent announcing of VP8 video code support in Firefox, the WebM container and playing YouTube videos with Firefox, read Firefox, YouTube and WebM on hacks.mozilla.org.

Fennec on Android

2010-04-28T16:10:00+12:00

A pre-alpha build of Fennec, the mobile version of Firefox, is available for Android devices.

Over the last few months, we’ve made some great progress on bringing Firefox to Android. Michael Wu, Brad Lassey, Alex Pakhotin and I have been focusing on getting a build ready that’s usable by a broader set of people, and we’re now ready to get that build out there.

John Gruber comments:

It’ll be interesting to see if Gecko can be turned into a worthy mobile competitor to WebKit.

Gecko already runs in a released mobile device. The Nokia N900 uses MicroB as its built in browser and that is based on Gecko. The web browser functionality of MicroB matches, if not exeeds, that of many WebKit based browsers on other mobiles which I think shows that Gecko can be a ‘worthy competitor’.

HTML 5 Video Seeking and Redirects

2010-04-27T14:00:00+12:00

Bug 560806 has landed on trunk. Seeking and duration works for videos hosted on some servers where it didn’t before.

Firefox’s implementation of HTML 5 video allows seeking in unbuffered portions of the video as long as the server that hosts the video supports byte range requests.

Firefox needs to know in advance if it can seek the video so it can correctly display the ‘progress’ bar. It also needs this information so it can compute the duration (which requires seeking into the video to find the end time).

Most servers that support byte range requests send an Accept-Ranges header. If we see this on the initial request we know the server supports seeking.

Some servers support byte range requests but do not send this header. Amazon S3 is an example of a server that does this. To detect if these servers support seeking we make our inital request for the video include a ‘Range’ header that askes for the bytes from zero to the end of the file. Servers like S3 that support byte ranges will respond with a ‘206’ response code meaning a partial request is returned. This is sent even though the byte range we asked for is actually for the entire file. Once we see the ‘206’ we mark the resource as seekable.

A very few servers don’t send an Accept-Ranges header and recognize our byte range request as actually being for the entire file and return a ‘200’ response code. Unfortunately we don’t recognize resources hosted on these servers as seekable. This means Ogg media on these servers don’t have a duration since we can’t seek to find it.

Another issue we came across, and exists in Firefox 3.5 and Firefox 3.6, is detecting seeking when the initial request for the media resource results in a ‘3XX’ response code. This is a redirect to another URL. Current Firefox versions (and possibly other browsers) do not pass on the value of headers to the redirected URL. This means the ‘Range’ header is removed and we don’t get a ‘206’ response code on the redirected request. In servers like Amazon S3 that don’t send ‘Accept-Ranges’ this means we don’t support seeking.

This is why videos hosted on Amazon S3 and similar serving platforms that arrive there via a redirect don’t get a duration, show progress or allow seeking. It’s a fairly common practice to hide the fact that Amazon S3 is being used by having a redirect from a URL on the web site domain that redirects to the S3 location which results in this problem.

The following are two URL’s that show the problem. The first is a direct link to the video. The second redirects to the final URL. In the first you can seek and see a duration of ‘1:00’. In the second you can’t seek and no duration is shown:

This is Bug 560806 and has been fixed and is now on Firefox trunk and in the nightly builds. The fix was to explicitly pass the ‘Range’ header along to the final URL. This workaround exists in a few other areas in Firefox and Bug 311599 discusses a similar issue for plugins.

If your own videos are not seekable or showing a duration then you may be facing a similar issue and it might be worthwhile trying a nightly build and seeing if that fixes it. Other things to look at are the server configuration to make sure that byte range requests are supported and the Accept-Ranges header is being sent. If you use Amazon S3 you might want to try and get them to support Accept-Ranges. This forum post discusses the issue.

Changes to Firefox YCbCr Color Conversion Code

2010-03-12T13:30:00+13:00

I posted previously about colour conversion libraries. To replace the liboggplay conversion routines we decided to try the Chromium libraries.

Bug 551277 adds the Chromium code and integrates it with the new video layer painting code. The Chromium code uses a C fallback if MMX is not available but this is done at compile time. I modified it do runtime detection using Mozilla’s SSE.h and moved the generic C routine from the Linux specific implementation to be used on all platforms that don’t have MMX. Some other minor modifications were done (function name, namspacing).

I added an ‘update.sh’ script that can be used to update against the latest Chromium code. Given the path to the Chromium source it applies a patch for the changes I made.

Bug 551378 adds YCbCr 4:4:4 conversion (YV24). The Chromium code supported 4:2:0 and 4:2:2 (YV12 and YV16 respectively). Theora also allows 4:4:4 per spec. There aren’t many 4:4:4 Theora videos out in the wild and the fix in this bug only adds support in the slower C implementation. I detect if the video is a 4:4:4 video and fallback to the C code.

At some point optimized MMX routines should be done for this. I’ve raised Bug 551844 for this. This would probably make a good first bug for anyone who wants something small and self contained to work on. It would be good to contribute this back to Chromium when done too.

Xiph has a test suite with example videos in the different formats.

HTML 5 Video Element for Internet Explorer

2010-02-22T13:54:00+13:00

Cristian Adam has been working on a plugin for Internet Explorer that implements the HTML 5 video element.

He has released a new version of the Xiph Ogg Codecs with a technical preview of the <video> element for IE. A screencast of it working is available at Cristian’s website.

Comparing Colour Space Conversion Libraries

2010-02-19T17:00:00+13:00

The libtheora api gives YCbCr data as the result of a decoded frame. The current method of displaying data through Firefox requires it to be converted to RGBA. The conversion of YCbCr to RGBA turns out to be a bottleneck.

In current Firefox builds we use the conversion routines provided by liboggplay. The Ogg backend is being modified to reduce the third party library usage (Started by me and continued now by Chris Pearce) and the liboggplay usage will go away. I looked at some of the colour space conversion routines available to get an indication of their relative performance.

I tested the following colour space conversion routines:

Using integer math and lookup tables.
A basic C version using floating point math.
liboggplay. BSD license.
FrameWave (also known as the AMD Performance Library). Apache License 2.0.
Intel Integrated Performance Primitives. Commercial license.
Moonlight’s colour space conversion routines. I tested the C and MMX implementation from Moonlight. LGPL2 license.
The conversion routines from Chromium. BSD license.
libswscale. LGPL license.

For testing I modified plogg to decode a theora file and time the colour space conversion portion of the process. I have a series of command line switches to pick the implementation of the colour space conversion. I used a movie trailer Theora file I had handy (480x260) and these were the results:

Implementation	Total Time	Frame Time	Relative
FrameWave	1.00752	0.00033584	0.65
Chromium	1.02053	0.00034017	0.66
Intel IPP (optimized)	1.32477	0.00044159	0.85
Moonlight MMX	1.35934	0.00045311	0.87
liboggplay	1.55765	0.00051921	1.00
libswscale	2.68307	0.00089435	1.72
Intel IPP (default)	4.21107	0.00140369	2.70
Integer C	4.81204	0.00160401	3.09
Moonlight C	6.80419	0.00226806	4.37
Floating Point C	11.16770	0.00372257	7.17

The ‘Total Time’ is the time spent in the conversion code for every frame in the file (3,000 frames). ‘Frame Time’ is the average time per frame. ‘Relative’ is the time taken relative to the liboggplay implementation. Since this is what we are currently using this makes it easy to see what sort of improvement we could get by switching. The testing was done on a 1.83 GHz Core Duo laptop running Linux.

From the results it seems that FrameWave and Chromium are the fastest and very close in performance. The license for the Chromium colour space conversion code is probably a bit better fit for our usage though and it’s smaller and easier to integrate if we were to choose to use it. The ‘default’ Intel IPP version uses the non-processor specific implementation whereas the ‘optimized’ uses routines optimized for the particular processor on the machine I used for testing.

I’m interested in any comments on the libraries listed and recommendations for other libraries that I could try. I’ll put the source for the test program on github shortly and update this post with the link. Results from different machines would be useful.

On2 Stockholders Approve Google Merger

2010-02-18T15:00:00+13:00

Last year Google announced that they were going to acquire On2 Technologies. This is the company that produced (and open sourced) the VP3 video codec that Theora is based on. Google wrote on their blog:

Although we’re not in a position to discuss specific product plans until after the deal closes, we are committed to innovation in video quality on the web, and we believe that On2 Technologies’ team and technology will help us further that goal.

The deal has been delayed awaiting approval of the majority of the On2 stockholders to agree on the acquisition. According to the On2 website this happened today:

On2 Technologies, Inc. (NYSE Amex: ONT) today announced that its stockholders approved the merger of On2 with a wholly owned subsidiary of Google Inc. at its Reconvened Special Meeting held earlier today.

On2 stockholders holding in excess of a majority of the outstanding shares of On2 Common Stock voted in favor of the merger proposal.

It’ll be interesting to see what comes from this. On2 have other video codecs. (VP6 used in Flash on some video sites, VP7 and their latest codec VP8.