Refactoring of Firefox HTML video element patch
I've making some changes to the patch implementing the HTML <video> element for Firefox.
Previously the patch attached to the bug contained the implementation of the video element and a Theora backend implemented using liboggplay.
This required pulling in a large amount of third party library code making it more difficult to get reviews for the patch and to get feedback on the changes required for the video element itself. To hopefully fix this I'm changing bug 382267 to only include the video element implementation with no video decoders.
Applying the patch from that bug to the Firefox trunk source and building with the --enable-video option will provide support for <video> but all videos will fail to load due to not having a supported decoder. This is obviously not very useful but will hopefully make the patch easier to get reviewed and landed.
I've added bug 422538 which will contain support for decoding Theora videos. This is the code taken out of the original bug. Applying the patch from this bug on top of the video element patch and building with --enable-ogg will give <video> element support that can play Theora videos.
As part of the refactoring I've made it easier for different video decoder backends to be implemented. There is an abstract class with pure virtual methods that can be inherited from. I have a 'work in progress' implementation of a GStreamer backend that I'll write about later and attach a patch to bug 422540. This backend allows playing any video format supported by GStreamer. I've tested it with the Fluendo H.264 decoder and it plays fine. Keep an eye on the bug if you're interested in this.
An advantage of this refactoring is that embedders of Gecko can choose a video backend that suits their purposes (for legal or other reasons they may choose not to have support for a particular format), or implement their own.
Currently the interface for implementing video decoder backends is a C++ abstract class. I'm considering using XPCOM so that implementations can be built and dynamically loaded. This could allow extensions to be written containing video backends that will work with the <video> element. One of the questions asked of me at the W3C video on the web workshop was whether third party providers of DRM enabled codecs could leverage the <video> element. This could provide a pathway to doing that.
The git repository will be restructured so that the 'master' branch contains the <video> element support only. The 'ogg' and 'gstreamer' branches will contain the implementations of the backends.
Categories: firefox, ogg, gstreamer
Previously the patch attached to the bug contained the implementation of the video element and a Theora backend implemented using liboggplay.
This required pulling in a large amount of third party library code making it more difficult to get reviews for the patch and to get feedback on the changes required for the video element itself. To hopefully fix this I'm changing bug 382267 to only include the video element implementation with no video decoders.
Applying the patch from that bug to the Firefox trunk source and building with the --enable-video option will provide support for <video> but all videos will fail to load due to not having a supported decoder. This is obviously not very useful but will hopefully make the patch easier to get reviewed and landed.
I've added bug 422538 which will contain support for decoding Theora videos. This is the code taken out of the original bug. Applying the patch from this bug on top of the video element patch and building with --enable-ogg will give <video> element support that can play Theora videos.
As part of the refactoring I've made it easier for different video decoder backends to be implemented. There is an abstract class with pure virtual methods that can be inherited from. I have a 'work in progress' implementation of a GStreamer backend that I'll write about later and attach a patch to bug 422540. This backend allows playing any video format supported by GStreamer. I've tested it with the Fluendo H.264 decoder and it plays fine. Keep an eye on the bug if you're interested in this.
An advantage of this refactoring is that embedders of Gecko can choose a video backend that suits their purposes (for legal or other reasons they may choose not to have support for a particular format), or implement their own.
Currently the interface for implementing video decoder backends is a C++ abstract class. I'm considering using XPCOM so that implementations can be built and dynamically loaded. This could allow extensions to be written containing video backends that will work with the <video> element. One of the questions asked of me at the W3C video on the web workshop was whether third party providers of DRM enabled codecs could leverage the <video> element. This could provide a pathway to doing that.
The git repository will be restructured so that the 'master' branch contains the <video> element support only. The 'ogg' and 'gstreamer' branches will contain the implementations of the backends.
Categories: firefox, ogg, gstreamer
Labels: mozilla
posted by Chris Double at 12:14 PM
7 Comments:
Great to see you are making progress. These sound like great ideas that will make native video in mozilla more flexible and useful for everyone.
In gecko 1.8 we had pluggable implementations of SVG renderers. We supported cairo, GDI+ and/or libart backends. Perhaps a perusal of that code would help you.
Thanks for the pointer Robert, I'll take a look.
"This could allow extensions to be written containing video backends that will work with the <video> element. One of the questions asked of me at the W3C video on the web workshop was whether third party providers of DRM enabled codecs could leverage the <video> element. This could provide a pathway to doing that."
The problem with that idea is that new backends will inevitably require platform-specific compiled code - they won't be writing their codecs in JS. And what are the chances that, when you visit such a site on Linux, they'll have provided a compiled version for that platform? Isn't there a risk here that we'll enable certain format providers to more easily leverage their platform monopoly?
Can multiple back-ends be used? ie can firefox include baseline theora support with gstreamer, quicktime or MS-video ontop (depending on the platform)?
How would this play-out? would the binaries that mozilla distribute include hooks into quicktime on apple, and ms-video on windows?
It seems like a bad idea not have a free codec supported as the baseline for the video tag.
A codec agnostic video tag will further bifurcate the free vs proprietary platforms. As proprietary OS will chose not to include support free codecs in "their" media platforms, while free systems will be unable to support the popular proprietary technologies. This cripples the "out of the box" free software experience and provides a proprietary wedge into the web platform.
a codec agnostic video tag without a base-line codec seem to be contrary to Mozilla stated goals of an "open web".
The svg stuff is cool but for general usage a codec agnostic approach reduces the video tags potential to something slightly better than existing plugin systems with a javascript library wrapping plugin calls to html5 video api. There is already gstreamer, quicktime and ms-video plugins.
In the short term enabling/supporting propitiatory media back ends may seem like a good idea for improved user experience and video accessibility...
But Ultimately it reduces Mozilla Firefox reach by ensuring the open web platform in which free software thrives is never fully actualized.
gerv, yes that's a risk and something to be considered. On the other hand, the authors of the proprietary codecs (I'm thinking of the 'network television on the web' custom players here) don't generally support Linux anyway.
Part of that reason can be the difficulty of supporting the base program functionality around the codecs (the UI, updates, etc). If the browser can be a suitable vehicle for that, and all that is needed is for them to write a small plugin, then it may make it more likely for them to consider a Linux version. That's what I was hearing at the workshop, but whether it would in fact happen I don't know.
My primary reason for considering the plugin approach was actually to make my job easier to produce backends :-)
mike, I completely agree that a free baseline codec is important and I'm actively involved in discussions on that.
I do see the big advantage of a video and audio element is having a supported free baseline codec that is guaranteed to exist in supporting browsers.
There are other advantages to the video element apart from that though. One issue with plugin based approach (which you point out that there already exist plugins for vlc, etc) is that they don't participate in the HTML rendering pipeline. There is no guarantee of being able to display HTML over the top of the playing video, to manipulate it with transforms, etc. This is not a problem with the video element.
Enabling gstreamer support doesn't just open the door to using proprietary codecs. It enables the use of other free codecs as well such as Dirac.
Multiple backends can be used, and I use the Ogg backend and fallback to the gstreamer one in my own testing. How best to negotiate that fallback, etc is still something I need to look at.
Post a Comment
<< Home