Skip navigation.

Log in | Sign up

Opera Core Concerns

Official blog for Core developers at Opera

(re-)Introducing <video>

, , , , , ,

Today the desktop team released a new years pre-alpha of Opera 10.50 which includes the first public preview of our new HTML5 <video> implementation. It was Opera that first proposed <video> and made the first proof-of-concept build way back in 2007. In this post I will talk about what we've done since then, the decisions we've made and where we hope to go next.

Codecs

From the very beginning, one of the most important questions has been which audio and video formats should be used with HTML5. The issue has been debated heavily in the web standards community, but no consensus has been reached yet. We believe that the web platform must be built on open standards and will therefore continue to support the Ogg formats: the Vorbis audio codec and the Theora video codec. These, in addition to plain WAVE PCM audio, are our "core codecs" which we will support on all desktop platforms.

Features

While the core codec support remains the same, almost everything else has changed. The specification has evolved a lot since <video> was first added, and we try to follow the spec to the letter. This is a non-exhaustive list of what we consider more or less done:

  • Native controls (the controls attribute)
  • Scripted controls (e.g. using play(), .currentTime and ontimeupdate)
  • Poster image (the poster attribute)
  • Painting <video> frames on <canvas>
  • Selecting source depending on codec and environment (e.g. <source src="video-720p.ogv" type="video/ogg" media="(min-device-height: 720px)">)
  • Scripted codec detection (e.g. canPlayType('video/ogg; codecs=vorbis,theora'))

HTML5 <video> section then and now (rotated 90°; scale 1:73)

As this is a pre-alpha, there are also some parts that are not finished. Most importantly, we don't support seeking at all yet. As a side effect we don't have the duration of files until we have played through to the end, so the seekbar isn't very useful right now. This is at the top of our list of things to fix before a final release. We are also working on improving bandwidth management so that we will be able to honor (the absence of) the autobuffer attribute.

Implementation

In our original <video> demo, we were using the libogg, libvorbis and libtheora libraries. For this release those libraries are still included, but we have adopted the GStreamer media framework as an extra layer between the browser core and the raw decoding. Among other things, this allows processing to take place in a separate thread, which has improved responsiveness and audio quality.

For platforms where GStreamer is natively available, we are simply using the system-installed version. Thus, if you are using Linux or FreeBSD, make sure to install at least the GStreamer "base" and "good" plugins, otherwise <video> won't work at all. (Update: on Debian/Ubuntu the package names are gstreamer0.10-plugins-base and gstreamer0.10-plugins-good.) Having done this, Opera will be able to play anything that GStreamer can handle, which fortunately includes our core codecs. We also try to detect a number of GStreamer plugins so that we can give truthful answers to scripts that ask e.g. canPlayType('audio/flac') or canPlayType('video/x-msvideo'). We hope you have fun playing with this, but stick to Ogg for anything serious that should cross-platform and cross-browser.

On Windows we have made a minimal GStreamer configuration which keeps only the features necessary to decode the above mentioned core codecs. As required by the GStreamer license (LGPL), our modified source code is available. Unfortunately we don't have a Mac version of GStreamer ready for today's release, so <video> is not available on Mac just yet.

Many thanks to the developers of the GStreamer and Xiph projects, for excellent software and the occasional bit of advice on IRC.

Native controls in Opera 10.50 pre-alpha

The future

In the coming year, we hope that adoption of <video> continues and that we will see exciting new applications made possible by having audio and video as integral parts of the web platform. We will work hard to help realize this potential both by improving our own implementation and by contributing to the development of HTML and other specifications. A few things we think are important right now:

  • Fullscreen video playback is a must-have feature.
  • Subtitles/captions (static as well as scripted).
  • Media Fragment URIs will enable users to link to a specific point in time of a video and more.
  • More CSS to make <video> and the whole web even more awesome.
  • Let us not forget SVG's own <audio> and <video> elements.
  • Security, performance and stability (this is always important).

(No promises on when any specific feature will be done.)

Demos

If you have read this far, you deserve to see some demos!

With this I wish everyone a happy new year, may there be an abundance of <video> for all!

Carakan Revisited

Comments

lucideer 31. December 2009, 23:24

Originally posted by Philip Jägenstedt:

SVG's own <audio> and <video> elements


:eyes:

endless love 31. December 2009, 23:29

:heart:Happy New Year 2010 :hat:
......oooO.............
......(....)..............
......)../....Oooo.....
.....(_/.....(..../......
..............)../........
.............(_/.........
....oooO...............
.....(....)...............
......)../....Oooo.....
.....(_/.....(....)......
...............)../.......
..............(_/........
....oooO...............
.....(....)...............
......)../....Oooo.....
.....(_/.....(....)......
...............)../........
..............(_/.........

Daniel James Hendrycks 31. December 2009, 23:40

:hat: :drunk:

Tamil 1. January 2010, 00:33

:up:

Witold Baryluk 1. January 2010, 01:01

Hmm, i have 100% cpu usage on the "9elements HTML5 Canvas and Audio Experiment" and not very smooth, (but it looks pretty simple). But this is because canvas element is very very big. After chaning size of opera window to be smaller, it runs extreamly smooth (but still 100% cpu usage).

BTW. Canvas element on this page isn't resized when i change size of Opera window. It is needed to reload page.

I also have many "mmap() failed: Cannot allocate memory" in the console of "Support Tester" page.

I also think that sound isn't mixed when two or more video is on the one (or multiple) pages.

Anyway, grate work. Many examples works very smooth. Thanks!

When i play the same OGG directly in the mplayer it is slightly smoother, with better sound synchronization. (even with preloaded files into the opera cache, and proxy cache, 10MB/s, or from disk)

Edit: tested it using mplayer side-by-side with opera. Opera have some problems. Image freezes sometimes, then goes quickly to synchronize. Even when it plays continuously it have only about 10 FPS. Mplayer doesn't drop anything (25 fps). CPU usage about 25% (with both opera and mplayer playing!). Audio is playing perfectly.

I also tested it sidebyside with Totem (which uses gstreamer). Totem is much smoother than opera (again 25% CPU with both opera and totem). But totem also have from time to time some problems similar to opera (like freezing and then quickly playing to synchronoize to audio and time). But just after synchronization it runs with full framerate (25 fps), just like mplayer.

Patrick H. Lauke 1. January 2010, 01:50

good start into the new year...can't wait to see this evolve. for some reason the video and canvas performance thing did nothing on my machine (win/3186) though.

d4rkn1ght 1. January 2010, 03:15

:yes:

Abhinav 1. January 2010, 03:55

Witold Baryluk 1. January 2010, 07:01

Just wanted to add something more. It is about quick start. I doesn't see any buffering. Just loaded page, i hit "Play" button, and few miliseconds later I see moving video and hear audio. Amazing! In most flash player i would wait about 10 seconds. Here in Opera I doesn't wait even 1 second.

Charles Schloss 1. January 2010, 07:04

I would say http://9elements.com/io/projects/html5/canvas/is one of my favorites as it also follows the mouse

Pallab De 1. January 2010, 07:06

Cool. It took so long that I almost forgot about it!

And also congrats on getting engaged.

Eion Robb 1. January 2010, 09:00

If we have a more complete installation of GStreamer already installed on OSX/Windows (say, because we're using GStreamer in Pidgin for voice/video), will the demos that need MPEG-4 work, or does Opera use it's own minimal-GStreamer only?

Philip Jägenstedt 1. January 2010, 09:18

Originally posted by movax:

I also have many "mmap() failed: Cannot allocate memory" in the console of "Support Tester" page.



Which of the samples were you playing when this happened? Was it MP3? I think this happens sometimes with FFMPEG decoders... will add this to list of bugs.

Originally posted by movax:

I also think that sound isn't mixed when two or more video is on the one (or multiple) pages.



Internally we have one pipeline with one autoaudiosink element per video, which is similar to just starting two instances of totem. It's quite possible that this doesn't always work perfectly depending on the audio output method (OSS/ALSA/PulseAudio/etc) actually used. Are you just hearing one of the videos or is the sound distorted somehow?

Originally posted by movax:

When i play the same OGG directly in the mplayer it is slightly smoother, with better sound synchronization. (even with preloaded files into the opera cache, and proxy cache, 10MB/s, or from disk)



It is a known issue that the video is slightly delayed. Unless Opera is using 100% CPU there's no reason it shouldn't play at 25 fps though, so that's clearly a bug. This is likely to behave slightly different on different platforms too, but again I'll add this as a new bug.

Thanks movax!

malsumis 1. January 2010, 09:31

So, correct me if I'm wrong on this one, but as I understand it, right now anything other than vorbis/theora will not play on windows?

Why just don't use windows directShow filters that are installed, like any other media player would do?

Philip Jägenstedt 1. January 2010, 09:45

Originally posted by BigBrownChunx:

If we have a more complete installation of GStreamer already installed on OSX/Windows (say, because we're using GStreamer in Pidgin for voice/video), will the demos that need MPEG-4 work, or does Opera use it's own minimal-GStreamer only?



Since there is no standard distribution of GStreamer for Windows, we can only use our own. But regardless of platform, for any plugins to work they must be linked with the actual GStreamer library being used, we can't "mix and match" since our minimal GStreamer is missing quite a lot of things.

In short, expect Mac to work the same as Windows.

In using GStreamer, it is not our intention to support every format under the sun and doing that wouldn't be very useful since very few of them work cross-platform and cross-browser. The ability to use arbitrary codecs on Unix is a kind of "accident", a little something to play with if you will.

Philip Jägenstedt 1. January 2010, 10:06

Originally posted by malsumis:

Why just don't use windows directShow filters that are installed, like any other media player would do?



The main problem with this approach is that many DirectShow filters are written in such a way that they can only read from a local file, since the DirectShow framework makes this a lot easier than the "right way" of streaming input from a upstream filter. The net result is that we can't play a partially downloaded resource, which is rather useless.

What is left is MPEG-1, some old AVI formats, etc, which doesn't seem worthwhile supporting, especially as it would be Windows-only.

malsumis 1. January 2010, 10:37

Originally posted by foolip:

The main problem with this approach is that many DirectShow filters are written in such a way that they can only read from a local file, since the DirectShow framework makes this a lot easier than the "right way" of streaming input from a upstream filter. The net result is that we can't play a partially downloaded resource, which is rather useless.

What is left is MPEG-1, some old AVI formats, etc, which doesn't seem worthwhile supporting, especially as it would be Windows-only.


Sucks. So, are there no plans to implement mpeg-4? Apparently Firefox only supports ogg formats, but safari and chrome can deal with mpeg-4 also, and safari with it's proprietary qt format.

Google has youtube, so they won't make it any easier to adopt ogg as the standard. Maybe it would be wise to support mpeg-4 in some form?

Johan Borg 1. January 2010, 12:11

@malsumis: MPEG-4 is a good standard, but it's not open in the same sense as HTML, you have to pay to use it, not only if you include it in your software, but even to distribute content.

I can only speculate, but I don't think Google would offer more than $100 million for On2, the company that enabled the Theora project, if MPEG-4 was the solution to all their video needs.

malsumis 1. January 2010, 13:44

Originally posted by borg:

@malsumis: MPEG-4 is a good standard, but it's not open in the same sense as HTML, you have to pay to use it, not only if you include it in your software, but even to distribute content.

I can only speculate, but I don't think Google would offer more than $100 million for On2, the company that enabled the Theora project, if MPEG-4 was the solution to all their video needs.


I knew that mpeg-4 needs to be licensed. Didn't know that google bought on2. Interesting. The important thing is what will google
do with on2? Will it go the Microsoft route? Youtube supports mpeg-4 only right now, if they were serious, wouldn't they make it ogg-friendly? Dailymotion has some stupid mozilla agreement and only let's you demo their html5 service with firefox beta something.

And the rest of the web is basically stuck on flash, because right now basically no one cares about html5, even though you can gracefully degrade to flash if you're just a little bit on the smart side.

lucideer 1. January 2010, 14:04

Originally posted by malsumis:

Youtube supports mpeg-4 only right now, if they were serious, wouldn't they make it ogg-friendly?


The impression I got was that they opted for mpeg-4 reluctantly and wanted to go with ogg, but "needed" something "better", so are currently seeking alternatives (hence On2). That's largely speculation based on bits and pieces I've read here and there.

Originally posted by malsumis:

Dailymotion has some stupid mozilla agreement


I think it's less of an agreement and more that the project has stagnated. They released it, announced it saying "we will add support for other browsers soon" (paraphrased) and then left it to die. Or so it seems anyway.

Philip Jägenstedt 1. January 2010, 15:02

Extra points to anyone who can identify the movie in the screenshot :smile:

Serav 1. January 2010, 17:45

Is there no chance of adding the components of gstreamer that allow mpeg-4 playback to the windows install? Or would that mean having to pay license fees? And if so how do open source players like VLC support mpeg-4?

Supporting both would allow compatibility with a greater majority of video now using the <video> tag, and mpeg-4 is much more popular as a codec than ogg theora. This means most people will have copies of their videos in mpeg-4 format already, either for portable media players or recorded on mobiles or cameras. Having it universally supported in some form would allow less transcoding and hassle when inserting the video into blogs and other personal sites.

Daniel James Hendrycks 1. January 2010, 18:22

Originally posted by Philip Jägenstedt:

Extra points to anyone who can identify the movie in the screenshot


Malan from Disney?

Daniel James Hendrycks 1. January 2010, 18:24

Originally posted by Philip Jägenstedt:

no reason it shouldn't play at 25 fps though


When Vega adds HW acceleration will the FPS increase?

Philip Jägenstedt 1. January 2010, 19:23

Originally posted by Serav:

Is there no chance of adding the components of gstreamer that allow mpeg-4 playback to the windows install? Or would that mean having to pay license fees? And if so how do open source players like VLC support mpeg-4?



I can't comment on the legal situation or other projects, but Opera can't distribute MPEG-4 decoders at this time.

Philip Jägenstedt 1. January 2010, 19:25

Originally posted by DanielHendrycks:

Malan from Disney?



No, but there's a Chinese connection.

Originally posted by DanielHendrycks:

When Vega adds HW acceleration will that FPS increase?



It doesn't look like it's a performance problem right now, it's rather a timing problem. Hardware acceleration may help reduce CPU usage though.

theoddbod 1. January 2010, 20:10

I hope there'll also be a per-site option to disable the video or block certain ones.

Serav 2. January 2010, 00:14

Originally posted by foolip:

I can't comment on the legal situation or other projects, but Opera can't distribute MPEG-4 decoders at this time.



Fair enough. I was just curious so thanks for the answer.

Cutting Spoon 2. January 2010, 01:50

Timing issues remind me of the problem with MPlayer and MKV in Damn Small Linux; where technically MKV subs were supported, but nobody had done the timing work for DSL's custom core.

Tacking controls on to the face of VLC's existing browser plugin wrapper, would in fact be rather nice. I already open videos in Opera windows, VLC is cross-platform last time I checked. I suppose there are always legal hurdles, though.

Barraco Mármol Jerónimo 2. January 2010, 02:19

svg and video tags are really good ideas :smile: keep it up.

Witold Baryluk 2. January 2010, 10:44

Originally posted by lucideer:

I think it's less of an agreement and more that the project has stagnated. They released it, announced it saying "we will add support for other browsers soon" (paraphrased) and then left it to die. Or so it seems anyway.


Someone should learn them how to properly use video tag. Why the hell they are doing any hardcoded browser detection!? Just embeded video element directly, browsers will decide for they own if they are supporting it or not, and if not provide fallback content inside of the video tag body. Eh.

http://camendesign.com/code/video_for_everybody Example how it should be done.


NEOAethyr 2. January 2010, 15:09

@Philip Jägenstedt

Are you using ffmpeg/ffdshow for the theora video decoding?
http://sourceforge.net/projects/ffdshow-tryout/

If so then you should have no prob any video/audio pretty much.
I assume it depends on what this GStreamer thing allows in ogm files.

Halli supposedly shares his splitter code with those who ask:
http://haali.su/mkv/
That does ogm, avi, mkv and mp4.

Then with flash video's, you could rip the code from media player classic (probably from mpc-hc) for the stand alone flv splitter.
I've only seen that outputted to ax format, I dn if linux can deal with that or not.

If you could get flv files working.
A script that gets the direct link off youtube and puts it in a video tag.
You could disable the flash pugin and still play it back :D.
That would rock.

Anyways whatever you decide to do.
Good luck to ya.
It's better then flash anyways.

Philip Jägenstedt 2. January 2010, 17:32

Originally posted by NEOAethyr:

Are you using ffmpeg/ffdshow for the theora video decoding?



No, we use GStreamer, which is a media framework just like DirectShow. We don't mix the two currently, because there doesn't seem to be any benefit in doing so.

Pharaoh_Atem 4. January 2010, 07:11

I think it was actually possible to use DirectShow codecs in GStreamer, but meh... I haven't really played with GStreamer on Windows in quite some time. Nevertheless, it sounds kind of bad that if a system version of GStreamer is installed, and in the PATH, that it wouldn't be used on Windows or Mac, though it is on Linux...

It is rather trivial to add in detection logic to the installer to find GStreamer and configure Opera to use it, at least, if it is was a script based installer. And of course, if no GStreamer installation is detected, then Opera's own minimal one is installed and used instead.

Tim Lawrence 4. January 2010, 08:26

How about OSS4 support? I find OSS much more useful than Gstreamer.

Philip Jägenstedt 4. January 2010, 09:18

Originally posted by panzeroceania:

How about OSS4 support? I find OSS much more useful than Gstreamer.



OSS is a sound system while GStreamer is a media framework (like DirectSound and DirectShow). GStreamer may already be using OSS for audio output, depending on your system setup.

Pharaoh_Atem 4. January 2010, 11:16

What?? No gitweb to access and nicely browse the source tree with?

Daniel James Hendrycks 4. January 2010, 16:28

Do you plan for FLAC or are you waiting for a losseless ogg video codec to go hand in hand with FLAC?

Philip Jägenstedt 4. January 2010, 19:18

Sorry about that Pharaoh_Atem, we might add that. Our changes aren't very interesting, but for now you have to clone the repository to have a look.

Philip Jägenstedt 4. January 2010, 19:23

Originally posted by DanielHendrycks:

Do you plan for FLAC or are you waiting for a losseless ogg video codec to go hand in hand with FLAC?



FLAC (and lots of other codecs) are properly detected and used on Unix (flacdec is in gst-plugins-good) if you want to play with it. As FLAC isn't terribly useful on the web due to its size, we probably won't support it on other platforms (keep in mind that every added codec adds to the download size of Opera).

The lossless Ogg video you talk about actually already exists -- Dirac has a lossless mode. However, lossless video seems even less useful than lossless audio on the wider web :smile:

Daniel James Hendrycks 4. January 2010, 19:46

Ok, maybe in years we will see it when we can transfer more information faster.

Write a comment

You must be logged in to write a comment. If you're not a registered member, please sign up.