Ogv.js: Difference between revisions

From XiphWiki
Jump to navigation Jump to search
(fix caps in title :D)
(updates; has soft-launched on Wikipedia/Wikimedia Commons)
 
(One intermediate revision by the same user not shown)
Line 1: Line 1:
At least libogg, libvorbis, and libtheora can be built to JavaScript using [http://emscripten.org/ emscripten], and to Flash with [http://adobe-flash.github.io/crossbridge/ Crossbridge], with only slight modifications.
At least libogg, libvorbis, libopus, and libtheora can be built to JavaScript using [http://emscripten.org/ emscripten] pretty straightforwardly.


'''[https://github.com/brion/ogv.js ogv.js]''' is a wrapper around these with basic JavaScript/ActionScript3 interfaces and video/audio playback wrappers semi-compatible with the native HTML5 <video>/<audio> elements.
'''[https://github.com/brion/ogv.js ogv.js]''' is a wrapper around these with basic JavaScript interfaces and video/audio playback wrappers semi-compatible with the native HTML5 <video>/<audio> elements.


Currently the internal interfaces are very much in development. I ([[User:Brion]]) hope to get these in use for playback of audio and video content on Wikipedia and Wikimedia Commons in browsers that don't support Ogg or WebM natively, as our community has rejected the idea of making and serving out MP4 transcodes for those browsers.
Currently (August 2015) in use for audio/video playback in Safari/IE/Edge browsers on Wikipedia and [https://commons.wikimedia.org/wiki/Commons:Media_of_the_day Wikimedia Commons].


Generic decoding, or even encoding, should be possible in the future but needs better APIs to be sane.
Generic decoding, or even encoding, should be possible in the future but needs better APIs to be sane.
Line 9: Line 9:
== Building ==
== Building ==


The build scripts make slight changes to configure scripts which may or may not be necessary, need to dive in in more detail.
See the [https://github.com/brion/ogv.js#readme readme.md] for details of building the full ogv.js package. The standard autotools build systems are used with the emscripten compiler front-end.


There's also a slight fix needed in libtheora: https://github.com/brion/theora/commit/06a0e4acf9c35f4bd31b8788a8a573cb89262333 which is probably safe to upstream (check this!)
You can also [https://github.com/brion/ogv.js/releases download a pre-built release of ogv.js].


== Demo ==
== Demo ==


Playback demo at: https://brionv.com/misc/ogv.js/demo/
Playback demo at: https://brionv.com/misc/ogv.js/demo/
Integration into MediaWiki demo at: https://ogvjs-testing.wmflabs.org/ (defaults to native codecs when available, still a work in progress)


JavaScript video works in all current browsers, audio works if Web Audio API is there (latest Firefox, Safari, Chrome), or with a Flash audio shim for IE.
JavaScript video works in all current browsers, audio works if Web Audio API is there (latest Firefox, Safari, Chrome), or with a Flash audio shim for IE.
Flash video/audio playback works in all current browsers and IE 9; IE 6/7/8 support is planned.


== Notes ==
== Notes ==
Line 28: Line 28:
* libvorbis
* libvorbis
** experiments with using tremor instead of libvorbis showed it produces significantly smaller JS code. Performance, however, seems to be better with libvorbis, so the switch to tremor was not pursued.
** experiments with using tremor instead of libvorbis showed it produces significantly smaller JS code. Performance, however, seems to be better with libvorbis, so the switch to tremor was not pursued.
* libopus
** it works!
* Internet Explorer
* Internet Explorer
** JS player works on IE 10 and 11 only, but requires a Flash shim for audio
** JS player works on IE 10 and 11 only, but requires a Flash shim for audio
** Flash player with JS wrapper works in IE 9/10/11; working on 6/7/8 support
* Edge
** JS player works using native Web Audio, no Flash required. Even works on Windows 10 Mobile!
* Safari
* Safari
** JS player works on OS X 10.8/Safari 6.1, OS X 10.9/Safari 7, and iOS 7
** JS player works on OS X 10.8/Safari 6.1, OS X 10.9/Safari 7, and iOS 8
** Safari JIT crash on iOS 7 64-bit seems to have been resolved by moving to the new emscripten LLVM backend
** Safari JIT crashes on iOS 7
** Audio must be triggered from a UI event to work on iOS (no autoplay)
** Audio must be triggered from a UI event to work on iOS (no auto play)


== Performance ==
== Performance ==


Performance seems adequate on recent-ish desktops/laptops in the latest browsers, but is woefully poor for medium-sized videos on most mobile phones and tablets.
Performance seems adequate on recent-ish desktops/laptops in the latest browsers, and on 64-bit iOS devices, but is woefully poor for medium-sized videos on 32-bit mobile phones and tablets.


'''Internet Explorer'''
'''Internet Explorer'''
* JavaScript supported only on IE 10/11; needs Flash for audio and for older browsers
* Older versions than IE 10 not supported
* JavaScript performance is not as good as other browsers, and there's a lot of GC churn visible in dev tools UI responsiveness profiling
* Flash audio shim required due to lack of Web Audio
* Flash player performance is similar to IE 11's JS on MacBook Pro, but has fewer slowdowns
* JavaScript performance is not as good as other browsers
* Flash player performs about the same as IE 11 JS on a Surface RT (Nvidia Tegra 3 CPU)
 
* Flash player performs significantly better than IE 11 JavaScript on an old Atom-based tablet where JS can't quite handle 360p/30
'''Edge/Windows 10'''
* Works natively
* Faster than IE 11 but not as fast as others
** Experimental asm.js optimizations bump performance back in line (set in about:flags)


'''Safari/Mac OS X'''
'''Safari/Mac OS X'''
* JS and Flash performance about the same on Safari 6.1/7; JS won't work or too slow on 6 and older
* JS has JIT bugs on 6.0, too slow on 5 and older
* Flash should work on old Safari (not yet tested)
* Works well on 6.1/7/8/9


'''Safari/iOS'''
'''Safari/iOS'''
* JavaScript only, no Flash
* 64-bit iOS 7 devices hit decode speed targets at 360p/30fps video
* 64-bit iOS 7 devices hit decode speed targets at 360p/30fps video
** tested iPhone 5s
** tested iPhone 5s, iPad Air
* 32-bit iOS 7 devices play audio, or video up to 160p/15fps acceptably
* 32-bit iOS 7 devices play audio, or video up to 160p/15fps acceptably
** tested iPod Touch 5th-gen and iPad 3
** tested iPod Touch 5th-gen, iPhone 4s, iPad 3
* iOS 6 seems to support the right APIs but doesn't play correctly, and the JIT is much too slow.
* iOS 6 seems to support the right APIs but doesn't play correctly, and the JIT is much too slow.
* non-Safari iOS browsers always disable the JIT and are much too slow.
* non-Safari iOS browsers always disable the JIT and are much too slow.
Line 62: Line 67:
'''Other modern browsers'''
'''Other modern browsers'''
* Other modern browsers don't actually need this for playback as they support Ogg Theora/Vorbis or WebM natively. Still of interest for experimental transcoding or encoding scenarios though.
* Other modern browsers don't actually need this for playback as they support Ogg Theora/Vorbis or WebM natively. Still of interest for experimental transcoding or encoding scenarios though.
* Unsurprisingly, Flash version performs about the same in any browser on the same hardware.
* Of modern browsers, Firefox performs JS best (asm.js optimizations).
* Of modern browsers, Firefox performs JS best (asm.js optimizations). Beware that current release Firefox 27 only uses the asm.js optimizations on the first run; this is fixed in Firefox 30 nightlies.
* Chrome and Opera (the new Chromium-based one) perform pretty well too.
* Chrome and Opera (the new Chromium-based one) perform pretty well too, maybe slightly better than Safari.


== HTML5 integration ==
== HTML5 integration ==


Current playback demo uses a custom wrapper 'ogvjs' or 'ogvswf' element which is extended to support a subset of the [https://developer.mozilla.org/en-US/docs/Web/API/HTMLMediaElement HTMLMediaElement API]. With some additional selection/loader code this could be used to dynamically replace a <video> or <audio> tag.
Current playback demo uses a custom wrapper 'ogvjs' element which is extended to support a subset of the [https://developer.mozilla.org/en-US/docs/Web/API/HTMLMediaElement HTMLMediaElement API]. With some additional selection/loader code this can be used to dynamically replace a <video> or <audio> tag.


WebRTC and MediaStreams also do not appear to allow easily plugging in custom JS codecs, but audio and video frames could be sent over WebRTC data channels as ArrayBuffers.
WebRTC and MediaStreams also do not appear to allow easily plugging in custom JS codecs, but audio and video frames could be sent over WebRTC data channels as ArrayBuffers.

Latest revision as of 07:34, 13 August 2015

At least libogg, libvorbis, libopus, and libtheora can be built to JavaScript using emscripten pretty straightforwardly.

ogv.js is a wrapper around these with basic JavaScript interfaces and video/audio playback wrappers semi-compatible with the native HTML5 <video>/<audio> elements.

Currently (August 2015) in use for audio/video playback in Safari/IE/Edge browsers on Wikipedia and Wikimedia Commons.

Generic decoding, or even encoding, should be possible in the future but needs better APIs to be sane.

Building

See the readme.md for details of building the full ogv.js package. The standard autotools build systems are used with the emscripten compiler front-end.

You can also download a pre-built release of ogv.js.

Demo

Playback demo at: https://brionv.com/misc/ogv.js/demo/

Integration into MediaWiki demo at: https://ogvjs-testing.wmflabs.org/ (defaults to native codecs when available, still a work in progress)

JavaScript video works in all current browsers, audio works if Web Audio API is there (latest Firefox, Safari, Chrome), or with a Flash audio shim for IE.

Notes

  • libtheora
    • Must pass --disable-asm for emscripten, as it misdetects the host/target?
    • needs a slight fix to a function signature to quiet an error from emscripten compiler
  • libvorbis
    • experiments with using tremor instead of libvorbis showed it produces significantly smaller JS code. Performance, however, seems to be better with libvorbis, so the switch to tremor was not pursued.
  • libopus
    • it works!
  • Internet Explorer
    • JS player works on IE 10 and 11 only, but requires a Flash shim for audio
  • Edge
    • JS player works using native Web Audio, no Flash required. Even works on Windows 10 Mobile!
  • Safari
    • JS player works on OS X 10.8/Safari 6.1, OS X 10.9/Safari 7, and iOS 8
    • Safari JIT crashes on iOS 7
    • Audio must be triggered from a UI event to work on iOS (no auto play)

Performance

Performance seems adequate on recent-ish desktops/laptops in the latest browsers, and on 64-bit iOS devices, but is woefully poor for medium-sized videos on 32-bit mobile phones and tablets.

Internet Explorer

  • Older versions than IE 10 not supported
  • Flash audio shim required due to lack of Web Audio
  • JavaScript performance is not as good as other browsers

Edge/Windows 10

  • Works natively
  • Faster than IE 11 but not as fast as others
    • Experimental asm.js optimizations bump performance back in line (set in about:flags)

Safari/Mac OS X

  • JS has JIT bugs on 6.0, too slow on 5 and older
  • Works well on 6.1/7/8/9

Safari/iOS

  • 64-bit iOS 7 devices hit decode speed targets at 360p/30fps video
    • tested iPhone 5s, iPad Air
  • 32-bit iOS 7 devices play audio, or video up to 160p/15fps acceptably
    • tested iPod Touch 5th-gen, iPhone 4s, iPad 3
  • iOS 6 seems to support the right APIs but doesn't play correctly, and the JIT is much too slow.
  • non-Safari iOS browsers always disable the JIT and are much too slow.

Other modern browsers

  • Other modern browsers don't actually need this for playback as they support Ogg Theora/Vorbis or WebM natively. Still of interest for experimental transcoding or encoding scenarios though.
  • Of modern browsers, Firefox performs JS best (asm.js optimizations).
  • Chrome and Opera (the new Chromium-based one) perform pretty well too.

HTML5 integration

Current playback demo uses a custom wrapper 'ogvjs' element which is extended to support a subset of the HTMLMediaElement API. With some additional selection/loader code this can be used to dynamically replace a <video> or <audio> tag.

WebRTC and MediaStreams also do not appear to allow easily plugging in custom JS codecs, but audio and video frames could be sent over WebRTC data channels as ArrayBuffers.

Live encoding from getUserMedia may be possible but I suspect frame rate will be awful as there's no way to grab the bits directly... and of course encoding is likely to be slow.

Transcoding from a native HTML5 video element may be similarly complex.

Related projects