Difference between revisions of "Ogv.js"

From XiphWiki
Jump to navigation Jump to search
(→‎Notes: Updating libvorbis <-> tremor usage)
(updates; has soft-launched on Wikipedia/Wikimedia Commons)
 
(5 intermediate revisions by 2 users not shown)
Line 1: Line 1:
At least libogg, libvorbis, and libtheora can be built to JavaScript using [http://emscripten.org/ emscripten] with only slight modifications.
+
At least libogg, libvorbis, libopus, and libtheora can be built to JavaScript using [http://emscripten.org/ emscripten] pretty straightforwardly.
 +
 
 +
'''[https://github.com/brion/ogv.js ogv.js]''' is a wrapper around these with basic JavaScript interfaces and video/audio playback wrappers semi-compatible with the native HTML5 <video>/<audio> elements.
 +
 
 +
Currently (August 2015) in use for audio/video playback in Safari/IE/Edge browsers on Wikipedia and [https://commons.wikimedia.org/wiki/Commons:Media_of_the_day Wikimedia Commons].
 +
 
 +
Generic decoding, or even encoding, should be possible in the future but needs better APIs to be sane.
  
 
== Building ==
 
== Building ==
  
Experimental work in progress script to build these: https://github.com/brion/ogv.js based on previous work for AudioCogs integration: https://github.com/devongovett/ogg.js
+
See the [https://github.com/brion/ogv.js#readme readme.md] for details of building the full ogv.js package. The standard autotools build systems are used with the emscripten compiler front-end.
 
 
The build scripts make slight changes to configure scripts which may or may not be necessary, need to dive in in more detail.
 
  
There's also a slight fix needed in libtheora: https://github.com/brion/theora/commit/06a0e4acf9c35f4bd31b8788a8a573cb89262333 which is probably safe to upstream (check this!)
+
You can also [https://github.com/brion/ogv.js/releases download a pre-built release of ogv.js].
  
 
== Demo ==
 
== Demo ==
Line 13: Line 17:
 
Playback demo at: https://brionv.com/misc/ogv.js/demo/
 
Playback demo at: https://brionv.com/misc/ogv.js/demo/
  
Video works in all current browsers, audio works if Web Audio API is there (latest Firefox, Safari, Chrome), or with a Flash audio shim for IE.
+
Integration into MediaWiki demo at: https://ogvjs-testing.wmflabs.org/ (defaults to native codecs when available, still a work in progress)
 +
 
 +
JavaScript video works in all current browsers, audio works if Web Audio API is there (latest Firefox, Safari, Chrome), or with a Flash audio shim for IE.
  
 
== Notes ==
 
== Notes ==
  
 
* libtheora
 
* libtheora
** Must pass --disable-asm
+
** Must pass --disable-asm for emscripten, as it misdetects the host/target?
 
** needs a slight fix to a function signature to quiet an error from emscripten compiler
 
** needs a slight fix to a function signature to quiet an error from emscripten compiler
 
* libvorbis
 
* libvorbis
** experimented with using tremor instead of libvorbis because it produces significantly smaller JS code. Performance, however, seems to be better with libvorbis, so the switch to tremor was reverted.
+
** experiments with using tremor instead of libvorbis showed it produces significantly smaller JS code. Performance, however, seems to be better with libvorbis, so the switch to tremor was not pursued.
 +
* libopus
 +
** it works!
 
* Internet Explorer
 
* Internet Explorer
** works on IE 10 and 11 only
+
** JS player works on IE 10 and 11 only, but requires a Flash shim for audio
** Flash audio shim works pretty well
+
* Edge
 +
** JS player works using native Web Audio, no Flash required. Even works on Windows 10 Mobile!
 
* Safari
 
* Safari
** works on OS X 10.8/Safari 6.1, OS X 10.9/Safari 7, and iOS 7
+
** JS player works on OS X 10.8/Safari 6.1, OS X 10.9/Safari 7, and iOS 8
** Safari JIT crash seems to have been resolved by moving to the new emscripten LLVM backend
+
** Safari JIT crashes on iOS 7
** Audio must be triggered from a UI event to work on iOS (no autoplay)
+
** Audio must be triggered from a UI event to work on iOS (no auto play)
 +
 
 +
== Performance ==
 +
 
 +
Performance seems adequate on recent-ish desktops/laptops in the latest browsers, and on 64-bit iOS devices, but is woefully poor for medium-sized videos on 32-bit mobile phones and tablets.
 +
 
 +
'''Internet Explorer'''
 +
* Older versions than IE 10 not supported
 +
* Flash audio shim required due to lack of Web Audio
 +
* JavaScript performance is not as good as other browsers
 +
 
 +
'''Edge/Windows 10'''
 +
* Works natively
 +
* Faster than IE 11 but not as fast as others
 +
** Experimental asm.js optimizations bump performance back in line (set in about:flags)
 +
 
 +
'''Safari/Mac OS X'''
 +
* JS has JIT bugs on 6.0, too slow on 5 and older
 +
* Works well on 6.1/7/8/9
  
Performance seems adequate on recentish desktops/laptops in the latest browsers, but is woefully poor on most mobile phones and tablets. The 64-bit iPhone 5s more or less hits decode speed targets at ~360p video; other iOS devices struggle to play 160p.
+
'''Safari/iOS'''
 +
* 64-bit iOS 7 devices hit decode speed targets at 360p/30fps video
 +
** tested iPhone 5s, iPad Air
 +
* 32-bit iOS 7 devices play audio, or video up to 160p/15fps acceptably
 +
** tested iPod Touch 5th-gen, iPhone 4s, iPad 3
 +
* iOS 6 seems to support the right APIs but doesn't play correctly, and the JIT is much too slow.
 +
* non-Safari iOS browsers always disable the JIT and are much too slow.
 +
 
 +
'''Other modern browsers'''
 +
* Other modern browsers don't actually need this for playback as they support Ogg Theora/Vorbis or WebM natively. Still of interest for experimental transcoding or encoding scenarios though.
 +
* Of modern browsers, Firefox performs JS best (asm.js optimizations).
 +
* Chrome and Opera (the new Chromium-based one) perform pretty well too.
  
 
== HTML5 integration ==
 
== HTML5 integration ==
  
Current playback demo simply outputs to a &lt;canvas> element, doing YUV to RGB conversion in software.
+
Current playback demo uses a custom wrapper 'ogvjs' element which is extended to support a subset of the [https://developer.mozilla.org/en-US/docs/Web/API/HTMLMediaElement HTMLMediaElement API]. With some additional selection/loader code this can be used to dynamically replace a <video> or <audio> tag.
  
It's not clear to me that playback could be integrated into an actual &lt;audio> or &lt;video> element, but similar JavaScript interfaces could be wrapped around a &lt;canvas>.
+
WebRTC and MediaStreams also do not appear to allow easily plugging in custom JS codecs, but audio and video frames could be sent over WebRTC data channels as ArrayBuffers.
  
WebRTC also does not appear to allow easily plugging in custom JS codecs, but audio and video frames could be sent over the data channels as ArrayBuffers.
+
Live encoding from getUserMedia may be possible but I suspect frame rate will be awful as there's no way to grab the bits directly... and of course encoding is likely to be slow.
  
Live encoding may be possible connected to getUserMedia, I have not attempted this yet.
+
Transcoding from a native HTML5 video element may be similarly complex.
  
 
== Related projects ==
 
== Related projects ==
  
* [[iOS]] codecs (todo)
+
* [[iOS]] codecs
* Flash codecs (see [[Talk:JavaScript#Flash codecs|Talk page]])
+
 
 +
{{DISPLAYTITLE:ogv.js}}

Latest revision as of 07:34, 13 August 2015

At least libogg, libvorbis, libopus, and libtheora can be built to JavaScript using emscripten pretty straightforwardly.

ogv.js is a wrapper around these with basic JavaScript interfaces and video/audio playback wrappers semi-compatible with the native HTML5 <video>/<audio> elements.

Currently (August 2015) in use for audio/video playback in Safari/IE/Edge browsers on Wikipedia and Wikimedia Commons.

Generic decoding, or even encoding, should be possible in the future but needs better APIs to be sane.

Building

See the readme.md for details of building the full ogv.js package. The standard autotools build systems are used with the emscripten compiler front-end.

You can also download a pre-built release of ogv.js.

Demo

Playback demo at: https://brionv.com/misc/ogv.js/demo/

Integration into MediaWiki demo at: https://ogvjs-testing.wmflabs.org/ (defaults to native codecs when available, still a work in progress)

JavaScript video works in all current browsers, audio works if Web Audio API is there (latest Firefox, Safari, Chrome), or with a Flash audio shim for IE.

Notes

  • libtheora
    • Must pass --disable-asm for emscripten, as it misdetects the host/target?
    • needs a slight fix to a function signature to quiet an error from emscripten compiler
  • libvorbis
    • experiments with using tremor instead of libvorbis showed it produces significantly smaller JS code. Performance, however, seems to be better with libvorbis, so the switch to tremor was not pursued.
  • libopus
    • it works!
  • Internet Explorer
    • JS player works on IE 10 and 11 only, but requires a Flash shim for audio
  • Edge
    • JS player works using native Web Audio, no Flash required. Even works on Windows 10 Mobile!
  • Safari
    • JS player works on OS X 10.8/Safari 6.1, OS X 10.9/Safari 7, and iOS 8
    • Safari JIT crashes on iOS 7
    • Audio must be triggered from a UI event to work on iOS (no auto play)

Performance

Performance seems adequate on recent-ish desktops/laptops in the latest browsers, and on 64-bit iOS devices, but is woefully poor for medium-sized videos on 32-bit mobile phones and tablets.

Internet Explorer

  • Older versions than IE 10 not supported
  • Flash audio shim required due to lack of Web Audio
  • JavaScript performance is not as good as other browsers

Edge/Windows 10

  • Works natively
  • Faster than IE 11 but not as fast as others
    • Experimental asm.js optimizations bump performance back in line (set in about:flags)

Safari/Mac OS X

  • JS has JIT bugs on 6.0, too slow on 5 and older
  • Works well on 6.1/7/8/9

Safari/iOS

  • 64-bit iOS 7 devices hit decode speed targets at 360p/30fps video
    • tested iPhone 5s, iPad Air
  • 32-bit iOS 7 devices play audio, or video up to 160p/15fps acceptably
    • tested iPod Touch 5th-gen, iPhone 4s, iPad 3
  • iOS 6 seems to support the right APIs but doesn't play correctly, and the JIT is much too slow.
  • non-Safari iOS browsers always disable the JIT and are much too slow.

Other modern browsers

  • Other modern browsers don't actually need this for playback as they support Ogg Theora/Vorbis or WebM natively. Still of interest for experimental transcoding or encoding scenarios though.
  • Of modern browsers, Firefox performs JS best (asm.js optimizations).
  • Chrome and Opera (the new Chromium-based one) perform pretty well too.

HTML5 integration

Current playback demo uses a custom wrapper 'ogvjs' element which is extended to support a subset of the HTMLMediaElement API. With some additional selection/loader code this can be used to dynamically replace a <video> or <audio> tag.

WebRTC and MediaStreams also do not appear to allow easily plugging in custom JS codecs, but audio and video frames could be sent over WebRTC data channels as ArrayBuffers.

Live encoding from getUserMedia may be possible but I suspect frame rate will be awful as there's no way to grab the bits directly... and of course encoding is likely to be slow.

Transcoding from a native HTML5 video element may be similarly complex.

Related projects