XiphWiki - User contributions [en]

Vorbis-tools

2013-02-13T19:06:02Z

Rillian: grammar fix

Several tools to use, manipulate and create Vorbis files.

* [http://downloads.xiph.org/releases/vorbis/ Release packages]
* [https://svn.xiph.org/trunk/vorbis-tools/ Source repository]

Contains:
* ogg123
* oggenc
* oggdec
* ogginfo
* vcut
* vorbiscomment

[[Category:Vorbis]]

Vorbis-tools

2013-02-13T19:05:42Z

Rillian: Replace out of date download link with generic ones.

Several tools to use, manipulate and create Vorbis files.

* [http://downloads.xiph.org/releases/vorbis/ Release packages]
* [https://svn.xiph.org/trunk/vorbis-tools/ Source repository]

Contain:
* ogg123
* oggenc
* oggdec
* ogginfo
* vcut
* vorbiscomment

[[Category:Vorbis]]

Daala

2012-12-07T22:44:34Z

Rillian: /* Weekly meetings */ Fix date

Daala is the current working name of a next generation video codec— to be renamed once someone insists on something better. So far the best proposed alternative is PatentCake.

For now the purposes of this page is to collect notes about things which have been discussed in informal public IRC discussion about the next generation initiative. Participants in these discussions have included Timothy Terriberry, Jason Garrett-Glaser, Loren Merritt, Ben Schwartz, Greg Maxwell, and others.

See also: [https://xiph.org/daala/ https://xiph.org/daala/]

== Weekly meetings ==

We've been having weekly progress meetings on mumble.

* 2012 June 4 [https://people.xiph.org/~giles/2012/daala_20120604.txt minutes] (actually a work week)
* 2012 June 22 [https://people.xiph.org/~giles/2012/daala_20120622.txt minutes]
* 2012 June 29 [https://people.xiph.org/~giles/2012/daala_20120629.txt minutes] [https://people.xiph.org/~giles/2012/daala_20120629.opus recording]
* 2012 July 6 [https://people.xiph.org/~giles/2012/daala_20120706.txt minutes]
* 2012 July 13 [https://people.xiph.org/~giles/2012/daala_20120713.txt minutes]
* 2012 July 20 [https://people.xiph.org/~giles/2012/daala_20120720.txt minutes]
* 2012 July 27 [https://people.xiph.org/~giles/2012/daala_20120727.txt minutes]
* 2012 August 3 [https://people.xiph.org/~giles/2012/daala_20120803.txt minutes] [https://people.xiph.org/~giles/2012/daala_20120803.opus recording]
* 2012 August 10 [https://people.xiph.org/~giles/2012/daala_20120810.txt minutes]
* 2012 August 17 - no meeting
* 2012 August 24 - [https://people.xiph.org/~giles/2012/daala_20120824.txt minutes] [https://people.xiph.org/~giles/2012/daala_20120824.opus recording]
* 2012 August 31 - no meeting
* 2012 September 7 - no meeting
* 2012 September 14 - no meeting
* 2012 September 21 - [https://people.xiph.org/~giles/2012/daala_20120921.txt minutes]
* 2012 September 28 - [https://people.xiph.org/~giles/2012/daala_20120928.txt minutes] [https://people.xiph.org/~giles/2012/daala_20120928.opus recording]
* 2012 October 5 - [https://people.xiph.org/~giles/2012/daala_20121005.opus recording]
* 20120 October 26 -
* 2012 Novemeber 2 - no meeting
* 2012 December 7 - [https://people.xiph.org/~giles/2012/daala_20121207.txt minutes]

= Techniques =

The discussed overall structure so far has been a variable size lapped-DCT block based codec with lapping done via pre/post filtering with a specially structured (lifting) linear phase transform along the edges along with overlapped block motion compensation and the expected trimmings. The lapping can be optimized for energy compaction and other useful properties, including invert-ability, and yields excellent results with efficient finite precision math.

Other components which have been discussed include:

==Techniques applicable to all frame types==
* Multisymbol arithmetic coding
** Timothy has some trial code showing speed-up proportional to the number of bits coded at once. (ec_test.c)
* Mode prediction using the previously decoded data, e.g. coding the mode using a probability function derived from trained predictors on the surrounding blocks.
** This will be terrible for robustness but may significantly reduce signalling overhead, allowing many more modes, and provide continuous adaptation between signalling free and fully signalled modes.
* Explore legendre polynomial basis transforms instead of DCT
** May have better perceptual properties and/or result in 'less compromised' efficient implementations.
* Coefficient domain prediction to allow efficient energy preserving quantization.
* Variable partition size/shape and the use of good predictors appears to remove most of the benefit of directional transforms.
** Perhaps 45deg is still useful?
** How does this change with partition sizes? Directional transforms are clearly not that useful with 4x4.
* Transform-post filtering to allow merging smaller transform blocks (like TF merging in CELT) may allow more flexible partitioning then outright using mixed block sizes.
* Perturbed quantization mode-signalling has been discussed but mostly laughed at. ;)
* Special block modes well suited to solid color/cartoon like content— avoiding ringing.
** Are pixel prediction modes too slow?
* In general— what markov random field techniques can be applied with acceptable performance. Any?
* Designed for parallel encode and decode within each frame
** Important because
*** the proposed techniques need a lot more CPU than H.264 and VP8 for both encode and decode
*** Moore's law for single-threaded throughput is dead. Future hardware is all multicore/GPU.
** Implies
*** Getting the order of application right for the lapping filters.
*** Mandatory slicing? Maybe some kind of multilevel entropy coding to reduce redundancy between slices while minimizing the single-threaded portion of decode.

* Using PVQ and energy conservation: see http://jmvalin.ca/video/video_pvq_v3.pdf

==Techniques applicable to inter frames==
* Using x264 as a test-bed Jason and Loren demonstrated 15% rate/distortion improvements from using 10-bit intermediaries and references, estimated as being 1/3rd from quality calculation in the 10-bit space, 1/3rd from the higher precision references, and 1/3rd from higher intermediate precision in calculations (e.g. MC filter processing).
** Increased reference precision competes for memory with increased number of references. The improvements demonstrated appear to be a greater win than increasing the reference count once there are four references or so.
* Super-resolution techniques for motion-compensation references have been discussed— in particular it appears that the half-pel location is where intelligent filtering matters the most so staged computation could be effectively used to allow more expensive filtering at that level.
** Edge-directed interpolation techniques might be effectively applied to increase motion compensation accuracy, but most of the techniques known to be very effective are too slow.
** Speculation has been offered that a significant part of MC inaccuracy may be due to blending in a physically incorrect (gamma-corrected) space, though no real conclusions were made. Academic papers on motion compensation accuracy seem to have ignored this issue.
* Timothy has an example code base for a variable partition size blocking-free motion compensation scheme which merges OBMC (overlapped block motion compensation) and CGI (control-grid interpolation) with an interesting prediction/sub-division scheme and whole-frame trellis optimization of motion vectors. (daala-exp)

==Basic features==
* YUV 4:4:4, 4:2:2 , 4:2:0 subsamplings, 8-bit, 10bit.
* Alpha channel — need testing material!
* 8-bit RGB compatible mode? (e.g. YCoCg, internally or at least flagging for it)
* Efficient 3D? — need testing material!
* Lossless?
** The value of this is disputable. If nothing else it's arguable that stuffing lossless into a lossy format may be the only way to get lossless into many people's hands. Also, see below
* Good support for decode side droppable frames?
** Hopefully the referencing structure will be flexible enough to enable this even if it's not an intentional feature.

==Frills==
* Optionally storing a checksum of the expected decoded frame for decoder/encoder mismatch detection.
* Expose the number of referential descendants of a given frame (or even the whole reference DAG) for most efficient allocation of FEC.

==Wingdings==
Crazy crap that might be interesting or at least fun to make fun of...
* >10bit?
** Use cases don't seem well enough defined yet. Significant complexity. Any prospective hardware developer may hire assassins.
** Possible compromise: the video reference structure contains a backbone that can be decoded at only N bits of depth (e.g. 10), and higher precisions are only supported outside of this reference chain.
**# Precision by truncation: decode is performed twice on each frame, identically, at low and high precision. The only difference between them is the bit-depth of the transform, or possibly of the transform and MC filters. Only low-precision outputs can be referenced by subsequent frames. Useful if high-precision content is still worth watching at low precision.
**# Precision by gamma: decode is performed once at low precision as normal. Then the output frame is converted to linear-light at high precision, after which another layer of residuals is added. The second layer can be permitted to reference previous high-precision frames... tricky to use both sets of references though. Useful if high precision is used for storing linear data, but people still want to watch it on "low-end" hardware.
* Some high end digital cameras are operating jpeg-derivatives in a special mode that keeps the image in the native linear RGB bayer format in order to avoid lossy/slow demosaicing on the camera. In particular this allows white balancing in post without excessive loss. Probably out of scope for Daala itself.
** Bayer, 4:2:0, 4:2:2, and Interlacing are all special cases of a more general pattern in which the output frames are decimated/subsampled in a regular fashion. All such subsamplings could be supported by a unified framework in which the video is always stored with all planes fully sampled, with a header indicating the recommended subsampling for display. In such cases, the encoder can regard the transform as highly overcomplete, and simply ignore unneeded coefficients (presumably by leaving high frequency residuals coded as zero). This structure would in effect turn the codec into a motion-compensated interpolating/deinterlacing filter. Whether this approach is sensible presumably depends in part on how the transform is structured. It would be especially easy if the transform's highest-frequencies were coded by a wavelet-like layer.
* Lossless intra-ability: The ability to losslessly rewrite any frame as an intra frame (perhaps with significant bitrate overhead) in order to make frame accurate cuts possible.
** Or best handled by making sure that containers have working pre-roll, but presumably common GOP sizes will be greater than the number of references so even if losslessly reencoding the references is expensive it may be cheaper than pre-roll. Do both?
** Can be had for 'free' if lossless is supported, plus the right header flags to restuff the references from lossless copies in a packed hidden frame.
** Use of explicitly (rather than staged) super-resolution and/or deeper references may make this functionality unattractive due to increased overhead.
* Internal overlays which could be swapped without re-encoding? (e.g. advertising, station ID). Could also be automatically generated by a Sufficiently Advanced™ encoder to improve efficiencies for static sprites over moving backgrounds.
** Complicates making the complexity bounded. No Sufficiently Advanced™ encoder likely to ever exist. But perhaps the station id/advertising uses fully justify this.
** Could be done externally to the video codec, but if so it's no likely to be useful for anyone ever.
* A secondary reference implementation in OpenCL, maintained throughout development, to make sure that the codec is GPU-friendly and can be done efficiently using OpenCL primitives.
* SWAR-friendly arithmetic. For example, choosing transform coefficients so that no intermediate product overflows 16 bits (tricky for signed values) can sometimes enable (e.g.) 4 parallel operations in one uint64_t. This can allow a pure C reference implementation to run faster, which is valuable for initial adoption and ports to new platforms.
* Parametric decode-side blur.
** Symmetrical blur in regions that are smooth on scales longer than the block size. Could be signaled or derived from observed DC values.
** Motion blur so that moving objects are blurred along the motion vector. May require coding a shutter speed parameter (0..1 as a fraction of the inter-frame interval).
* Fancy block property prediction. (Not clear how these prediction interact with intra pred)
** Predict block properties (quantizer, energy, etc.) from MV. (0,0) probably means small delta. Larger MV's may correspond to larger deltas ... although at low shutter speeds large MVs may correlate with reduced overall HF energy.
** Predict delta spectral shape from source block spectral shape. HF/LF ratio of the delta may be correlated with the same ratio in its source blocks. Works well with decode-side fDCT.

==Negative results==

* Using Kurtosis for detecting text in a frame
** The idea was to detect a Bernouilli distribution but it's not robust and too noisy

Daala

2012-12-07T22:43:51Z

Rillian: /* Weekly meetings */ link to meeting summary

Daala is the current working name of a next generation video codec— to be renamed once someone insists on something better. So far the best proposed alternative is PatentCake.

For now the purposes of this page is to collect notes about things which have been discussed in informal public IRC discussion about the next generation initiative. Participants in these discussions have included Timothy Terriberry, Jason Garrett-Glaser, Loren Merritt, Ben Schwartz, Greg Maxwell, and others.

See also: [https://xiph.org/daala/ https://xiph.org/daala/]

== Weekly meetings ==

We've been having weekly progress meetings on mumble.

* 2012 June 4 [https://people.xiph.org/~giles/2012/daala_20120604.txt minutes] (actually a work week)
* 2012 June 22 [https://people.xiph.org/~giles/2012/daala_20120622.txt minutes]
* 2012 June 29 [https://people.xiph.org/~giles/2012/daala_20120629.txt minutes] [https://people.xiph.org/~giles/2012/daala_20120629.opus recording]
* 2012 July 6 [https://people.xiph.org/~giles/2012/daala_20120706.txt minutes]
* 2012 July 13 [https://people.xiph.org/~giles/2012/daala_20120713.txt minutes]
* 2012 July 20 [https://people.xiph.org/~giles/2012/daala_20120720.txt minutes]
* 2012 July 27 [https://people.xiph.org/~giles/2012/daala_20120727.txt minutes]
* 2012 August 3 [https://people.xiph.org/~giles/2012/daala_20120803.txt minutes] [https://people.xiph.org/~giles/2012/daala_20120803.opus recording]
* 2012 August 10 [https://people.xiph.org/~giles/2012/daala_20120810.txt minutes]
* 2012 August 17 - no meeting
* 2012 August 24 - [https://people.xiph.org/~giles/2012/daala_20120824.txt minutes] [https://people.xiph.org/~giles/2012/daala_20120824.opus recording]
* 2012 August 31 - no meeting
* 2012 September 7 - no meeting
* 2012 September 14 - no meeting
* 2012 September 21 - [https://people.xiph.org/~giles/2012/daala_20120921.txt minutes]
* 2012 September 28 - [https://people.xiph.org/~giles/2012/daala_20120928.txt minutes] [https://people.xiph.org/~giles/2012/daala_20120928.opus recording]
* 2012 October 5 - [https://people.xiph.org/~giles/2012/daala_20121005.opus recording]
* 20120 October 26 -
* 2012 Novemeber 2 - no meeting
* 2012 December 6 - [https://people.xiph.org/~giles/2012/daala_20121206.txt minutes]

= Techniques =

The discussed overall structure so far has been a variable size lapped-DCT block based codec with lapping done via pre/post filtering with a specially structured (lifting) linear phase transform along the edges along with overlapped block motion compensation and the expected trimmings. The lapping can be optimized for energy compaction and other useful properties, including invert-ability, and yields excellent results with efficient finite precision math.

Other components which have been discussed include:

==Techniques applicable to all frame types==
* Multisymbol arithmetic coding
** Timothy has some trial code showing speed-up proportional to the number of bits coded at once. (ec_test.c)
* Mode prediction using the previously decoded data, e.g. coding the mode using a probability function derived from trained predictors on the surrounding blocks.
** This will be terrible for robustness but may significantly reduce signalling overhead, allowing many more modes, and provide continuous adaptation between signalling free and fully signalled modes.
* Explore legendre polynomial basis transforms instead of DCT
** May have better perceptual properties and/or result in 'less compromised' efficient implementations.
* Coefficient domain prediction to allow efficient energy preserving quantization.
* Variable partition size/shape and the use of good predictors appears to remove most of the benefit of directional transforms.
** Perhaps 45deg is still useful?
** How does this change with partition sizes? Directional transforms are clearly not that useful with 4x4.
* Transform-post filtering to allow merging smaller transform blocks (like TF merging in CELT) may allow more flexible partitioning then outright using mixed block sizes.
* Perturbed quantization mode-signalling has been discussed but mostly laughed at. ;)
* Special block modes well suited to solid color/cartoon like content— avoiding ringing.
** Are pixel prediction modes too slow?
* In general— what markov random field techniques can be applied with acceptable performance. Any?
* Designed for parallel encode and decode within each frame
** Important because
*** the proposed techniques need a lot more CPU than H.264 and VP8 for both encode and decode
*** Moore's law for single-threaded throughput is dead. Future hardware is all multicore/GPU.
** Implies
*** Getting the order of application right for the lapping filters.
*** Mandatory slicing? Maybe some kind of multilevel entropy coding to reduce redundancy between slices while minimizing the single-threaded portion of decode.

* Using PVQ and energy conservation: see http://jmvalin.ca/video/video_pvq_v3.pdf

==Techniques applicable to inter frames==
* Using x264 as a test-bed Jason and Loren demonstrated 15% rate/distortion improvements from using 10-bit intermediaries and references, estimated as being 1/3rd from quality calculation in the 10-bit space, 1/3rd from the higher precision references, and 1/3rd from higher intermediate precision in calculations (e.g. MC filter processing).
** Increased reference precision competes for memory with increased number of references. The improvements demonstrated appear to be a greater win than increasing the reference count once there are four references or so.
* Super-resolution techniques for motion-compensation references have been discussed— in particular it appears that the half-pel location is where intelligent filtering matters the most so staged computation could be effectively used to allow more expensive filtering at that level.
** Edge-directed interpolation techniques might be effectively applied to increase motion compensation accuracy, but most of the techniques known to be very effective are too slow.
** Speculation has been offered that a significant part of MC inaccuracy may be due to blending in a physically incorrect (gamma-corrected) space, though no real conclusions were made. Academic papers on motion compensation accuracy seem to have ignored this issue.
* Timothy has an example code base for a variable partition size blocking-free motion compensation scheme which merges OBMC (overlapped block motion compensation) and CGI (control-grid interpolation) with an interesting prediction/sub-division scheme and whole-frame trellis optimization of motion vectors. (daala-exp)

==Basic features==
* YUV 4:4:4, 4:2:2 , 4:2:0 subsamplings, 8-bit, 10bit.
* Alpha channel — need testing material!
* 8-bit RGB compatible mode? (e.g. YCoCg, internally or at least flagging for it)
* Efficient 3D? — need testing material!
* Lossless?
** The value of this is disputable. If nothing else it's arguable that stuffing lossless into a lossy format may be the only way to get lossless into many people's hands. Also, see below
* Good support for decode side droppable frames?
** Hopefully the referencing structure will be flexible enough to enable this even if it's not an intentional feature.

==Frills==
* Optionally storing a checksum of the expected decoded frame for decoder/encoder mismatch detection.
* Expose the number of referential descendants of a given frame (or even the whole reference DAG) for most efficient allocation of FEC.

==Wingdings==
Crazy crap that might be interesting or at least fun to make fun of...
* >10bit?
** Use cases don't seem well enough defined yet. Significant complexity. Any prospective hardware developer may hire assassins.
** Possible compromise: the video reference structure contains a backbone that can be decoded at only N bits of depth (e.g. 10), and higher precisions are only supported outside of this reference chain.
**# Precision by truncation: decode is performed twice on each frame, identically, at low and high precision. The only difference between them is the bit-depth of the transform, or possibly of the transform and MC filters. Only low-precision outputs can be referenced by subsequent frames. Useful if high-precision content is still worth watching at low precision.
**# Precision by gamma: decode is performed once at low precision as normal. Then the output frame is converted to linear-light at high precision, after which another layer of residuals is added. The second layer can be permitted to reference previous high-precision frames... tricky to use both sets of references though. Useful if high precision is used for storing linear data, but people still want to watch it on "low-end" hardware.
* Some high end digital cameras are operating jpeg-derivatives in a special mode that keeps the image in the native linear RGB bayer format in order to avoid lossy/slow demosaicing on the camera. In particular this allows white balancing in post without excessive loss. Probably out of scope for Daala itself.
** Bayer, 4:2:0, 4:2:2, and Interlacing are all special cases of a more general pattern in which the output frames are decimated/subsampled in a regular fashion. All such subsamplings could be supported by a unified framework in which the video is always stored with all planes fully sampled, with a header indicating the recommended subsampling for display. In such cases, the encoder can regard the transform as highly overcomplete, and simply ignore unneeded coefficients (presumably by leaving high frequency residuals coded as zero). This structure would in effect turn the codec into a motion-compensated interpolating/deinterlacing filter. Whether this approach is sensible presumably depends in part on how the transform is structured. It would be especially easy if the transform's highest-frequencies were coded by a wavelet-like layer.
* Lossless intra-ability: The ability to losslessly rewrite any frame as an intra frame (perhaps with significant bitrate overhead) in order to make frame accurate cuts possible.
** Or best handled by making sure that containers have working pre-roll, but presumably common GOP sizes will be greater than the number of references so even if losslessly reencoding the references is expensive it may be cheaper than pre-roll. Do both?
** Can be had for 'free' if lossless is supported, plus the right header flags to restuff the references from lossless copies in a packed hidden frame.
** Use of explicitly (rather than staged) super-resolution and/or deeper references may make this functionality unattractive due to increased overhead.
* Internal overlays which could be swapped without re-encoding? (e.g. advertising, station ID). Could also be automatically generated by a Sufficiently Advanced™ encoder to improve efficiencies for static sprites over moving backgrounds.
** Complicates making the complexity bounded. No Sufficiently Advanced™ encoder likely to ever exist. But perhaps the station id/advertising uses fully justify this.
** Could be done externally to the video codec, but if so it's no likely to be useful for anyone ever.
* A secondary reference implementation in OpenCL, maintained throughout development, to make sure that the codec is GPU-friendly and can be done efficiently using OpenCL primitives.
* SWAR-friendly arithmetic. For example, choosing transform coefficients so that no intermediate product overflows 16 bits (tricky for signed values) can sometimes enable (e.g.) 4 parallel operations in one uint64_t. This can allow a pure C reference implementation to run faster, which is valuable for initial adoption and ports to new platforms.
* Parametric decode-side blur.
** Symmetrical blur in regions that are smooth on scales longer than the block size. Could be signaled or derived from observed DC values.
** Motion blur so that moving objects are blurred along the motion vector. May require coding a shutter speed parameter (0..1 as a fraction of the inter-frame interval).
* Fancy block property prediction. (Not clear how these prediction interact with intra pred)
** Predict block properties (quantizer, energy, etc.) from MV. (0,0) probably means small delta. Larger MV's may correspond to larger deltas ... although at low shutter speeds large MVs may correlate with reduced overall HF energy.
** Predict delta spectral shape from source block spectral shape. HF/LF ratio of the delta may be correlated with the same ratio in its source blocks. Works well with decode-side fDCT.

==Negative results==

* Using Kurtosis for detecting text in a frame
** The idea was to detect a Bernouilli distribution but it's not robust and too noisy

Opus-1.0.2

2012-12-06T03:48:52Z

Rillian: /* Other minor changes */ s/minor//

Opus 1.0.2 fixes an out-of-bounds read that could be triggered by a malicious Opus packet causing an integer wrap-around in the padding code. Considering that the packet would have to be at least 16 MB in size and that no out-of-bounds write is possible, the severity is very low. This new release also has the following changes:

== Quality-impacting ==
* Changed the behaviour of the PLC to always fill the caller's buffer
* Properly decode in-band FEC for packets with mutiple Opus frames
* Hybrid mode quality improvements and fixes
* Fixed bugs in the CELT mode PLC
* Redundant mode transition fixes

== Other changes ==
* Stack reduction
* Doc fixes (many)
* 16-bit fixes
* Misc build fixes
* New API calls: OPUS_GET_LAST_PACKET_DURATION ctl() and opus_packet_get_nb_samples()
* Minor code cleanup

Opus-1.0.2

2012-12-05T23:10:04Z

Rillian: /* Other minor changes */ s/Extra/New/ API

Opus 1.0.2 fixes an out-of-bounds read that could be triggered by a malicious Opus packet causing an integer wrap-around in the padding code. Considering that the packet would have to be at least 16 MB in size and that no out-of-bounds write is possible, the severity is very low. This new release also has the following changes:

== Quality-impacting ==
* Changed the behaviour of the PLC to always fill the caller's buffer
* Hybrid mode quality improvements and fixes
* Fixed bugs in the CELT mode PLC
* Redundant mode transition fixes

== Other minor changes ==
* Stack reduction
* Doc fixes (many)
* 16-bit fixes
* Misc build fixes
* New API calls: OPUS_GET_LAST_PACKET_DURATION ctl() and opus_packet_get_nb_samples()
* Minor code cleanup

Opus-1.0.2

2012-12-05T22:58:04Z

Rillian: /* Quality-impacting */

Opus 1.0.2 fixes an out-of-bounds read that could be triggered by a malicious Opus packet causing an integer wrap-around in the padding code. Considering that the packet would have to be at least 16 MB in size and that no out-of-bounds write is possible, the severity is very low. This new release also has the following changes:

== Quality-impacting ==
* Changed the behaviour of the PLC to always fill the caller's buffer
* Hybrid mode quality improvements and fixes
* Fixed bugs in the CELT mode PLC
* Redundant mode transition fixes

== Other minor changes ==
* Stack reduction
* Doc fixes (many)
* 16-bit fixes
* Misc build fixes
* Extra API: OPUS_GET_LAST_PACKET_DURATION ctl() and opus_packet_get_nb_samples()
* Minor code cleanup

OPUS TODO

2012-12-04T18:03:46Z

Rillian: /* Spec */ link to drafts

== 1.0.2 ==

* multi-frame FEC/PLC fix
* PLC fix
* opus_packet_get_duration()
* OPUS_GET_FRAME_SIZE() for decoder??
* <strike>Add license headers to all dist files</strike> DONE
* <strike>Fix remaining build issues with MSVC</strike> DONE
* <strike>Add OPUS_EXPORT override for chrome</strike> DONE

== 1.1-beta ==

* tune transient detector
* variable frame size?
* LOTS of testing
* re-tune hybrid rate allocation
* re-tune mode switching decisions
* figure out how to use speech/music detection optimally
* everything from 1.0.2

== Lower priority ==

* Handle packets with PLC frames followed by FEC
* Better handling for the case where FEC has a different bandwidth than the current mode
* PLC transitions on unprotected SILK-SILK bandwidth changes?

== Spec ==
* Ogg mapping. See [[http://tools.ietf.org/html/draft-ietf-codec-oggopus IETF draft]]
* Matroska mapping. See: [[MatroskaOpus]]
* RTP payload format See [[http://tools.ietf.org/html/draft-spittka-payload-rtp-opus IETF draft]]

== Website ==
* De-uglify webpage
* Promotional material

== Other ==

* Oggz-validate (should also validate opus toc)

== Opus-tools ==
* A simple real time streaming example tool
* Replaygain (half done— needs a gain tool)

== Experiments ==

* Test exp_analysis and void_my_warranty.patch

== Future work ==
* Smart automatic mode decision
* psymodel based VBR
* Remove copy in inverse MDCT
* Save some float<->int conversions
* Improvements to LP mode CBR (greg has some code)

OPUS TODO

2012-12-04T18:00:21Z

Rillian: /* 1.0.2 */ other recent changes

== 1.0.2 ==

* multi-frame FEC/PLC fix
* PLC fix
* opus_packet_get_duration()
* OPUS_GET_FRAME_SIZE() for decoder??
* <strike>Add license headers to all dist files</strike> DONE
* <strike>Fix remaining build issues with MSVC</strike> DONE
* <strike>Add OPUS_EXPORT override for chrome</strike> DONE

== 1.1-beta ==

* tune transient detector
* variable frame size?
* LOTS of testing
* re-tune hybrid rate allocation
* re-tune mode switching decisions
* figure out how to use speech/music detection optimally
* everything from 1.0.2

== Lower priority ==

* Handle packets with PLC frames followed by FEC
* Better handling for the case where FEC has a different bandwidth than the current mode
* PLC transitions on unprotected SILK-SILK bandwidth changes?

== Spec ==
* Ogg mapping
* Matroska mapping. See: [[MatroskaOpus]]
* RTP payload format

== Website ==
* De-uglify webpage
* Promotional material

== Other ==

* Oggz-validate (should also validate opus toc)

== Opus-tools ==
* A simple real time streaming example tool
* Replaygain (half done— needs a gain tool)

== Experiments ==

* Test exp_analysis and void_my_warranty.patch

== Future work ==
* Smart automatic mode decision
* psymodel based VBR
* Remove copy in inverse MDCT
* Save some float<->int conversions
* Improvements to LP mode CBR (greg has some code)

Daala

2012-11-02T22:01:33Z

Rillian: /* Weekly meetings */ I've not been making updates lately

Daala is the current working name of a next generation video codec— to be renamed once someone insists on something better. So far the best proposed alternative is PatentCake.

For now the purposes of this page is to collect notes about things which have been discussed in informal public IRC discussion about the next generation initiative. Participants in these discussions have included Timothy Terriberry, Jason Garrett-Glaser, Loren Merritt, Ben Schwartz, Greg Maxwell, and others.

See also: [https://xiph.org/daala/ https://xiph.org/daala/]

== Weekly meetings ==

We've been having weekly progress meetings on mumble.

* 2012 June 4 [https://people.xiph.org/~giles/2012/daala_20120604.txt minutes] (actually a work week)
* 2012 June 22 [https://people.xiph.org/~giles/2012/daala_20120622.txt minutes]
* 2012 June 29 [https://people.xiph.org/~giles/2012/daala_20120629.txt minutes] [https://people.xiph.org/~giles/2012/daala_20120629.opus recording]
* 2012 July 6 [https://people.xiph.org/~giles/2012/daala_20120706.txt minutes]
* 2012 July 13 [https://people.xiph.org/~giles/2012/daala_20120713.txt minutes]
* 2012 July 20 [https://people.xiph.org/~giles/2012/daala_20120720.txt minutes]
* 2012 July 27 [https://people.xiph.org/~giles/2012/daala_20120727.txt minutes]
* 2012 August 3 [https://people.xiph.org/~giles/2012/daala_20120803.txt minutes] [https://people.xiph.org/~giles/2012/daala_20120803.opus recording]
* 2012 August 10 [https://people.xiph.org/~giles/2012/daala_20120810.txt minutes]
* 2012 August 17 - no meeting
* 2012 August 24 - [https://people.xiph.org/~giles/2012/daala_20120824.txt minutes] [https://people.xiph.org/~giles/2012/daala_20120824.opus recording]
* 2012 August 31 - no meeting
* 2012 September 7 - no meeting
* 2012 September 14 - no meeting
* 2012 September 21 - [https://people.xiph.org/~giles/2012/daala_20120921.txt minutes]
* 2012 September 28 - [https://people.xiph.org/~giles/2012/daala_20120928.txt minutes] [https://people.xiph.org/~giles/2012/daala_20120928.opus recording]
* 2012 October 5 - [https://people.xiph.org/~giles/2012/daala_20121005.opus recording]
* 20120 October 26 -
* 2012 Novemeber 2 - no meeting

= Techniques =

The discussed overall structure so far has been a variable size lapped-DCT block based codec with lapping done via pre/post filtering with a specially structured (lifting) linear phase transform along the edges along with overlapped block motion compensation and the expected trimmings. The lapping can be optimized for energy compaction and other useful properties, including invert-ability, and yields excellent results with efficient finite precision math.

Other components which have been discussed include:

==Techniques applicable to all frame types==
* Multisymbol arithmetic coding
** Timothy has some trial code showing speed-up proportional to the number of bits coded at once. (ec_test.c)
* Mode prediction using the previously decoded data, e.g. coding the mode using a probability function derived from trained predictors on the surrounding blocks.
** This will be terrible for robustness but may significantly reduce signalling overhead, allowing many more modes, and provide continuous adaptation between signalling free and fully signalled modes.
* Explore legendre polynomial basis transforms instead of DCT
** May have better perceptual properties and/or result in 'less compromised' efficient implementations.
* Coefficient domain prediction to allow efficient energy preserving quantization.
* Variable partition size/shape and the use of good predictors appears to remove most of the benefit of directional transforms.
** Perhaps 45deg is still useful?
** How does this change with partition sizes? Directional transforms are clearly not that useful with 4x4.
* Transform-post filtering to allow merging smaller transform blocks (like TF merging in CELT) may allow more flexible partitioning then outright using mixed block sizes.
* Perturbed quantization mode-signalling has been discussed but mostly laughed at. ;)
* Special block modes well suited to solid color/cartoon like content— avoiding ringing.
** Are pixel prediction modes too slow?
* In general— what markov random field techniques can be applied with acceptable performance. Any?
* Designed for parallel encode and decode within each frame
** Important because
*** the proposed techniques need a lot more CPU than H.264 and VP8 for both encode and decode
*** Moore's law for single-threaded throughput is dead. Future hardware is all multicore/GPU.
** Implies
*** Getting the order of application right for the lapping filters.
*** Mandatory slicing? Maybe some kind of multilevel entropy coding to reduce redundancy between slices while minimizing the single-threaded portion of decode.

* Using PVQ and energy conservation: see http://jmvalin.ca/video/video_pvq_v3.pdf

==Techniques applicable to inter frames==
* Using x264 as a test-bed Jason and Loren demonstrated 15% rate/distortion improvements from using 10-bit intermediaries and references, estimated as being 1/3rd from quality calculation in the 10-bit space, 1/3rd from the higher precision references, and 1/3rd from higher intermediate precision in calculations (e.g. MC filter processing).
** Increased reference precision competes for memory with increased number of references. The improvements demonstrated appear to be a greater win than increasing the reference count once there are four references or so.
* Super-resolution techniques for motion-compensation references have been discussed— in particular it appears that the half-pel location is where intelligent filtering matters the most so staged computation could be effectively used to allow more expensive filtering at that level.
** Edge-directed interpolation techniques might be effectively applied to increase motion compensation accuracy, but most of the techniques known to be very effective are too slow.
** Speculation has been offered that a significant part of MC inaccuracy may be due to blending in a physically incorrect (gamma-corrected) space, though no real conclusions were made. Academic papers on motion compensation accuracy seem to have ignored this issue.
* Timothy has an example code base for a variable partition size blocking-free motion compensation scheme which merges OBMC (overlapped block motion compensation) and CGI (control-grid interpolation) with an interesting prediction/sub-division scheme and whole-frame trellis optimization of motion vectors. (daala-exp)

==Basic features==
* YUV 4:4:4, 4:2:2 , 4:2:0 subsamplings, 8-bit, 10bit.
* Alpha channel — need testing material!
* 8-bit RGB compatible mode? (e.g. YCoCg, internally or at least flagging for it)
* Efficient 3D? — need testing material!
* Lossless?
** The value of this is disputable. If nothing else it's arguable that stuffing lossless into a lossy format may be the only way to get lossless into many people's hands. Also, see below
* Good support for decode side droppable frames?
** Hopefully the referencing structure will be flexible enough to enable this even if it's not an intentional feature.

==Frills==
* Optionally storing a checksum of the expected decoded frame for decoder/encoder mismatch detection.
* Expose the number of referential descendants of a given frame (or even the whole reference DAG) for most efficient allocation of FEC.

==Wingdings==
Crazy crap that might be interesting or at least fun to make fun of...
* >10bit?
** Use cases don't seem well enough defined yet. Significant complexity. Any prospective hardware developer may hire assassins.
** Possible compromise: the video reference structure contains a backbone that can be decoded at only N bits of depth (e.g. 10), and higher precisions are only supported outside of this reference chain.
**# Precision by truncation: decode is performed twice on each frame, identically, at low and high precision. The only difference between them is the bit-depth of the transform, or possibly of the transform and MC filters. Only low-precision outputs can be referenced by subsequent frames. Useful if high-precision content is still worth watching at low precision.
**# Precision by gamma: decode is performed once at low precision as normal. Then the output frame is converted to linear-light at high precision, after which another layer of residuals is added. The second layer can be permitted to reference previous high-precision frames... tricky to use both sets of references though. Useful if high precision is used for storing linear data, but people still want to watch it on "low-end" hardware.
* Some high end digital cameras are operating jpeg-derivatives in a special mode that keeps the image in the native linear RGB bayer format in order to avoid lossy/slow demosaicing on the camera. In particular this allows white balancing in post without excessive loss. Probably out of scope for Daala itself.
** Bayer, 4:2:0, 4:2:2, and Interlacing are all special cases of a more general pattern in which the output frames are decimated/subsampled in a regular fashion. All such subsamplings could be supported by a unified framework in which the video is always stored with all planes fully sampled, with a header indicating the recommended subsampling for display. In such cases, the encoder can regard the transform as highly overcomplete, and simply ignore unneeded coefficients (presumably by leaving high frequency residuals coded as zero). This structure would in effect turn the codec into a motion-compensated interpolating/deinterlacing filter. Whether this approach is sensible presumably depends in part on how the transform is structured. It would be especially easy if the transform's highest-frequencies were coded by a wavelet-like layer.
* Lossless intra-ability: The ability to losslessly rewrite any frame as an intra frame (perhaps with significant bitrate overhead) in order to make frame accurate cuts possible.
** Or best handled by making sure that containers have working pre-roll, but presumably common GOP sizes will be greater than the number of references so even if losslessly reencoding the references is expensive it may be cheaper than pre-roll. Do both?
** Can be had for 'free' if lossless is supported, plus the right header flags to restuff the references from lossless copies in a packed hidden frame.
** Use of explicitly (rather than staged) super-resolution and/or deeper references may make this functionality unattractive due to increased overhead.
* Internal overlays which could be swapped without re-encoding? (e.g. advertising, station ID). Could also be automatically generated by a Sufficiently Advanced™ encoder to improve efficiencies for static sprites over moving backgrounds.
** Complicates making the complexity bounded. No Sufficiently Advanced™ encoder likely to ever exist. But perhaps the station id/advertising uses fully justify this.
** Could be done externally to the video codec, but if so it's no likely to be useful for anyone ever.
* A secondary reference implementation in OpenCL, maintained throughout development, to make sure that the codec is GPU-friendly and can be done efficiently using OpenCL primitives.
* SWAR-friendly arithmetic. For example, choosing transform coefficients so that no intermediate product overflows 16 bits (tricky for signed values) can sometimes enable (e.g.) 4 parallel operations in one uint64_t. This can allow a pure C reference implementation to run faster, which is valuable for initial adoption and ports to new platforms.
* Parametric decode-side blur.
** Symmetrical blur in regions that are smooth on scales longer than the block size. Could be signaled or derived from observed DC values.
** Motion blur so that moving objects are blurred along the motion vector. May require coding a shutter speed parameter (0..1 as a fraction of the inter-frame interval).
* Fancy block property prediction. (Not clear how these prediction interact with intra pred)
** Predict block properties (quantizer, energy, etc.) from MV. (0,0) probably means small delta. Larger MV's may correspond to larger deltas ... although at low shutter speeds large MVs may correlate with reduced overall HF energy.
** Predict delta spectral shape from source block spectral shape. HF/LF ratio of the delta may be correlated with the same ratio in its source blocks. Works well with decode-side fDCT.

==Negative results==

* Using Kurtosis for detecting text in a frame
** The idea was to detect a Bernouilli distribution but it's not robust and too noisy

Daala

2012-10-05T20:30:35Z

Rillian: /* Weekly meetings */ Recording link for today's meeting.

Daala is the current working name of a next generation video codec— to be renamed once someone insists on something better. So far the best proposed alternative is PatentCake.

For now the purposes of this page is to collect notes about things which have been discussed in informal public IRC discussion about the next generation initiative. Participants in these discussions have included Timothy Terriberry, Jason Garrett-Glaser, Loren Merritt, Ben Schwartz, Greg Maxwell, and others.

See also: [https://xiph.org/daala/ https://xiph.org/daala/]

== Weekly meetings ==

We've been having weekly progress meetings on mumble.

* 2012 June 4 [https://people.xiph.org/~giles/2012/daala_20120604.txt minutes] (actually a work week)
* 2012 June 22 [https://people.xiph.org/~giles/2012/daala_20120622.txt minutes]
* 2012 June 29 [https://people.xiph.org/~giles/2012/daala_20120629.txt minutes] [https://people.xiph.org/~giles/2012/daala_20120629.opus recording]
* 2012 July 6 [https://people.xiph.org/~giles/2012/daala_20120706.txt minutes]
* 2012 July 13 [https://people.xiph.org/~giles/2012/daala_20120713.txt minutes]
* 2012 July 20 [https://people.xiph.org/~giles/2012/daala_20120720.txt minutes]
* 2012 July 27 [https://people.xiph.org/~giles/2012/daala_20120727.txt minutes]
* 2012 August 3 [https://people.xiph.org/~giles/2012/daala_20120803.txt minutes] [https://people.xiph.org/~giles/2012/daala_20120803.opus recording]
* 2012 August 10 [https://people.xiph.org/~giles/2012/daala_20120810.txt minutes]
* 2012 August 17 - no meeting
* 2012 August 24 - [https://people.xiph.org/~giles/2012/daala_20120824.txt minutes] [https://people.xiph.org/~giles/2012/daala_20120824.opus recording]
* 2012 August 31 - no meeting
* 2012 September 7 - no meeting
* 2012 September 14 - no meeting
* 2012 September 21 - [https://people.xiph.org/~giles/2012/daala_20120921.txt minutes]
* 2012 September 28 - [https://people.xiph.org/~giles/2012/daala_20120928.txt minutes] [https://people.xiph.org/~giles/2012/daala_20120928.opus recording]
* 2012 October 5 - [https://people.xiph.org/~giles/2012/daala_20121005.opus recording]

= Techniques =

The discussed overall structure so far has been a variable size lapped-DCT block based codec with lapping done via pre/post filtering with a specially structured (lifting) linear phase transform along the edges along with overlapped block motion compensation and the expected trimmings. The lapping can be optimized for energy compaction and other useful properties, including invert-ability, and yields excellent results with efficient finite precision math.

Other components which have been discussed include:

==Techniques applicable to all frame types==
* Multisymbol arithmetic coding
** Timothy has some trial code showing speed-up proportional to the number of bits coded at once. (ec_test.c)
* Mode prediction using the previously decoded data, e.g. coding the mode using a probability function derived from trained predictors on the surrounding blocks.
** This will be terrible for robustness but may significantly reduce signalling overhead, allowing many more modes, and provide continuous adaptation between signalling free and fully signalled modes.
* Explore legendre polynomial basis transforms instead of DCT
** May have better perceptual properties and/or result in 'less compromised' efficient implementations.
* Coefficient domain prediction to allow efficient energy preserving quantization.
* Variable partition size/shape and the use of good predictors appears to remove most of the benefit of directional transforms.
** Perhaps 45deg is still useful?
** How does this change with partition sizes? Directional transforms are clearly not that useful with 4x4.
* Transform-post filtering to allow merging smaller transform blocks (like TF merging in CELT) may allow more flexible partitioning then outright using mixed block sizes.
* Perturbed quantization mode-signalling has been discussed but mostly laughed at. ;)
* Special block modes well suited to solid color/cartoon like content— avoiding ringing.
** Are pixel prediction modes too slow?
* In general— what markov random field techniques can be applied with acceptable performance. Any?
* Designed for parallel encode and decode within each frame
** Important because
*** the proposed techniques need a lot more CPU than H.264 and VP8 for both encode and decode
*** Moore's law for single-threaded throughput is dead. Future hardware is all multicore/GPU.
** Implies
*** Getting the order of application right for the lapping filters.
*** Mandatory slicing? Maybe some kind of multilevel entropy coding to reduce redundancy between slices while minimizing the single-threaded portion of decode.

* Using PVQ and energy conservation: see http://jmvalin.ca/video/video_pvq_v3.pdf

==Techniques applicable to inter frames==
* Using x264 as a test-bed Jason and Loren demonstrated 15% rate/distortion improvements from using 10-bit intermediaries and references, estimated as being 1/3rd from quality calculation in the 10-bit space, 1/3rd from the higher precision references, and 1/3rd from higher intermediate precision in calculations (e.g. MC filter processing).
** Increased reference precision competes for memory with increased number of references. The improvements demonstrated appear to be a greater win than increasing the reference count once there are four references or so.
* Super-resolution techniques for motion-compensation references have been discussed— in particular it appears that the half-pel location is where intelligent filtering matters the most so staged computation could be effectively used to allow more expensive filtering at that level.
** Edge-directed interpolation techniques might be effectively applied to increase motion compensation accuracy, but most of the techniques known to be very effective are too slow.
** Speculation has been offered that a significant part of MC inaccuracy may be due to blending in a physically incorrect (gamma-corrected) space, though no real conclusions were made. Academic papers on motion compensation accuracy seem to have ignored this issue.
* Timothy has an example code base for a variable partition size blocking-free motion compensation scheme which merges OBMC (overlapped block motion compensation) and CGI (control-grid interpolation) with an interesting prediction/sub-division scheme and whole-frame trellis optimization of motion vectors. (daala-exp)

==Basic features==
* YUV 4:4:4, 4:2:2 , 4:2:0 subsamplings, 8-bit, 10bit.
* Alpha channel — need testing material!
* 8-bit RGB compatible mode? (e.g. YCoCg, internally or at least flagging for it)
* Efficient 3D? — need testing material!
* Lossless?
** The value of this is disputable. If nothing else it's arguable that stuffing lossless into a lossy format may be the only way to get lossless into many people's hands. Also, see below
* Good support for decode side droppable frames?
** Hopefully the referencing structure will be flexible enough to enable this even if it's not an intentional feature.

==Frills==
* Optionally storing a checksum of the expected decoded frame for decoder/encoder mismatch detection.
* Expose the number of referential descendants of a given frame (or even the whole reference DAG) for most efficient allocation of FEC.

==Wingdings==
Crazy crap that might be interesting or at least fun to make fun of...
* >10bit?
** Use cases don't seem well enough defined yet. Significant complexity. Any prospective hardware developer may hire assassins.
** Possible compromise: the video reference structure contains a backbone that can be decoded at only N bits of depth (e.g. 10), and higher precisions are only supported outside of this reference chain.
**# Precision by truncation: decode is performed twice on each frame, identically, at low and high precision. The only difference between them is the bit-depth of the transform, or possibly of the transform and MC filters. Only low-precision outputs can be referenced by subsequent frames. Useful if high-precision content is still worth watching at low precision.
**# Precision by gamma: decode is performed once at low precision as normal. Then the output frame is converted to linear-light at high precision, after which another layer of residuals is added. The second layer can be permitted to reference previous high-precision frames... tricky to use both sets of references though. Useful if high precision is used for storing linear data, but people still want to watch it on "low-end" hardware.
* Some high end digital cameras are operating jpeg-derivatives in a special mode that keeps the image in the native linear RGB bayer format in order to avoid lossy/slow demosaicing on the camera. In particular this allows white balancing in post without excessive loss. Probably out of scope for Daala itself.
** Bayer, 4:2:0, 4:2:2, and Interlacing are all special cases of a more general pattern in which the output frames are decimated/subsampled in a regular fashion. All such subsamplings could be supported by a unified framework in which the video is always stored with all planes fully sampled, with a header indicating the recommended subsampling for display. In such cases, the encoder can regard the transform as highly overcomplete, and simply ignore unneeded coefficients (presumably by leaving high frequency residuals coded as zero). This structure would in effect turn the codec into a motion-compensated interpolating/deinterlacing filter. Whether this approach is sensible presumably depends in part on how the transform is structured. It would be especially easy if the transform's highest-frequencies were coded by a wavelet-like layer.
* Lossless intra-ability: The ability to losslessly rewrite any frame as an intra frame (perhaps with significant bitrate overhead) in order to make frame accurate cuts possible.
** Or best handled by making sure that containers have working pre-roll, but presumably common GOP sizes will be greater than the number of references so even if losslessly reencoding the references is expensive it may be cheaper than pre-roll. Do both?
** Can be had for 'free' if lossless is supported, plus the right header flags to restuff the references from lossless copies in a packed hidden frame.
** Use of explicitly (rather than staged) super-resolution and/or deeper references may make this functionality unattractive due to increased overhead.
* Internal overlays which could be swapped without re-encoding? (e.g. advertising, station ID). Could also be automatically generated by a Sufficiently Advanced™ encoder to improve efficiencies for static sprites over moving backgrounds.
** Complicates making the complexity bounded. No Sufficiently Advanced™ encoder likely to ever exist. But perhaps the station id/advertising uses fully justify this.
** Could be done externally to the video codec, but if so it's no likely to be useful for anyone ever.
* A secondary reference implementation in OpenCL, maintained throughout development, to make sure that the codec is GPU-friendly and can be done efficiently using OpenCL primitives.
* SWAR-friendly arithmetic. For example, choosing transform coefficients so that no intermediate product overflows 16 bits (tricky for signed values) can sometimes enable (e.g.) 4 parallel operations in one uint64_t. This can allow a pure C reference implementation to run faster, which is valuable for initial adoption and ports to new platforms.
* Parametric decode-side blur.
** Symmetrical blur in regions that are smooth on scales longer than the block size. Could be signaled or derived from observed DC values.
** Motion blur so that moving objects are blurred along the motion vector. May require coding a shutter speed parameter (0..1 as a fraction of the inter-frame interval).
* Fancy block property prediction. (Not clear how these prediction interact with intra pred)
** Predict block properties (quantizer, energy, etc.) from MV. (0,0) probably means small delta. Larger MV's may correspond to larger deltas ... although at low shutter speeds large MVs may correlate with reduced overall HF energy.
** Predict delta spectral shape from source block spectral shape. HF/LF ratio of the delta may be correlated with the same ratio in its source blocks. Works well with decode-side fDCT.

==Negative results==

* Using Kurtosis for detecting text in a frame
** The idea was to detect a Bernouilli distribution but it's not robust and too noisy

Daala

2012-09-28T22:48:35Z

Rillian: /* Weekly meetings */ Link to minutes and recording

Daala is the current working name of a next generation video codec— to be renamed once someone insists on something better. So far the best proposed alternative is PatentCake.

For now the purposes of this page is to collect notes about things which have been discussed in informal public IRC discussion about the next generation initiative. Participants in these discussions have included Timothy Terriberry, Jason Garrett-Glaser, Loren Merritt, Ben Schwartz, Greg Maxwell, and others.

See also: [https://xiph.org/daala/ https://xiph.org/daala/]

== Weekly meetings ==

We've been having weekly progress meetings on mumble.

* 2012 June 4 [https://people.xiph.org/~giles/2012/daala_20120604.txt minutes] (actually a work week)
* 2012 June 22 [https://people.xiph.org/~giles/2012/daala_20120622.txt minutes]
* 2012 June 29 [https://people.xiph.org/~giles/2012/daala_20120629.txt minutes] [https://people.xiph.org/~giles/2012/daala_20120629.opus recording]
* 2012 July 6 [https://people.xiph.org/~giles/2012/daala_20120706.txt minutes]
* 2012 July 13 [https://people.xiph.org/~giles/2012/daala_20120713.txt minutes]
* 2012 July 20 [https://people.xiph.org/~giles/2012/daala_20120720.txt minutes]
* 2012 July 27 [https://people.xiph.org/~giles/2012/daala_20120727.txt minutes]
* 2012 August 3 [https://people.xiph.org/~giles/2012/daala_20120803.txt minutes] [https://people.xiph.org/~giles/2012/daala_20120803.opus recording]
* 2012 August 10 [https://people.xiph.org/~giles/2012/daala_20120810.txt minutes]
* 2012 August 17 - no meeting
* 2012 August 24 - [https://people.xiph.org/~giles/2012/daala_20120824.txt minutes] [https://people.xiph.org/~giles/2012/daala_20120824.opus recording]
* 2012 August 31 - no meeting
* 2012 September 7 - no meeting
* 2012 September 14 - no meeting
* 2012 September 21 - [https://people.xiph.org/~giles/2012/daala_20120921.txt minutes]
* 2012 September 28 - [https://people.xiph.org/~giles/2012/daala_20120928.txt minutes] [https://people.xiph.org/~giles/2012/daala_20120928.opus recording]

= Techniques =

The discussed overall structure so far has been a variable size lapped-DCT block based codec with lapping done via pre/post filtering with a specially structured (lifting) linear phase transform along the edges along with overlapped block motion compensation and the expected trimmings. The lapping can be optimized for energy compaction and other useful properties, including invert-ability, and yields excellent results with efficient finite precision math.

Other components which have been discussed include:

==Techniques applicable to all frame types==
* Multisymbol arithmetic coding
** Timothy has some trial code showing speed-up proportional to the number of bits coded at once. (ec_test.c)
* Mode prediction using the previously decoded data, e.g. coding the mode using a probability function derived from trained predictors on the surrounding blocks.
** This will be terrible for robustness but may significantly reduce signalling overhead, allowing many more modes, and provide continuous adaptation between signalling free and fully signalled modes.
* Explore legendre polynomial basis transforms instead of DCT
** May have better perceptual properties and/or result in 'less compromised' efficient implementations.
* Coefficient domain prediction to allow efficient energy preserving quantization.
* Variable partition size/shape and the use of good predictors appears to remove most of the benefit of directional transforms.
** Perhaps 45deg is still useful?
** How does this change with partition sizes? Directional transforms are clearly not that useful with 4x4.
* Transform-post filtering to allow merging smaller transform blocks (like TF merging in CELT) may allow more flexible partitioning then outright using mixed block sizes.
* Perturbed quantization mode-signalling has been discussed but mostly laughed at. ;)
* Special block modes well suited to solid color/cartoon like content— avoiding ringing.
** Are pixel prediction modes too slow?
* In general— what markov random field techniques can be applied with acceptable performance. Any?
* Designed for parallel encode and decode within each frame
** Important because
*** the proposed techniques need a lot more CPU than H.264 and VP8 for both encode and decode
*** Moore's law for single-threaded throughput is dead. Future hardware is all multicore/GPU.
** Implies
*** Getting the order of application right for the lapping filters.
*** Mandatory slicing? Maybe some kind of multilevel entropy coding to reduce redundancy between slices while minimizing the single-threaded portion of decode.

* Using PVQ and energy conservation: see http://jmvalin.ca/video/video_pvq_v3.pdf

==Techniques applicable to inter frames==
* Using x264 as a test-bed Jason and Loren demonstrated 15% rate/distortion improvements from using 10-bit intermediaries and references, estimated as being 1/3rd from quality calculation in the 10-bit space, 1/3rd from the higher precision references, and 1/3rd from higher intermediate precision in calculations (e.g. MC filter processing).
** Increased reference precision competes for memory with increased number of references. The improvements demonstrated appear to be a greater win than increasing the reference count once there are four references or so.
* Super-resolution techniques for motion-compensation references have been discussed— in particular it appears that the half-pel location is where intelligent filtering matters the most so staged computation could be effectively used to allow more expensive filtering at that level.
** Edge-directed interpolation techniques might be effectively applied to increase motion compensation accuracy, but most of the techniques known to be very effective are too slow.
** Speculation has been offered that a significant part of MC inaccuracy may be due to blending in a physically incorrect (gamma-corrected) space, though no real conclusions were made. Academic papers on motion compensation accuracy seem to have ignored this issue.
* Timothy has an example code base for a variable partition size blocking-free motion compensation scheme which merges OBMC (overlapped block motion compensation) and CGI (control-grid interpolation) with an interesting prediction/sub-division scheme and whole-frame trellis optimization of motion vectors. (daala-exp)

==Basic features==
* YUV 4:4:4, 4:2:2 , 4:2:0 subsamplings, 8-bit, 10bit.
* Alpha channel — need testing material!
* 8-bit RGB compatible mode? (e.g. YCoCg, internally or at least flagging for it)
* Efficient 3D? — need testing material!
* Lossless?
** The value of this is disputable. If nothing else it's arguable that stuffing lossless into a lossy format may be the only way to get lossless into many people's hands. Also, see below
* Good support for decode side droppable frames?
** Hopefully the referencing structure will be flexible enough to enable this even if it's not an intentional feature.

==Frills==
* Optionally storing a checksum of the expected decoded frame for decoder/encoder mismatch detection.
* Expose the number of referential descendants of a given frame (or even the whole reference DAG) for most efficient allocation of FEC.

==Wingdings==
Crazy crap that might be interesting or at least fun to make fun of...
* >10bit?
** Use cases don't seem well enough defined yet. Significant complexity. Any prospective hardware developer may hire assassins.
** Possible compromise: the video reference structure contains a backbone that can be decoded at only N bits of depth (e.g. 10), and higher precisions are only supported outside of this reference chain.
**# Precision by truncation: decode is performed twice on each frame, identically, at low and high precision. The only difference between them is the bit-depth of the transform, or possibly of the transform and MC filters. Only low-precision outputs can be referenced by subsequent frames. Useful if high-precision content is still worth watching at low precision.
**# Precision by gamma: decode is performed once at low precision as normal. Then the output frame is converted to linear-light at high precision, after which another layer of residuals is added. The second layer can be permitted to reference previous high-precision frames... tricky to use both sets of references though. Useful if high precision is used for storing linear data, but people still want to watch it on "low-end" hardware.
* Some high end digital cameras are operating jpeg-derivatives in a special mode that keeps the image in the native linear RGB bayer format in order to avoid lossy/slow demosaicing on the camera. In particular this allows white balancing in post without excessive loss. Probably out of scope for Daala itself.
** Bayer, 4:2:0, 4:2:2, and Interlacing are all special cases of a more general pattern in which the output frames are decimated/subsampled in a regular fashion. All such subsamplings could be supported by a unified framework in which the video is always stored with all planes fully sampled, with a header indicating the recommended subsampling for display. In such cases, the encoder can regard the transform as highly overcomplete, and simply ignore unneeded coefficients (presumably by leaving high frequency residuals coded as zero). This structure would in effect turn the codec into a motion-compensated interpolating/deinterlacing filter. Whether this approach is sensible presumably depends in part on how the transform is structured. It would be especially easy if the transform's highest-frequencies were coded by a wavelet-like layer.
* Lossless intra-ability: The ability to losslessly rewrite any frame as an intra frame (perhaps with significant bitrate overhead) in order to make frame accurate cuts possible.
** Or best handled by making sure that containers have working pre-roll, but presumably common GOP sizes will be greater than the number of references so even if losslessly reencoding the references is expensive it may be cheaper than pre-roll. Do both?
** Can be had for 'free' if lossless is supported, plus the right header flags to restuff the references from lossless copies in a packed hidden frame.
** Use of explicitly (rather than staged) super-resolution and/or deeper references may make this functionality unattractive due to increased overhead.
* Internal overlays which could be swapped without re-encoding? (e.g. advertising, station ID). Could also be automatically generated by a Sufficiently Advanced™ encoder to improve efficiencies for static sprites over moving backgrounds.
** Complicates making the complexity bounded. No Sufficiently Advanced™ encoder likely to ever exist. But perhaps the station id/advertising uses fully justify this.
** Could be done externally to the video codec, but if so it's no likely to be useful for anyone ever.
* A secondary reference implementation in OpenCL, maintained throughout development, to make sure that the codec is GPU-friendly and can be done efficiently using OpenCL primitives.
* SWAR-friendly arithmetic. For example, choosing transform coefficients so that no intermediate product overflows 16 bits (tricky for signed values) can sometimes enable (e.g.) 4 parallel operations in one uint64_t. This can allow a pure C reference implementation to run faster, which is valuable for initial adoption and ports to new platforms.
* Parametric decode-side blur.
** Symmetrical blur in regions that are smooth on scales longer than the block size. Could be signaled or derived from observed DC values.
** Motion blur so that moving objects are blurred along the motion vector. May require coding a shutter speed parameter (0..1 as a fraction of the inter-frame interval).
* Fancy block property prediction. (Not clear how these prediction interact with intra pred)
** Predict block properties (quantizer, energy, etc.) from MV. (0,0) probably means small delta. Larger MV's may correspond to larger deltas ... although at low shutter speeds large MVs may correlate with reduced overall HF energy.
** Predict delta spectral shape from source block spectral shape. HF/LF ratio of the delta may be correlated with the same ratio in its source blocks. Works well with decode-side fDCT.

==Negative results==

* Using Kurtosis for detecting text in a frame
** The idea was to detect a Bernouilli distribution but it's not robust and too noisy

Daala

2012-09-21T20:39:09Z

Rillian: /* Weekly meetings */ Post minutes from today's meeting.

Daala is the current working name of a next generation video codec— to be renamed once someone insists on something better. So far the best proposed alternative is PatentCake.

For now the purposes of this page is to collect notes about things which have been discussed in informal public IRC discussion about the next generation initiative. Participants in these discussions have included Timothy Terriberry, Jason Garrett-Glaser, Loren Merritt, Ben Schwartz, Greg Maxwell, and others.

See also: [https://xiph.org/daala/ https://xiph.org/daala/]

== Weekly meetings ==

We've been having weekly progress meetings on mumble.

* 2012 June 4 [https://people.xiph.org/~giles/2012/daala_20120604.txt minutes] (actually a work week)
* 2012 June 22 [https://people.xiph.org/~giles/2012/daala_20120622.txt minutes]
* 2012 June 29 [https://people.xiph.org/~giles/2012/daala_20120629.txt minutes] [https://people.xiph.org/~giles/2012/daala_20120629.opus recording]
* 2012 July 6 [https://people.xiph.org/~giles/2012/daala_20120706.txt minutes]
* 2012 July 13 [https://people.xiph.org/~giles/2012/daala_20120713.txt minutes]
* 2012 July 20 [https://people.xiph.org/~giles/2012/daala_20120720.txt minutes]
* 2012 July 27 [https://people.xiph.org/~giles/2012/daala_20120727.txt minutes]
* 2012 August 3 [https://people.xiph.org/~giles/2012/daala_20120803.txt minutes] [https://people.xiph.org/~giles/2012/daala_20120803.opus recording]
* 2012 August 10 [https://people.xiph.org/~giles/2012/daala_20120810.txt minutes]
* 2012 August 17 - no meeting
* 2012 August 24 - [https://people.xiph.org/~giles/2012/daala_20120824.txt minutes] [https://people.xiph.org/~giles/2012/daala_20120824.opus recording]
* 2012 August 31 - no meeting
* 2012 September 7 - no meeting
* 2012 September 14 - no meeting
* 2012 September 21 - [https://people.xiph.org/~giles/2012/daala_20120921.txt minutes]

= Techniques =

The discussed overall structure so far has been a variable size lapped-DCT block based codec with lapping done via pre/post filtering with a specially structured (lifting) linear phase transform along the edges along with overlapped block motion compensation and the expected trimmings. The lapping can be optimized for energy compaction and other useful properties, including invert-ability, and yields excellent results with efficient finite precision math.

Other components which have been discussed include:

==Techniques applicable to all frame types==
* Multisymbol arithmetic coding
** Timothy has some trial code showing speed-up proportional to the number of bits coded at once. (ec_test.c)
* Mode prediction using the previously decoded data, e.g. coding the mode using a probability function derived from trained predictors on the surrounding blocks.
** This will be terrible for robustness but may significantly reduce signalling overhead, allowing many more modes, and provide continuous adaptation between signalling free and fully signalled modes.
* Explore legendre polynomial basis transforms instead of DCT
** May have better perceptual properties and/or result in 'less compromised' efficient implementations.
* Coefficient domain prediction to allow efficient energy preserving quantization.
* Variable partition size/shape and the use of good predictors appears to remove most of the benefit of directional transforms.
** Perhaps 45deg is still useful?
** How does this change with partition sizes? Directional transforms are clearly not that useful with 4x4.
* Transform-post filtering to allow merging smaller transform blocks (like TF merging in CELT) may allow more flexible partitioning then outright using mixed block sizes.
* Perturbed quantization mode-signalling has been discussed but mostly laughed at. ;)
* Special block modes well suited to solid color/cartoon like content— avoiding ringing.
** Are pixel prediction modes too slow?
* In general— what markov random field techniques can be applied with acceptable performance. Any?
* Designed for parallel encode and decode within each frame
** Important because
*** the proposed techniques need a lot more CPU than H.264 and VP8 for both encode and decode
*** Moore's law for single-threaded throughput is dead. Future hardware is all multicore/GPU.
** Implies
*** Getting the order of application right for the lapping filters.
*** Mandatory slicing? Maybe some kind of multilevel entropy coding to reduce redundancy between slices while minimizing the single-threaded portion of decode.

* Using PVQ and energy conservation: see http://jmvalin.ca/video/video_pvq_v3.pdf

==Techniques applicable to inter frames==
* Using x264 as a test-bed Jason and Loren demonstrated 15% rate/distortion improvements from using 10-bit intermediaries and references, estimated as being 1/3rd from quality calculation in the 10-bit space, 1/3rd from the higher precision references, and 1/3rd from higher intermediate precision in calculations (e.g. MC filter processing).
** Increased reference precision competes for memory with increased number of references. The improvements demonstrated appear to be a greater win than increasing the reference count once there are four references or so.
* Super-resolution techniques for motion-compensation references have been discussed— in particular it appears that the half-pel location is where intelligent filtering matters the most so staged computation could be effectively used to allow more expensive filtering at that level.
** Edge-directed interpolation techniques might be effectively applied to increase motion compensation accuracy, but most of the techniques known to be very effective are too slow.
** Speculation has been offered that a significant part of MC inaccuracy may be due to blending in a physically incorrect (gamma-corrected) space, though no real conclusions were made. Academic papers on motion compensation accuracy seem to have ignored this issue.
* Timothy has an example code base for a variable partition size blocking-free motion compensation scheme which merges OBMC (overlapped block motion compensation) and CGI (control-grid interpolation) with an interesting prediction/sub-division scheme and whole-frame trellis optimization of motion vectors. (daala-exp)

==Basic features==
* YUV 4:4:4, 4:2:2 , 4:2:0 subsamplings, 8-bit, 10bit.
* Alpha channel — need testing material!
* 8-bit RGB compatible mode? (e.g. YCoCg, internally or at least flagging for it)
* Efficient 3D? — need testing material!
* Lossless?
** The value of this is disputable. If nothing else it's arguable that stuffing lossless into a lossy format may be the only way to get lossless into many people's hands. Also, see below
* Good support for decode side droppable frames?
** Hopefully the referencing structure will be flexible enough to enable this even if it's not an intentional feature.

==Frills==
* Optionally storing a checksum of the expected decoded frame for decoder/encoder mismatch detection.
* Expose the number of referential descendants of a given frame (or even the whole reference DAG) for most efficient allocation of FEC.

==Wingdings==
Crazy crap that might be interesting or at least fun to make fun of...
* >10bit?
** Use cases don't seem well enough defined yet. Significant complexity. Any prospective hardware developer may hire assassins.
** Possible compromise: the video reference structure contains a backbone that can be decoded at only N bits of depth (e.g. 10), and higher precisions are only supported outside of this reference chain.
**# Precision by truncation: decode is performed twice on each frame, identically, at low and high precision. The only difference between them is the bit-depth of the transform, or possibly of the transform and MC filters. Only low-precision outputs can be referenced by subsequent frames. Useful if high-precision content is still worth watching at low precision.
**# Precision by gamma: decode is performed once at low precision as normal. Then the output frame is converted to linear-light at high precision, after which another layer of residuals is added. The second layer can be permitted to reference previous high-precision frames... tricky to use both sets of references though. Useful if high precision is used for storing linear data, but people still want to watch it on "low-end" hardware.
* Some high end digital cameras are operating jpeg-derivatives in a special mode that keeps the image in the native linear RGB bayer format in order to avoid lossy/slow demosaicing on the camera. In particular this allows white balancing in post without excessive loss. Probably out of scope for Daala itself.
** Bayer, 4:2:0, 4:2:2, and Interlacing are all special cases of a more general pattern in which the output frames are decimated/subsampled in a regular fashion. All such subsamplings could be supported by a unified framework in which the video is always stored with all planes fully sampled, with a header indicating the recommended subsampling for display. In such cases, the encoder can regard the transform as highly overcomplete, and simply ignore unneeded coefficients (presumably by leaving high frequency residuals coded as zero). This structure would in effect turn the codec into a motion-compensated interpolating/deinterlacing filter. Whether this approach is sensible presumably depends in part on how the transform is structured. It would be especially easy if the transform's highest-frequencies were coded by a wavelet-like layer.
* Lossless intra-ability: The ability to losslessly rewrite any frame as an intra frame (perhaps with significant bitrate overhead) in order to make frame accurate cuts possible.
** Or best handled by making sure that containers have working pre-roll, but presumably common GOP sizes will be greater than the number of references so even if losslessly reencoding the references is expensive it may be cheaper than pre-roll. Do both?
** Can be had for 'free' if lossless is supported, plus the right header flags to restuff the references from lossless copies in a packed hidden frame.
** Use of explicitly (rather than staged) super-resolution and/or deeper references may make this functionality unattractive due to increased overhead.
* Internal overlays which could be swapped without re-encoding? (e.g. advertising, station ID). Could also be automatically generated by a Sufficiently Advanced™ encoder to improve efficiencies for static sprites over moving backgrounds.
** Complicates making the complexity bounded. No Sufficiently Advanced™ encoder likely to ever exist. But perhaps the station id/advertising uses fully justify this.
** Could be done externally to the video codec, but if so it's no likely to be useful for anyone ever.
* A secondary reference implementation in OpenCL, maintained throughout development, to make sure that the codec is GPU-friendly and can be done efficiently using OpenCL primitives.
* SWAR-friendly arithmetic. For example, choosing transform coefficients so that no intermediate product overflows 16 bits (tricky for signed values) can sometimes enable (e.g.) 4 parallel operations in one uint64_t. This can allow a pure C reference implementation to run faster, which is valuable for initial adoption and ports to new platforms.
* Parametric decode-side blur.
** Symmetrical blur in regions that are smooth on scales longer than the block size. Could be signaled or derived from observed DC values.
** Motion blur so that moving objects are blurred along the motion vector. May require coding a shutter speed parameter (0..1 as a fraction of the inter-frame interval).
* Fancy block property prediction. (Not clear how these prediction interact with intra pred)
** Predict block properties (quantizer, energy, etc.) from MV. (0,0) probably means small delta. Larger MV's may correspond to larger deltas ... although at low shutter speeds large MVs may correlate with reduced overall HF energy.
** Predict delta spectral shape from source block spectral shape. HF/LF ratio of the delta may be correlated with the same ratio in its source blocks. Works well with decode-side fDCT.

==Negative results==

* Using Kurtosis for detecting text in a frame
** The idea was to detect a Bernouilli distribution but it's not robust and too noisy

MatroskaOpus

2012-09-14T22:49:38Z

Rillian: /* DRAFT */ empty is the same as not present

== '''DRAFT''' ==

This is an encapsulation spec for the [[Opus]] codec in [[http://matroska.org/ Matroska]]. There are a number of outstanding functional issues with muxing Opus in Matroska, and until those are resolved, use of this spec is NOT RECOMMENDED.

- CodecID is A_OPUS
- SampleFrequecy is 48000
- Channels is number of output PCM channels
- CodecPrivate is the 'OpusHead' packet, identical to the Ogg mapping

The 'OpusHead' format is defined by the [[http://tools.ietf.org/html/draft-terriberry-oggopus Ogg Opus]] mapping. In particular it includes pre-skip, gain, and the channel mapping table required for correct surround output.

The second 'OpusTags' header packet from Ogg Opus is not used in the Matroska encapsulation. Matroska has its own system for tag metadata, and this avoids duplicating it and the need for sub-framing to index multiple packets within the CodecPrivate element.

If the CodecPrivate is empty or not present and Channels is 1 or 2, players MAY treat it as a sane set of defaults, I guess. e.g. channel mapping family 0, no pre-skip or gain. For Channels > 2 the track MUST be rejected, since there's no way to map the encoded substreams to channels.

== Open Questions ==

Seeking in Opus files requires a pre-roll (recommended to be at least 80 ms). However, currently Matroska requires its index entries to point directly to the data whose timestamp matches the corresponding seek point, not some place arbitrarily before that timestamp. These two requirements are incompatible, and mean that seeking in Opus will be broken in all existing Matroska software. In particularly unlucky cases (e.g., around a transient), playing back audio decoded without any pre-roll can produce extremely loud (possibly equipment-damaging) results. We need a new element to signal this, e.g. Track::TrackEntry::PreRoll.

Should we say muxers MAY or SHOULD NOT produce simple streams without filling in CodecPrivate?

How does the OpusHead pre-skip field interact with the timestamps? The SimpleBlock timestamp is signed 16 bits, so the format can signal about half of the pre-skip if playback timestamps are to start at zero. Moritz suggests this won't work because the resolution of the timestamps is controlled by the muxer, so the SimpleBlock timestamp offset isn't sample accurate anyway.[[http://lists.matroska.org/pipermail/matroska-devel/2012-September/004254.html ref]]

One could set an incorrect timestamp on the skipped blocks, and rely on the decoder to drop them based on the OpusHead preskip value. As long as the initial blocks are timestamped <= start of output this shouldn't affect seeking.

How important is it that timestamps start at zero in a Matroska file?

The SimpleBlock structure also has an 'invisible' bit, which tells the player to decode, but not display, the contained frames. This lets the muxer signal the pre-skip semantics with frame accuracy, but not sample accuracy. If players implement this it will at least help with sync. Libav does not appear to support the invisible bit.

How can sample-accurate end-time trimming work in Matroska? Currently all software encapsulating Vorbis in Matroska is broken in this regard, and muxing a Vorbis file in Matroska causes it to get longer (i.e., produce more audio output than the original Ogg file). It would be unfortunate to repeat this disaster for Opus. This needs a new element specifying the number of samples to trim, perhaps a new BlockGroup child.

If new elements are required, can they be defined so as to enable correct seeking in rolling intra (a.k.a intra refresh) video as well?

MatroskaOpus

2012-09-14T21:45:47Z

Rillian: /* Open Questions */ updates from today's dicussion with Moritz

== '''DRAFT''' ==

This is an encapsulation spec for the [[Opus]] codec in [[http://matroska.org/ Matroska]]. There are a number of outstanding functional issues with muxing Opus in Matroska, and until those are resolved, use of this spec is NOT RECOMMENDED.

- CodecID is A_OPUS
- SampleFrequecy is 48000
- Channels is number of output PCM channels
- CodecPrivate is the 'OpusHead' packet, identical to the Ogg mapping

The 'OpusHead' format is defined by the [[http://tools.ietf.org/html/draft-terriberry-oggopus Ogg Opus]] mapping. In particular it includes pre-skip, gain, and the channel mapping table required for correct surround output.

The second 'OpusTags' header packet from Ogg Opus is not used in the Matroska encapsulation. Matroska has its own system for tag metadata, and this avoids duplicating it and the need for sub-framing to index multiple packets within the CodecPrivate element.

If the CodecPrivate is empty and Channels is 1 or 2, players MAY treat it as a sane set of defaults, I guess. e.g. channel mapping family 0, no pre-skip or gain. For Channels > 2 the track MUST be rejected, since there's no way to map the encoded substreams to channels.

== Open Questions ==

Seeking in Opus files requires a pre-roll (recommended to be at least 80 ms). However, currently Matroska requires its index entries to point directly to the data whose timestamp matches the corresponding seek point, not some place arbitrarily before that timestamp. These two requirements are incompatible, and mean that seeking in Opus will be broken in all existing Matroska software. In particularly unlucky cases (e.g., around a transient), playing back audio decoded without any pre-roll can produce extremely loud (possibly equipment-damaging) results. We need a new element to signal this, e.g. Track::TrackEntry::PreRoll.

Should we say muxers MAY or SHOULD NOT produce simple streams without filling in CodecPrivate?

How does the OpusHead pre-skip field interact with the timestamps? The SimpleBlock timestamp is signed 16 bits, so the format can signal about half of the pre-skip if playback timestamps are to start at zero. Moritz suggests this won't work because the resolution of the timestamps is controlled by the muxer, so the SimpleBlock timestamp offset isn't sample accurate anyway.[[http://lists.matroska.org/pipermail/matroska-devel/2012-September/004254.html ref]]

One could set an incorrect timestamp on the skipped blocks, and rely on the decoder to drop them based on the OpusHead preskip value. As long as the initial blocks are timestamped <= start of output this shouldn't affect seeking.

How important is it that timestamps start at zero in a Matroska file?

The SimpleBlock structure also has an 'invisible' bit, which tells the player to decode, but not display, the contained frames. This lets the muxer signal the pre-skip semantics with frame accuracy, but not sample accuracy. If players implement this it will at least help with sync. Libav does not appear to support the invisible bit.

How can sample-accurate end-time trimming work in Matroska? Currently all software encapsulating Vorbis in Matroska is broken in this regard, and muxing a Vorbis file in Matroska causes it to get longer (i.e., produce more audio output than the original Ogg file). It would be unfortunate to repeat this disaster for Opus. This needs a new element specifying the number of samples to trim, perhaps a new BlockGroup child.

MatroskaOpus

2012-09-14T17:20:29Z

Rillian: /* Open Questions */ remove duplicate paragraph

== '''DRAFT''' ==

This is an encapsulation spec for the [[Opus]] codec in [[http://matroska.org/ Matroska]]. There are a number of outstanding functional issues with muxing Opus in Matroska, and until those are resolved, use of this spec is NOT RECOMMENDED.

- CodecID is A_OPUS
- SampleFrequecy is 48000
- Channels is number of output PCM channels
- CodecPrivate is the 'OpusHead' packet, identical to the Ogg mapping

The 'OpusHead' format is defined by the [[http://tools.ietf.org/html/draft-terriberry-oggopus Ogg Opus]] mapping. In particular it includes pre-skip, gain, and the channel mapping table required for correct surround output.

The second 'OpusTags' header packet from Ogg Opus is not used in the Matroska encapsulation. Matroska has its own system for tag metadata, and this avoids duplicating it and the need for sub-framing to index multiple packets within the CodecPrivate element.

If the CodecPrivate is empty and Channels is 1 or 2, players MAY treat it as a sane set of defaults, I guess. e.g. channel mapping family 0, no pre-skip or gain. For Channels > 2 the track MUST be rejected, since there's no way to map the encoded substreams to channels.

== Open Questions ==

Seeking in Opus files requires a pre-roll (recommended to be at least 80 ms). However, currently Matroska requires its index entries to point directly to the data whose timestamp matches the corresponding seek point, not some place arbitrarily before that timestamp. These two requirements are incompatible, and mean that seeking in Opus will be broken in all existing Matroska software. In particularly unlucky cases (e.g., around a transient), playing back audio decoded without any pre-roll can produce extremely loud (possibly equipment-damaging) results. We need a new element to signal this, e.g. Track::TrackEntry::PreRoll.

Should we say muxers MAY or SHOULD NOT produce simple streams without filling in CodecPrivate?

How does the OpusHead pre-skip field interact with the timestamps? The SimpleBlock timestamp is signed 16 bits, so the format can signal about half of the pre-skip if playback timestamps are to start at zero.

One could set an incorrect timestamp on the skipped blocks, and rely on the decoder to drop them based on the OpusHead preskip value. As long as the initial blocks are timestamped <= start of output this shouldn't affect seeking.

How important is it that timestamps start at zero in a Matroska file?

The SimpleBlock structure also has an 'invisible' bit, which tells the player to decode, but not display, the contained frames. This lets the muxer signal the pre-skip semantics with frame accuracy, but not sample accuracy. If players implement this it will at least help with sync. Libav does not appear to support the invisible bit.

How can sample-accurate end-time trimming work in Matroska? Currently all software encapsulating Vorbis in Matroska is broken in this regard, and muxing a Vorbis file in Matroska causes it to get longer (i.e., produce more audio output than the original Ogg file). It would be unfortunate to repeat this disaster for Opus.

MatroskaOpus

2012-09-14T17:19:19Z

Rillian: /* Open Questions */ typo

== '''DRAFT''' ==

This is an encapsulation spec for the [[Opus]] codec in [[http://matroska.org/ Matroska]]. There are a number of outstanding functional issues with muxing Opus in Matroska, and until those are resolved, use of this spec is NOT RECOMMENDED.

- CodecID is A_OPUS
- SampleFrequecy is 48000
- Channels is number of output PCM channels
- CodecPrivate is the 'OpusHead' packet, identical to the Ogg mapping

The 'OpusHead' format is defined by the [[http://tools.ietf.org/html/draft-terriberry-oggopus Ogg Opus]] mapping. In particular it includes pre-skip, gain, and the channel mapping table required for correct surround output.

The second 'OpusTags' header packet from Ogg Opus is not used in the Matroska encapsulation. Matroska has its own system for tag metadata, and this avoids duplicating it and the need for sub-framing to index multiple packets within the CodecPrivate element.

If the CodecPrivate is empty and Channels is 1 or 2, players MAY treat it as a sane set of defaults, I guess. e.g. channel mapping family 0, no pre-skip or gain. For Channels > 2 the track MUST be rejected, since there's no way to map the encoded substreams to channels.

== Open Questions ==

Seeking in Opus files requires a pre-roll (recommended to be at least 80 ms). However, currently Matroska requires its index entries to point directly to the data whose timestamp matches the corresponding seek point, not some place arbitrarily before that timestamp. These two requirements are incompatible, and mean that seeking in Opus will be broken in all existing Matroska software. In particularly unlucky cases (e.g., around a transient), playing back audio decoded without any pre-roll can produce extremely loud (possibly equipment-damaging) results. We need a new element to signal this, e.g. Track::TrackEntry::PreRoll.

Should we say muxers MAY or SHOULD NOT produce simple streams without filling in CodecPrivate?

How does the OpusHead pre-skip field interact with the timestamps? The SimpleBlock timestamp is signed 16 bits, so the format can signal about half of the pre-skip if playback timestamps are to start at zero.

One could set an incorrect timestamp on the skipped blocks, and rely on the decoder to drop them based on the OpusHead preskip value. As long as the initial blocks are timestamped <= start of output this shouldn't affect seeking.

How important is it that timestamps start at zero in a Matroska file?

The SimpleBlock structure also has an 'invisible' bit, which tells the player to decode, but not display, the contained frames. This lets the muxer signal the pre-skip semantics with frame accuracy, but not sample accuracy. If players implement this it will at least help with sync. Libav does not appear to support the invisible bit.

Seeking in Opus files requires a pre-roll (recommended to be at least 80 ms). However, currently Matroska requires its index entries to point directly to the data whose timestamp matches the corresponding seek point, not some place arbitrarily before that timestamp. These two requirements are incompatible, and mean that seeking in Opus will be broken in all existing Matroska software. In particularly unlucky cases (e.g., around a transient), playing back audio decoded without any pre-roll can produce extremely loud (possibly equipment-damaging) results. We need a new element to signal this, e.g. Track::TrackEntry::PreRoll.

How can sample-accurate end-time trimming work in Matroska? Currently all software encapsulating Vorbis in Matroska is broken in this regard, and muxing a Vorbis file in Matroska causes it to get longer (i.e., produce more audio output than the original Ogg file). It would be unfortunate to repeat this disaster for Opus.

MatroskaOpus

2012-09-14T17:16:44Z

Rillian: /* DRAFT */ Link to the OggOpus draft

== '''DRAFT''' ==

This is an encapsulation spec for the [[Opus]] codec in [[http://matroska.org/ Matroska]]. There are a number of outstanding functional issues with muxing Opus in Matroska, and until those are resolved, use of this spec is NOT RECOMMENDED.

- CodecID is A_OPUS
- SampleFrequecy is 48000
- Channels is number of output PCM channels
- CodecPrivate is the 'OpusHead' packet, identical to the Ogg mapping

The 'OpusHead' format is defined by the [[http://tools.ietf.org/html/draft-terriberry-oggopus Ogg Opus]] mapping. In particular it includes pre-skip, gain, and the channel mapping table required for correct surround output.

The second 'OpusTags' header packet from Ogg Opus is not used in the Matroska encapsulation. Matroska has its own system for tag metadata, and this avoids duplicating it and the need for sub-framing to index multiple packets within the CodecPrivate element.

If the CodecPrivate is empty and Channels is 1 or 2, players MAY treat it as a sane set of defaults, I guess. e.g. channel mapping family 0, no pre-skip or gain. For Channels > 2 the track MUST be rejected, since there's no way to map the encoded substreams to channels.

== Open Questions ==

Seeking in Opus files requires a pre-roll (recommended to be at least 80 ms). However, currently Matroska requires its index entries to point directly to the data whose timestamp matches the corresponding seek point, not some place arbitrarily before that timestamp. These two requirements are incompatible, and mean that seeking in Opus will be broken in all existing Matroska software. In particularly unlucky cases (e.g., around a transient), playing back audio decoded without any pre-roll can produce extremely loud (possibly equipment-damaging) results. We need a new element to signal this, e.g. Track::TrackEntry::PreRoll.

Should we say muxers MAY or SHOULD NOT produce simple streams without filling in CodecPrivate?

How does the OpusHead pre-skip field interact with the timestamps? The SimpleBlock timestamp is signed 16 bits, so the format can signal about half of the pre-skip if playback timestamps are to start at zero.

One could set an incorrect timestamp on the skipped blocks, and rely on the decoder to drop them based on the OpusHead preskip value. As long as the initial blocks are timestamped <= start of output this shouldn't affect seeking.

How important is it that timestamps start at zero in a Matroska file?

The SimpleBlock structure also has an 'invisible' bit, which tell the player to decode, but not display, the contained frames. This lets the muxer signal the pre-skip semantics with frame accuracy, but not sample accuracy. If players implement this it will at least help with sync. Libav does not appear to support the invisible bit.

Seeking in Opus files requires a pre-roll (recommended to be at least 80 ms). However, currently Matroska requires its index entries to point directly to the data whose timestamp matches the corresponding seek point, not some place arbitrarily before that timestamp. These two requirements are incompatible, and mean that seeking in Opus will be broken in all existing Matroska software. In particularly unlucky cases (e.g., around a transient), playing back audio decoded without any pre-roll can produce extremely loud (possibly equipment-damaging) results. We need a new element to signal this, e.g. Track::TrackEntry::PreRoll.

How can sample-accurate end-time trimming work in Matroska? Currently all software encapsulating Vorbis in Matroska is broken in this regard, and muxing a Vorbis file in Matroska causes it to get longer (i.e., produce more audio output than the original Ogg file). It would be unfortunate to repeat this disaster for Opus.

MatroskaOpus

2012-09-14T17:11:24Z

Rillian: /* DRAFT */ remove unhelpful quotes

== '''DRAFT''' ==

This is an encapsulation spec for the [[Opus]] codec in [[http://matroska.org/ Matroska]]. There are a number of outstanding functional issues with muxing Opus in Matroska, and until those are resolved, use of this spec is NOT RECOMMENDED.

- CodecID is A_OPUS
- SampleFrequecy is 48000
- Channels is number of output PCM channels
- CodecPrivate is the 'OpusHead' packet, identical to the OggOpus mapping

The second 'OpusTags' header packet from OggOpus is not used in the Matroska encapsulation. Matroska has its own system for tag metadata, and this avoids duplicating it and the need for sub-framing to index multiple packets within the CodecPrivate element.

If the CodecPrivate is empty and Channels is 1 or 2, players MAY treat it as a sane set of defaults, I guess. e.g. channel mapping family 0, no pre-skip or gain. For Channels > 2 the track MUST be rejected, since there's no way to map the encoded substreams to channels.

== Open Questions ==

Seeking in Opus files requires a pre-roll (recommended to be at least 80 ms). However, currently Matroska requires its index entries to point directly to the data whose timestamp matches the corresponding seek point, not some place arbitrarily before that timestamp. These two requirements are incompatible, and mean that seeking in Opus will be broken in all existing Matroska software. In particularly unlucky cases (e.g., around a transient), playing back audio decoded without any pre-roll can produce extremely loud (possibly equipment-damaging) results. We need a new element to signal this, e.g. Track::TrackEntry::PreRoll.

Should we say muxers MAY or SHOULD NOT produce simple streams without filling in CodecPrivate?

How does the OpusHead pre-skip field interact with the timestamps? The SimpleBlock timestamp is signed 16 bits, so the format can signal about half of the pre-skip if playback timestamps are to start at zero.

One could set an incorrect timestamp on the skipped blocks, and rely on the decoder to drop them based on the OpusHead preskip value. As long as the initial blocks are timestamped <= start of output this shouldn't affect seeking.

How important is it that timestamps start at zero in a Matroska file?

The SimpleBlock structure also has an 'invisible' bit, which tell the player to decode, but not display, the contained frames. This lets the muxer signal the pre-skip semantics with frame accuracy, but not sample accuracy. If players implement this it will at least help with sync. Libav does not appear to support the invisible bit.

Seeking in Opus files requires a pre-roll (recommended to be at least 80 ms). However, currently Matroska requires its index entries to point directly to the data whose timestamp matches the corresponding seek point, not some place arbitrarily before that timestamp. These two requirements are incompatible, and mean that seeking in Opus will be broken in all existing Matroska software. In particularly unlucky cases (e.g., around a transient), playing back audio decoded without any pre-roll can produce extremely loud (possibly equipment-damaging) results. We need a new element to signal this, e.g. Track::TrackEntry::PreRoll.

How can sample-accurate end-time trimming work in Matroska? Currently all software encapsulating Vorbis in Matroska is broken in this regard, and muxing a Vorbis file in Matroska causes it to get longer (i.e., produce more audio output than the original Ogg file). It would be unfortunate to repeat this disaster for Opus.

MatroskaOpus

2012-09-14T17:10:24Z

Rillian: /* Questions */ open questions are open

== '''DRAFT''' ==

This is an encapsulation spec for the [[Opus]] codec in [[http://matroska.org/ Matroska]]. There are a number of outstanding functional issues with muxing Opus in Matroska, and until those are resolved, use of this spec is NOT RECOMMENDED.

- CodecID is A_OPUS
- SampleFrequecy is 48000
- Channels is number of output PCM channels
- CodecPrivate is the 'OpusHead' packet, identical to the OggOpus mapping

The second 'OpusTags' header packet from OggOpus is not used in the Matroska encapsulation. Matroska has its own system for tag metadata, and this avoids duplicating it and the need for sub-framing to index multiple packets within the CodecPrivate element.

If the CodecPrivate is empty and 'Channels' is 1 or 2, players MAY treat it as a sane set of defaults, I guess. e.g. channel mapping family 0, no pre-skip or gain. For 'Channels > 2' the track MUST be rejected, since there's no way to map the encoded substreams to channels.

== Open Questions ==

Seeking in Opus files requires a pre-roll (recommended to be at least 80 ms). However, currently Matroska requires its index entries to point directly to the data whose timestamp matches the corresponding seek point, not some place arbitrarily before that timestamp. These two requirements are incompatible, and mean that seeking in Opus will be broken in all existing Matroska software. In particularly unlucky cases (e.g., around a transient), playing back audio decoded without any pre-roll can produce extremely loud (possibly equipment-damaging) results. We need a new element to signal this, e.g. Track::TrackEntry::PreRoll.

Should we say muxers MAY or SHOULD NOT produce simple streams without filling in CodecPrivate?

How does the OpusHead pre-skip field interact with the timestamps? The SimpleBlock timestamp is signed 16 bits, so the format can signal about half of the pre-skip if playback timestamps are to start at zero.

One could set an incorrect timestamp on the skipped blocks, and rely on the decoder to drop them based on the OpusHead preskip value. As long as the initial blocks are timestamped <= start of output this shouldn't affect seeking.

How important is it that timestamps start at zero in a Matroska file?

The SimpleBlock structure also has an 'invisible' bit, which tell the player to decode, but not display, the contained frames. This lets the muxer signal the pre-skip semantics with frame accuracy, but not sample accuracy. If players implement this it will at least help with sync. Libav does not appear to support the invisible bit.

Seeking in Opus files requires a pre-roll (recommended to be at least 80 ms). However, currently Matroska requires its index entries to point directly to the data whose timestamp matches the corresponding seek point, not some place arbitrarily before that timestamp. These two requirements are incompatible, and mean that seeking in Opus will be broken in all existing Matroska software. In particularly unlucky cases (e.g., around a transient), playing back audio decoded without any pre-roll can produce extremely loud (possibly equipment-damaging) results. We need a new element to signal this, e.g. Track::TrackEntry::PreRoll.

How can sample-accurate end-time trimming work in Matroska? Currently all software encapsulating Vorbis in Matroska is broken in this regard, and muxing a Vorbis file in Matroska causes it to get longer (i.e., produce more audio output than the original Ogg file). It would be unfortunate to repeat this disaster for Opus.

MatroskaOpus

2012-09-14T17:10:04Z

Rillian: /* Questions */ update questions with the PreRoll element idea

== '''DRAFT''' ==

This is an encapsulation spec for the [[Opus]] codec in [[http://matroska.org/ Matroska]]. There are a number of outstanding functional issues with muxing Opus in Matroska, and until those are resolved, use of this spec is NOT RECOMMENDED.

- CodecID is A_OPUS
- SampleFrequecy is 48000
- Channels is number of output PCM channels
- CodecPrivate is the 'OpusHead' packet, identical to the OggOpus mapping

The second 'OpusTags' header packet from OggOpus is not used in the Matroska encapsulation. Matroska has its own system for tag metadata, and this avoids duplicating it and the need for sub-framing to index multiple packets within the CodecPrivate element.

If the CodecPrivate is empty and 'Channels' is 1 or 2, players MAY treat it as a sane set of defaults, I guess. e.g. channel mapping family 0, no pre-skip or gain. For 'Channels > 2' the track MUST be rejected, since there's no way to map the encoded substreams to channels.

== Questions ==

Seeking in Opus files requires a pre-roll (recommended to be at least 80 ms). However, currently Matroska requires its index entries to point directly to the data whose timestamp matches the corresponding seek point, not some place arbitrarily before that timestamp. These two requirements are incompatible, and mean that seeking in Opus will be broken in all existing Matroska software. In particularly unlucky cases (e.g., around a transient), playing back audio decoded without any pre-roll can produce extremely loud (possibly equipment-damaging) results. We need a new element to signal this, e.g. Track::TrackEntry::PreRoll.

Should we say muxers MAY or SHOULD NOT produce simple streams without filling in CodecPrivate?

How does the OpusHead pre-skip field interact with the timestamps? The SimpleBlock timestamp is signed 16 bits, so the format can signal about half of the pre-skip if playback timestamps are to start at zero.

One could set an incorrect timestamp on the skipped blocks, and rely on the decoder to drop them based on the OpusHead preskip value. As long as the initial blocks are timestamped <= start of output this shouldn't affect seeking.

How important is it that timestamps start at zero in a Matroska file?

The SimpleBlock structure also has an 'invisible' bit, which tell the player to decode, but not display, the contained frames. This lets the muxer signal the pre-skip semantics with frame accuracy, but not sample accuracy. If players implement this it will at least help with sync. Libav does not appear to support the invisible bit.

Seeking in Opus files requires a pre-roll (recommended to be at least 80 ms). However, currently Matroska requires its index entries to point directly to the data whose timestamp matches the corresponding seek point, not some place arbitrarily before that timestamp. These two requirements are incompatible, and mean that seeking in Opus will be broken in all existing Matroska software. In particularly unlucky cases (e.g., around a transient), playing back audio decoded without any pre-roll can produce extremely loud (possibly equipment-damaging) results. We need a new element to signal this, e.g. Track::TrackEntry::PreRoll.

How can sample-accurate end-time trimming work in Matroska? Currently all software encapsulating Vorbis in Matroska is broken in this regard, and muxing a Vorbis file in Matroska causes it to get longer (i.e., produce more audio output than the original Ogg file). It would be unfortunate to repeat this disaster for Opus.

MatroskaOpus

2012-09-14T17:08:09Z

Rillian: /* DRAFT */ remove the simpler mapping since we have consensus for the OpusHead proposal

== '''DRAFT''' ==

This is an encapsulation spec for the [[Opus]] codec in [[http://matroska.org/ Matroska]]. There are a number of outstanding functional issues with muxing Opus in Matroska, and until those are resolved, use of this spec is NOT RECOMMENDED.

- CodecID is A_OPUS
- SampleFrequecy is 48000
- Channels is number of output PCM channels
- CodecPrivate is the 'OpusHead' packet, identical to the OggOpus mapping

The second 'OpusTags' header packet from OggOpus is not used in the Matroska encapsulation. Matroska has its own system for tag metadata, and this avoids duplicating it and the need for sub-framing to index multiple packets within the CodecPrivate element.

If the CodecPrivate is empty and 'Channels' is 1 or 2, players MAY treat it as a sane set of defaults, I guess. e.g. channel mapping family 0, no pre-skip or gain. For 'Channels > 2' the track MUST be rejected, since there's no way to map the encoded substreams to channels.

== Questions ==

Should we say muxers MAY or SHOULD NOT produce simple streams without filling in CodecPrivate?

How does the OpusHead pre-skip field interact with the timestamps? The SimpleBlock timestamp is signed 16 bits, so the format can signal about half of the pre-skip if playback timestamps are to start at zero.

One could set an incorrect timestamp on the skipped blocks, and rely on the decoder to drop them based on the OpusHead preskip value. As long as the initial blocks are timestamped <= start of output this shouldn't affect seeking.

How important is it that timestamps start at zero in a Matroska file?

The SimpleBlock structure also has an 'invisible' bit, which tell the player to decode, but not display, the contained frames. This lets the muxer signal the pre-skip semantics with frame accuracy, but not sample accuracy. If players implement this it will at least help with sync. Libav does not appear to support the invisible bit.

Seeking in Opus files requires a pre-roll (recommended to be at least 80 ms). However, currently Matroska requires its index entries to point directly to the data whose timestamp matches the corresponding seek point, not some place arbitrarily before that timestamp. These two requirements are incompatible, and mean that seeking in Opus will be broken in all existing Matroska software. In particularly unlucky cases (e.g., around a transient), playing back audio decoded without any pre-roll can produce extremely loud (possibly equipment-damaging) results.

How can sample-accurate end-time trimming work in Matroska? Currently all software encapsulating Vorbis in Matroska is broken in this regard, and muxing a Vorbis file in Matroska causes it to get longer (i.e., produce more audio output than the original Ogg file). It would be unfortunate to repeat this disaster for Opus.

Daala

2012-09-08T17:29:48Z

Rillian: /* Weekly meetings */ no meeting last this week: travel and Opus work

Daala is the current working name of a next generation video codec— to be renamed once someone insists on something better. So far the best proposed alternative is PatentCake.

For now the purposes of this page is to collect notes about things which have been discussed in informal public IRC discussion about the next generation initiative. Participants in these discussions have included Timothy Terriberry, Jason Garrett-Glaser, Loren Merritt, Ben Schwartz, Greg Maxwell, and others.

See also: [https://xiph.org/daala/ https://xiph.org/daala/]

== Weekly meetings ==

We've been having weekly progress meetings on mumble.

* 2012 June 4 [https://people.xiph.org/~giles/2012/daala_20120604.txt minutes] (actually a work week)
* 2012 June 22 [https://people.xiph.org/~giles/2012/daala_20120622.txt minutes]
* 2012 June 29 [https://people.xiph.org/~giles/2012/daala_20120629.txt minutes] [https://people.xiph.org/~giles/2012/daala_20120629.opus recording]
* 2012 July 6 [https://people.xiph.org/~giles/2012/daala_20120706.txt minutes]
* 2012 July 13 [https://people.xiph.org/~giles/2012/daala_20120713.txt minutes]
* 2012 July 20 [https://people.xiph.org/~giles/2012/daala_20120720.txt minutes]
* 2012 July 27 [https://people.xiph.org/~giles/2012/daala_20120727.txt minutes]
* 2012 August 3 [https://people.xiph.org/~giles/2012/daala_20120803.txt minutes] [https://people.xiph.org/~giles/2012/daala_20120803.opus recording]
* 2012 August 10 [https://people.xiph.org/~giles/2012/daala_20120810.txt minutes]
* 2012 August 17 - no meeting
* 2012 August 24 - [https://people.xiph.org/~giles/2012/daala_20120824.txt minutes] [https://people.xiph.org/~giles/2012/daala_20120824.opus recording]
* 2012 August 31 - no meeting
* 2012 September 7 - no meeting

= Techniques =

The discussed overall structure so far has been a variable size lapped-DCT block based codec with lapping done via pre/post filtering with a specially structured (lifting) linear phase transform along the edges along with overlapped block motion compensation and the expected trimmings. The lapping can be optimized for energy compaction and other useful properties, including invert-ability, and yields excellent results with efficient finite precision math.

Other components which have been discussed include:

==Techniques applicable to all frame types==
* Multisymbol arithmetic coding
** Timothy has some trial code showing speed-up proportional to the number of bits coded at once. (ec_test.c)
* Mode prediction using the previously decoded data, e.g. coding the mode using a probability function derived from trained predictors on the surrounding blocks.
** This will be terrible for robustness but may significantly reduce signalling overhead, allowing many more modes, and provide continuous adaptation between signalling free and fully signalled modes.
* Explore legendre polynomial basis transforms instead of DCT
** May have better perceptual properties and/or result in 'less compromised' efficient implementations.
* Coefficient domain prediction to allow efficient energy preserving quantization.
* Variable partition size/shape and the use of good predictors appears to remove most of the benefit of directional transforms.
** Perhaps 45deg is still useful?
** How does this change with partition sizes? Directional transforms are clearly not that useful with 4x4.
* Transform-post filtering to allow merging smaller transform blocks (like TF merging in CELT) may allow more flexible partitioning then outright using mixed block sizes.
* Perturbed quantization mode-signalling has been discussed but mostly laughed at. ;)
* Special block modes well suited to solid color/cartoon like content— avoiding ringing.
** Are pixel prediction modes too slow?
* In general— what markov random field techniques can be applied with acceptable performance. Any?
* Designed for parallel encode and decode within each frame
** Important because
*** the proposed techniques need a lot more CPU than H.264 and VP8 for both encode and decode
*** Moore's law for single-threaded throughput is dead. Future hardware is all multicore/GPU.
** Implies
*** Getting the order of application right for the lapping filters.
*** Mandatory slicing? Maybe some kind of multilevel entropy coding to reduce redundancy between slices while minimizing the single-threaded portion of decode.

* Using PVQ and energy conservation: see http://jmvalin.ca/video/video_pvq_v3.pdf

==Techniques applicable to inter frames==
* Using x264 as a test-bed Jason and Loren demonstrated 15% rate/distortion improvements from using 10-bit intermediaries and references, estimated as being 1/3rd from quality calculation in the 10-bit space, 1/3rd from the higher precision references, and 1/3rd from higher intermediate precision in calculations (e.g. MC filter processing).
** Increased reference precision competes for memory with increased number of references. The improvements demonstrated appear to be a greater win than increasing the reference count once there are four references or so.
* Super-resolution techniques for motion-compensation references have been discussed— in particular it appears that the half-pel location is where intelligent filtering matters the most so staged computation could be effectively used to allow more expensive filtering at that level.
** Edge-directed interpolation techniques might be effectively applied to increase motion compensation accuracy, but most of the techniques known to be very effective are too slow.
** Speculation has been offered that a significant part of MC inaccuracy may be due to blending in a physically incorrect (gamma-corrected) space, though no real conclusions were made. Academic papers on motion compensation accuracy seem to have ignored this issue.
* Timothy has an example code base for a variable partition size blocking-free motion compensation scheme which merges OBMC (overlapped block motion compensation) and CGI (control-grid interpolation) with an interesting prediction/sub-division scheme and whole-frame trellis optimization of motion vectors. (daala-exp)

==Basic features==
* YUV 4:4:4, 4:2:2 , 4:2:0 subsamplings, 8-bit, 10bit.
* Alpha channel — need testing material!
* 8-bit RGB compatible mode? (e.g. YCoCg, internally or at least flagging for it)
* Efficient 3D? — need testing material!
* Lossless?
** The value of this is disputable. If nothing else it's arguable that stuffing lossless into a lossy format may be the only way to get lossless into many people's hands. Also, see below
* Good support for decode side droppable frames?
** Hopefully the referencing structure will be flexible enough to enable this even if it's not an intentional feature.

==Frills==
* Optionally storing a checksum of the expected decoded frame for decoder/encoder mismatch detection.
* Expose the number of referential descendants of a given frame (or even the whole reference DAG) for most efficient allocation of FEC.

==Wingdings==
Crazy crap that might be interesting or at least fun to make fun of...
* >10bit?
** Use cases don't seem well enough defined yet. Significant complexity. Any prospective hardware developer may hire assassins.
** Possible compromise: the video reference structure contains a backbone that can be decoded at only N bits of depth (e.g. 10), and higher precisions are only supported outside of this reference chain.
**# Precision by truncation: decode is performed twice on each frame, identically, at low and high precision. The only difference between them is the bit-depth of the transform, or possibly of the transform and MC filters. Only low-precision outputs can be referenced by subsequent frames. Useful if high-precision content is still worth watching at low precision.
**# Precision by gamma: decode is performed once at low precision as normal. Then the output frame is converted to linear-light at high precision, after which another layer of residuals is added. The second layer can be permitted to reference previous high-precision frames... tricky to use both sets of references though. Useful if high precision is used for storing linear data, but people still want to watch it on "low-end" hardware.
* Some high end digital cameras are operating jpeg-derivatives in a special mode that keeps the image in the native linear RGB bayer format in order to avoid lossy/slow demosaicing on the camera. In particular this allows white balancing in post without excessive loss. Probably out of scope for Daala itself.
** Bayer, 4:2:0, 4:2:2, and Interlacing are all special cases of a more general pattern in which the output frames are decimated/subsampled in a regular fashion. All such subsamplings could be supported by a unified framework in which the video is always stored with all planes fully sampled, with a header indicating the recommended subsampling for display. In such cases, the encoder can regard the transform as highly overcomplete, and simply ignore unneeded coefficients (presumably by leaving high frequency residuals coded as zero). This structure would in effect turn the codec into a motion-compensated interpolating/deinterlacing filter. Whether this approach is sensible presumably depends in part on how the transform is structured. It would be especially easy if the transform's highest-frequencies were coded by a wavelet-like layer.
* Lossless intra-ability: The ability to losslessly rewrite any frame as an intra frame (perhaps with significant bitrate overhead) in order to make frame accurate cuts possible.
** Or best handled by making sure that containers have working pre-roll, but presumably common GOP sizes will be greater than the number of references so even if losslessly reencoding the references is expensive it may be cheaper than pre-roll. Do both?
** Can be had for 'free' if lossless is supported, plus the right header flags to restuff the references from lossless copies in a packed hidden frame.
** Use of explicitly (rather than staged) super-resolution and/or deeper references may make this functionality unattractive due to increased overhead.
* Internal overlays which could be swapped without re-encoding? (e.g. advertising, station ID). Could also be automatically generated by a Sufficiently Advanced™ encoder to improve efficiencies for static sprites over moving backgrounds.
** Complicates making the complexity bounded. No Sufficiently Advanced™ encoder likely to ever exist. But perhaps the station id/advertising uses fully justify this.
** Could be done externally to the video codec, but if so it's no likely to be useful for anyone ever.
* A secondary reference implementation in OpenCL, maintained throughout development, to make sure that the codec is GPU-friendly and can be done efficiently using OpenCL primitives.
* SWAR-friendly arithmetic. For example, choosing transform coefficients so that no intermediate product overflows 16 bits (tricky for signed values) can sometimes enable (e.g.) 4 parallel operations in one uint64_t. This can allow a pure C reference implementation to run faster, which is valuable for initial adoption and ports to new platforms.
* Parametric decode-side blur.
** Symmetrical blur in regions that are smooth on scales longer than the block size. Could be signaled or derived from observed DC values.
** Motion blur so that moving objects are blurred along the motion vector. May require coding a shutter speed parameter (0..1 as a fraction of the inter-frame interval).
* Fancy block property prediction. (Not clear how these prediction interact with intra pred)
** Predict block properties (quantizer, energy, etc.) from MV. (0,0) probably means small delta. Larger MV's may correspond to larger deltas ... although at low shutter speeds large MVs may correlate with reduced overall HF energy.
** Predict delta spectral shape from source block spectral shape. HF/LF ratio of the delta may be correlated with the same ratio in its source blocks. Works well with decode-side fDCT.

==Negative results==

* Using Kurtosis for detecting text in a frame
** The idea was to detect a Bernouilli distribution but it's not robust and too noisy

Daala

2012-08-31T20:08:18Z

Rillian: /* Weekly meetings */ No meeting this week.

Daala is the current working name of a next generation video codec— to be renamed once someone insists on something better. So far the best proposed alternative is PatentCake.

For now the purposes of this page is to collect notes about things which have been discussed in informal public IRC discussion about the next generation initiative. Participants in these discussions have included Timothy Terriberry, Jason Garrett-Glaser, Loren Merritt, Ben Schwartz, Greg Maxwell, and others.

See also: [https://xiph.org/daala/ https://xiph.org/daala/]

== Weekly meetings ==

We've been having weekly progress meetings on mumble.

* 2012 June 4 [https://people.xiph.org/~giles/2012/daala_20120604.txt minutes] (actually a work week)
* 2012 June 22 [https://people.xiph.org/~giles/2012/daala_20120622.txt minutes]
* 2012 June 29 [https://people.xiph.org/~giles/2012/daala_20120629.txt minutes] [https://people.xiph.org/~giles/2012/daala_20120629.opus recording]
* 2012 July 6 [https://people.xiph.org/~giles/2012/daala_20120706.txt minutes]
* 2012 July 13 [https://people.xiph.org/~giles/2012/daala_20120713.txt minutes]
* 2012 July 20 [https://people.xiph.org/~giles/2012/daala_20120720.txt minutes]
* 2012 July 27 [https://people.xiph.org/~giles/2012/daala_20120727.txt minutes]
* 2012 August 3 [https://people.xiph.org/~giles/2012/daala_20120803.txt minutes] [https://people.xiph.org/~giles/2012/daala_20120803.opus recording]
* 2012 August 10 [https://people.xiph.org/~giles/2012/daala_20120810.txt minutes]
* 2012 August 17 - no meeting
* 2012 August 24 - [https://people.xiph.org/~giles/2012/daala_20120824.txt minutes] [https://people.xiph.org/~giles/2012/daala_20120824.opus recording]
* 2012 August 31 - no meeting

= Techniques =

The discussed overall structure so far has been a variable size lapped-DCT block based codec with lapping done via pre/post filtering with a specially structured (lifting) linear phase transform along the edges along with overlapped block motion compensation and the expected trimmings. The lapping can be optimized for energy compaction and other useful properties, including invert-ability, and yields excellent results with efficient finite precision math.

Other components which have been discussed include:

==Techniques applicable to all frame types==
* Multisymbol arithmetic coding
** Timothy has some trial code showing speed-up proportional to the number of bits coded at once. (ec_test.c)
* Mode prediction using the previously decoded data, e.g. coding the mode using a probability function derived from trained predictors on the surrounding blocks.
** This will be terrible for robustness but may significantly reduce signalling overhead, allowing many more modes, and provide continuous adaptation between signalling free and fully signalled modes.
* Explore legendre polynomial basis transforms instead of DCT
** May have better perceptual properties and/or result in 'less compromised' efficient implementations.
* Coefficient domain prediction to allow efficient energy preserving quantization.
* Variable partition size/shape and the use of good predictors appears to remove most of the benefit of directional transforms.
** Perhaps 45deg is still useful?
** How does this change with partition sizes? Directional transforms are clearly not that useful with 4x4.
* Transform-post filtering to allow merging smaller transform blocks (like TF merging in CELT) may allow more flexible partitioning then outright using mixed block sizes.
* Perturbed quantization mode-signalling has been discussed but mostly laughed at. ;)
* Special block modes well suited to solid color/cartoon like content— avoiding ringing.
** Are pixel prediction modes too slow?
* In general— what markov random field techniques can be applied with acceptable performance. Any?
* Designed for parallel encode and decode within each frame
** Important because
*** the proposed techniques need a lot more CPU than H.264 and VP8 for both encode and decode
*** Moore's law for single-threaded throughput is dead. Future hardware is all multicore/GPU.
** Implies
*** Getting the order of application right for the lapping filters.
*** Mandatory slicing? Maybe some kind of multilevel entropy coding to reduce redundancy between slices while minimizing the single-threaded portion of decode.

* Using PVQ and energy conservation: see http://jmvalin.ca/video/video_pvq_v3.pdf

==Techniques applicable to inter frames==
* Using x264 as a test-bed Jason and Loren demonstrated 15% rate/distortion improvements from using 10-bit intermediaries and references, estimated as being 1/3rd from quality calculation in the 10-bit space, 1/3rd from the higher precision references, and 1/3rd from higher intermediate precision in calculations (e.g. MC filter processing).
** Increased reference precision competes for memory with increased number of references. The improvements demonstrated appear to be a greater win than increasing the reference count once there are four references or so.
* Super-resolution techniques for motion-compensation references have been discussed— in particular it appears that the half-pel location is where intelligent filtering matters the most so staged computation could be effectively used to allow more expensive filtering at that level.
** Edge-directed interpolation techniques might be effectively applied to increase motion compensation accuracy, but most of the techniques known to be very effective are too slow.
** Speculation has been offered that a significant part of MC inaccuracy may be due to blending in a physically incorrect (gamma-corrected) space, though no real conclusions were made. Academic papers on motion compensation accuracy seem to have ignored this issue.
* Timothy has an example code base for a variable partition size blocking-free motion compensation scheme which merges OBMC (overlapped block motion compensation) and CGI (control-grid interpolation) with an interesting prediction/sub-division scheme and whole-frame trellis optimization of motion vectors. (daala-exp)

==Basic features==
* YUV 4:4:4, 4:2:2 , 4:2:0 subsamplings, 8-bit, 10bit.
* Alpha channel — need testing material!
* 8-bit RGB compatible mode? (e.g. YCoCg, internally or at least flagging for it)
* Efficient 3D? — need testing material!
* Lossless?
** The value of this is disputable. If nothing else it's arguable that stuffing lossless into a lossy format may be the only way to get lossless into many people's hands. Also, see below
* Good support for decode side droppable frames?
** Hopefully the referencing structure will be flexible enough to enable this even if it's not an intentional feature.

==Frills==
* Optionally storing a checksum of the expected decoded frame for decoder/encoder mismatch detection.
* Expose the number of referential descendants of a given frame (or even the whole reference DAG) for most efficient allocation of FEC.

==Wingdings==
Crazy crap that might be interesting or at least fun to make fun of...
* >10bit?
** Use cases don't seem well enough defined yet. Significant complexity. Any prospective hardware developer may hire assassins.
** Possible compromise: the video reference structure contains a backbone that can be decoded at only N bits of depth (e.g. 10), and higher precisions are only supported outside of this reference chain.
**# Precision by truncation: decode is performed twice on each frame, identically, at low and high precision. The only difference between them is the bit-depth of the transform, or possibly of the transform and MC filters. Only low-precision outputs can be referenced by subsequent frames. Useful if high-precision content is still worth watching at low precision.
**# Precision by gamma: decode is performed once at low precision as normal. Then the output frame is converted to linear-light at high precision, after which another layer of residuals is added. The second layer can be permitted to reference previous high-precision frames... tricky to use both sets of references though. Useful if high precision is used for storing linear data, but people still want to watch it on "low-end" hardware.
* Some high end digital cameras are operating jpeg-derivatives in a special mode that keeps the image in the native linear RGB bayer format in order to avoid lossy/slow demosaicing on the camera. In particular this allows white balancing in post without excessive loss. Probably out of scope for Daala itself.
** Bayer, 4:2:0, 4:2:2, and Interlacing are all special cases of a more general pattern in which the output frames are decimated/subsampled in a regular fashion. All such subsamplings could be supported by a unified framework in which the video is always stored with all planes fully sampled, with a header indicating the recommended subsampling for display. In such cases, the encoder can regard the transform as highly overcomplete, and simply ignore unneeded coefficients (presumably by leaving high frequency residuals coded as zero). This structure would in effect turn the codec into a motion-compensated interpolating/deinterlacing filter. Whether this approach is sensible presumably depends in part on how the transform is structured. It would be especially easy if the transform's highest-frequencies were coded by a wavelet-like layer.
* Lossless intra-ability: The ability to losslessly rewrite any frame as an intra frame (perhaps with significant bitrate overhead) in order to make frame accurate cuts possible.
** Or best handled by making sure that containers have working pre-roll, but presumably common GOP sizes will be greater than the number of references so even if losslessly reencoding the references is expensive it may be cheaper than pre-roll. Do both?
** Can be had for 'free' if lossless is supported, plus the right header flags to restuff the references from lossless copies in a packed hidden frame.
** Use of explicitly (rather than staged) super-resolution and/or deeper references may make this functionality unattractive due to increased overhead.
* Internal overlays which could be swapped without re-encoding? (e.g. advertising, station ID). Could also be automatically generated by a Sufficiently Advanced™ encoder to improve efficiencies for static sprites over moving backgrounds.
** Complicates making the complexity bounded. No Sufficiently Advanced™ encoder likely to ever exist. But perhaps the station id/advertising uses fully justify this.
** Could be done externally to the video codec, but if so it's no likely to be useful for anyone ever.
* A secondary reference implementation in OpenCL, maintained throughout development, to make sure that the codec is GPU-friendly and can be done efficiently using OpenCL primitives.
* SWAR-friendly arithmetic. For example, choosing transform coefficients so that no intermediate product overflows 16 bits (tricky for signed values) can sometimes enable (e.g.) 4 parallel operations in one uint64_t. This can allow a pure C reference implementation to run faster, which is valuable for initial adoption and ports to new platforms.
* Parametric decode-side blur.
** Symmetrical blur in regions that are smooth on scales longer than the block size. Could be signaled or derived from observed DC values.
** Motion blur so that moving objects are blurred along the motion vector. May require coding a shutter speed parameter (0..1 as a fraction of the inter-frame interval).
* Fancy block property prediction. (Not clear how these prediction interact with intra pred)
** Predict block properties (quantizer, energy, etc.) from MV. (0,0) probably means small delta. Larger MV's may correspond to larger deltas ... although at low shutter speeds large MVs may correlate with reduced overall HF energy.
** Predict delta spectral shape from source block spectral shape. HF/LF ratio of the delta may be correlated with the same ratio in its source blocks. Works well with decode-side fDCT.

==Negative results==

* Using Kurtosis for detecting text in a frame
** The idea was to detect a Bernouilli distribution but it's not robust and too noisy

Daala

2012-08-24T21:54:37Z

Rillian: /* Weekly meetings */ Recording link for today's meeting.

Daala is the current working name of a next generation video codec— to be renamed once someone insists on something better. So far the best proposed alternative is PatentCake.

For now the purposes of this page is to collect notes about things which have been discussed in informal public IRC discussion about the next generation initiative. Participants in these discussions have included Timothy Terriberry, Jason Garrett-Glaser, Loren Merritt, Ben Schwartz, Greg Maxwell, and others.

See also: [https://xiph.org/daala/ https://xiph.org/daala/]

== Weekly meetings ==

We've been having weekly progress meetings on mumble.

* 2012 June 4 [https://people.xiph.org/~giles/2012/daala_20120604.txt minutes] (actually a work week)
* 2012 June 22 [https://people.xiph.org/~giles/2012/daala_20120622.txt minutes]
* 2012 June 29 [https://people.xiph.org/~giles/2012/daala_20120629.txt minutes] [https://people.xiph.org/~giles/2012/daala_20120629.opus recording]
* 2012 July 6 [https://people.xiph.org/~giles/2012/daala_20120706.txt minutes]
* 2012 July 13 [https://people.xiph.org/~giles/2012/daala_20120713.txt minutes]
* 2012 July 20 [https://people.xiph.org/~giles/2012/daala_20120720.txt minutes]
* 2012 July 27 [https://people.xiph.org/~giles/2012/daala_20120727.txt minutes]
* 2012 August 3 [https://people.xiph.org/~giles/2012/daala_20120803.txt minutes] [https://people.xiph.org/~giles/2012/daala_20120803.opus recording]
* 2012 August 10 [https://people.xiph.org/~giles/2012/daala_20120810.txt minutes]
* 2012 August 17 - no meeting
* 2012 August 24 - [https://people.xiph.org/~giles/2012/daala_20120824.txt minutes] [https://people.xiph.org/~giles/2012/daala_20120824.opus recording]

= Techniques =

The discussed overall structure so far has been a variable size lapped-DCT block based codec with lapping done via pre/post filtering with a specially structured (lifting) linear phase transform along the edges along with overlapped block motion compensation and the expected trimmings. The lapping can be optimized for energy compaction and other useful properties, including invert-ability, and yields excellent results with efficient finite precision math.

Other components which have been discussed include:

==Techniques applicable to all frame types==
* Multisymbol arithmetic coding
** Timothy has some trial code showing speed-up proportional to the number of bits coded at once. (ec_test.c)
* Mode prediction using the previously decoded data, e.g. coding the mode using a probability function derived from trained predictors on the surrounding blocks.
** This will be terrible for robustness but may significantly reduce signalling overhead, allowing many more modes, and provide continuous adaptation between signalling free and fully signalled modes.
* Explore legendre polynomial basis transforms instead of DCT
** May have better perceptual properties and/or result in 'less compromised' efficient implementations.
* Coefficient domain prediction to allow efficient energy preserving quantization.
* Variable partition size/shape and the use of good predictors appears to remove most of the benefit of directional transforms.
** Perhaps 45deg is still useful?
** How does this change with partition sizes? Directional transforms are clearly not that useful with 4x4.
* Transform-post filtering to allow merging smaller transform blocks (like TF merging in CELT) may allow more flexible partitioning then outright using mixed block sizes.
* Perturbed quantization mode-signalling has been discussed but mostly laughed at. ;)
* Special block modes well suited to solid color/cartoon like content— avoiding ringing.
** Are pixel prediction modes too slow?
* In general— what markov random field techniques can be applied with acceptable performance. Any?
* Designed for parallel encode and decode within each frame
** Important because
*** the proposed techniques need a lot more CPU than H.264 and VP8 for both encode and decode
*** Moore's law for single-threaded throughput is dead. Future hardware is all multicore/GPU.
** Implies
*** Getting the order of application right for the lapping filters.
*** Mandatory slicing? Maybe some kind of multilevel entropy coding to reduce redundancy between slices while minimizing the single-threaded portion of decode.

* Using PVQ and energy conservation: see http://jmvalin.ca/video/video_pvq_v3.pdf

==Techniques applicable to inter frames==
* Using x264 as a test-bed Jason and Loren demonstrated 15% rate/distortion improvements from using 10-bit intermediaries and references, estimated as being 1/3rd from quality calculation in the 10-bit space, 1/3rd from the higher precision references, and 1/3rd from higher intermediate precision in calculations (e.g. MC filter processing).
** Increased reference precision competes for memory with increased number of references. The improvements demonstrated appear to be a greater win than increasing the reference count once there are four references or so.
* Super-resolution techniques for motion-compensation references have been discussed— in particular it appears that the half-pel location is where intelligent filtering matters the most so staged computation could be effectively used to allow more expensive filtering at that level.
** Edge-directed interpolation techniques might be effectively applied to increase motion compensation accuracy, but most of the techniques known to be very effective are too slow.
** Speculation has been offered that a significant part of MC inaccuracy may be due to blending in a physically incorrect (gamma-corrected) space, though no real conclusions were made. Academic papers on motion compensation accuracy seem to have ignored this issue.
* Timothy has an example code base for a variable partition size blocking-free motion compensation scheme which merges OBMC (overlapped block motion compensation) and CGI (control-grid interpolation) with an interesting prediction/sub-division scheme and whole-frame trellis optimization of motion vectors. (daala-exp)

==Basic features==
* YUV 4:4:4, 4:2:2 , 4:2:0 subsamplings, 8-bit, 10bit.
* Alpha channel — need testing material!
* 8-bit RGB compatible mode? (e.g. YCoCg, internally or at least flagging for it)
* Efficient 3D? — need testing material!
* Lossless?
** The value of this is disputable. If nothing else it's arguable that stuffing lossless into a lossy format may be the only way to get lossless into many people's hands. Also, see below
* Good support for decode side droppable frames?
** Hopefully the referencing structure will be flexible enough to enable this even if it's not an intentional feature.

==Frills==
* Optionally storing a checksum of the expected decoded frame for decoder/encoder mismatch detection.
* Expose the number of referential descendants of a given frame (or even the whole reference DAG) for most efficient allocation of FEC.

==Wingdings==
Crazy crap that might be interesting or at least fun to make fun of...
* >10bit?
** Use cases don't seem well enough defined yet. Significant complexity. Any prospective hardware developer may hire assassins.
** Possible compromise: the video reference structure contains a backbone that can be decoded at only N bits of depth (e.g. 10), and higher precisions are only supported outside of this reference chain.
**# Precision by truncation: decode is performed twice on each frame, identically, at low and high precision. The only difference between them is the bit-depth of the transform, or possibly of the transform and MC filters. Only low-precision outputs can be referenced by subsequent frames. Useful if high-precision content is still worth watching at low precision.
**# Precision by gamma: decode is performed once at low precision as normal. Then the output frame is converted to linear-light at high precision, after which another layer of residuals is added. The second layer can be permitted to reference previous high-precision frames... tricky to use both sets of references though. Useful if high precision is used for storing linear data, but people still want to watch it on "low-end" hardware.
* Some high end digital cameras are operating jpeg-derivatives in a special mode that keeps the image in the native linear RGB bayer format in order to avoid lossy/slow demosaicing on the camera. In particular this allows white balancing in post without excessive loss. Probably out of scope for Daala itself.
** Bayer, 4:2:0, 4:2:2, and Interlacing are all special cases of a more general pattern in which the output frames are decimated/subsampled in a regular fashion. All such subsamplings could be supported by a unified framework in which the video is always stored with all planes fully sampled, with a header indicating the recommended subsampling for display. In such cases, the encoder can regard the transform as highly overcomplete, and simply ignore unneeded coefficients (presumably by leaving high frequency residuals coded as zero). This structure would in effect turn the codec into a motion-compensated interpolating/deinterlacing filter. Whether this approach is sensible presumably depends in part on how the transform is structured. It would be especially easy if the transform's highest-frequencies were coded by a wavelet-like layer.
* Lossless intra-ability: The ability to losslessly rewrite any frame as an intra frame (perhaps with significant bitrate overhead) in order to make frame accurate cuts possible.
** Or best handled by making sure that containers have working pre-roll, but presumably common GOP sizes will be greater than the number of references so even if losslessly reencoding the references is expensive it may be cheaper than pre-roll. Do both?
** Can be had for 'free' if lossless is supported, plus the right header flags to restuff the references from lossless copies in a packed hidden frame.
** Use of explicitly (rather than staged) super-resolution and/or deeper references may make this functionality unattractive due to increased overhead.
* Internal overlays which could be swapped without re-encoding? (e.g. advertising, station ID). Could also be automatically generated by a Sufficiently Advanced™ encoder to improve efficiencies for static sprites over moving backgrounds.
** Complicates making the complexity bounded. No Sufficiently Advanced™ encoder likely to ever exist. But perhaps the station id/advertising uses fully justify this.
** Could be done externally to the video codec, but if so it's no likely to be useful for anyone ever.
* A secondary reference implementation in OpenCL, maintained throughout development, to make sure that the codec is GPU-friendly and can be done efficiently using OpenCL primitives.
* SWAR-friendly arithmetic. For example, choosing transform coefficients so that no intermediate product overflows 16 bits (tricky for signed values) can sometimes enable (e.g.) 4 parallel operations in one uint64_t. This can allow a pure C reference implementation to run faster, which is valuable for initial adoption and ports to new platforms.
* Parametric decode-side blur.
** Symmetrical blur in regions that are smooth on scales longer than the block size. Could be signaled or derived from observed DC values.
** Motion blur so that moving objects are blurred along the motion vector. May require coding a shutter speed parameter (0..1 as a fraction of the inter-frame interval).
* Fancy block property prediction. (Not clear how these prediction interact with intra pred)
** Predict block properties (quantizer, energy, etc.) from MV. (0,0) probably means small delta. Larger MV's may correspond to larger deltas ... although at low shutter speeds large MVs may correlate with reduced overall HF energy.
** Predict delta spectral shape from source block spectral shape. HF/LF ratio of the delta may be correlated with the same ratio in its source blocks. Works well with decode-side fDCT.

==Negative results==

* Using Kurtosis for detecting text in a frame
** The idea was to detect a Bernouilli distribution but it's not robust and too noisy

Daala

2012-08-24T20:35:17Z

Rillian: /* Weekly meetings */ upload today's meeting minutes

Daala is the current working name of a next generation video codec— to be renamed once someone insists on something better. So far the best proposed alternative is PatentCake.

For now the purposes of this page is to collect notes about things which have been discussed in informal public IRC discussion about the next generation initiative. Participants in these discussions have included Timothy Terriberry, Jason Garrett-Glaser, Loren Merritt, Ben Schwartz, Greg Maxwell, and others.

See also: [https://xiph.org/daala/ https://xiph.org/daala/]

== Weekly meetings ==

We've been having weekly progress meetings on mumble.

* 2012 June 4 [https://people.xiph.org/~giles/2012/daala_20120604.txt minutes] (actually a work week)
* 2012 June 22 [https://people.xiph.org/~giles/2012/daala_20120622.txt minutes]
* 2012 June 29 [https://people.xiph.org/~giles/2012/daala_20120629.txt minutes] [https://people.xiph.org/~giles/2012/daala_20120629.opus recording]
* 2012 July 6 [https://people.xiph.org/~giles/2012/daala_20120706.txt minutes]
* 2012 July 13 [https://people.xiph.org/~giles/2012/daala_20120713.txt minutes]
* 2012 July 20 [https://people.xiph.org/~giles/2012/daala_20120720.txt minutes]
* 2012 July 27 [https://people.xiph.org/~giles/2012/daala_20120727.txt minutes]
* 2012 August 3 [https://people.xiph.org/~giles/2012/daala_20120803.txt minutes] [https://people.xiph.org/~giles/2012/daala_20120803.opus recording]
* 2012 August 10 [https://people.xiph.org/~giles/2012/daala_20120810.txt minutes]
* 2012 August 17 - no meeting
* 2012 August 24 - [https://people.xiph.org/~giles/2012/daala_20120824.txt minutes]

= Techniques =

The discussed overall structure so far has been a variable size lapped-DCT block based codec with lapping done via pre/post filtering with a specially structured (lifting) linear phase transform along the edges along with overlapped block motion compensation and the expected trimmings. The lapping can be optimized for energy compaction and other useful properties, including invert-ability, and yields excellent results with efficient finite precision math.

Other components which have been discussed include:

==Techniques applicable to all frame types==
* Multisymbol arithmetic coding
** Timothy has some trial code showing speed-up proportional to the number of bits coded at once. (ec_test.c)
* Mode prediction using the previously decoded data, e.g. coding the mode using a probability function derived from trained predictors on the surrounding blocks.
** This will be terrible for robustness but may significantly reduce signalling overhead, allowing many more modes, and provide continuous adaptation between signalling free and fully signalled modes.
* Explore legendre polynomial basis transforms instead of DCT
** May have better perceptual properties and/or result in 'less compromised' efficient implementations.
* Coefficient domain prediction to allow efficient energy preserving quantization.
* Variable partition size/shape and the use of good predictors appears to remove most of the benefit of directional transforms.
** Perhaps 45deg is still useful?
** How does this change with partition sizes? Directional transforms are clearly not that useful with 4x4.
* Transform-post filtering to allow merging smaller transform blocks (like TF merging in CELT) may allow more flexible partitioning then outright using mixed block sizes.
* Perturbed quantization mode-signalling has been discussed but mostly laughed at. ;)
* Special block modes well suited to solid color/cartoon like content— avoiding ringing.
** Are pixel prediction modes too slow?
* In general— what markov random field techniques can be applied with acceptable performance. Any?
* Designed for parallel encode and decode within each frame
** Important because
*** the proposed techniques need a lot more CPU than H.264 and VP8 for both encode and decode
*** Moore's law for single-threaded throughput is dead. Future hardware is all multicore/GPU.
** Implies
*** Getting the order of application right for the lapping filters.
*** Mandatory slicing? Maybe some kind of multilevel entropy coding to reduce redundancy between slices while minimizing the single-threaded portion of decode.

* Using PVQ and energy conservation: see http://jmvalin.ca/video/video_pvq_v3.pdf

==Techniques applicable to inter frames==
* Using x264 as a test-bed Jason and Loren demonstrated 15% rate/distortion improvements from using 10-bit intermediaries and references, estimated as being 1/3rd from quality calculation in the 10-bit space, 1/3rd from the higher precision references, and 1/3rd from higher intermediate precision in calculations (e.g. MC filter processing).
** Increased reference precision competes for memory with increased number of references. The improvements demonstrated appear to be a greater win than increasing the reference count once there are four references or so.
* Super-resolution techniques for motion-compensation references have been discussed— in particular it appears that the half-pel location is where intelligent filtering matters the most so staged computation could be effectively used to allow more expensive filtering at that level.
** Edge-directed interpolation techniques might be effectively applied to increase motion compensation accuracy, but most of the techniques known to be very effective are too slow.
** Speculation has been offered that a significant part of MC inaccuracy may be due to blending in a physically incorrect (gamma-corrected) space, though no real conclusions were made. Academic papers on motion compensation accuracy seem to have ignored this issue.
* Timothy has an example code base for a variable partition size blocking-free motion compensation scheme which merges OBMC (overlapped block motion compensation) and CGI (control-grid interpolation) with an interesting prediction/sub-division scheme and whole-frame trellis optimization of motion vectors. (daala-exp)

==Basic features==
* YUV 4:4:4, 4:2:2 , 4:2:0 subsamplings, 8-bit, 10bit.
* Alpha channel — need testing material!
* 8-bit RGB compatible mode? (e.g. YCoCg, internally or at least flagging for it)
* Efficient 3D? — need testing material!
* Lossless?
** The value of this is disputable. If nothing else it's arguable that stuffing lossless into a lossy format may be the only way to get lossless into many people's hands. Also, see below
* Good support for decode side droppable frames?
** Hopefully the referencing structure will be flexible enough to enable this even if it's not an intentional feature.

==Frills==
* Optionally storing a checksum of the expected decoded frame for decoder/encoder mismatch detection.
* Expose the number of referential descendants of a given frame (or even the whole reference DAG) for most efficient allocation of FEC.

==Wingdings==
Crazy crap that might be interesting or at least fun to make fun of...
* >10bit?
** Use cases don't seem well enough defined yet. Significant complexity. Any prospective hardware developer may hire assassins.
** Possible compromise: the video reference structure contains a backbone that can be decoded at only N bits of depth (e.g. 10), and higher precisions are only supported outside of this reference chain.
**# Precision by truncation: decode is performed twice on each frame, identically, at low and high precision. The only difference between them is the bit-depth of the transform, or possibly of the transform and MC filters. Only low-precision outputs can be referenced by subsequent frames. Useful if high-precision content is still worth watching at low precision.
**# Precision by gamma: decode is performed once at low precision as normal. Then the output frame is converted to linear-light at high precision, after which another layer of residuals is added. The second layer can be permitted to reference previous high-precision frames... tricky to use both sets of references though. Useful if high precision is used for storing linear data, but people still want to watch it on "low-end" hardware.
* Some high end digital cameras are operating jpeg-derivatives in a special mode that keeps the image in the native linear RGB bayer format in order to avoid lossy/slow demosaicing on the camera. In particular this allows white balancing in post without excessive loss. Probably out of scope for Daala itself.
** Bayer, 4:2:0, 4:2:2, and Interlacing are all special cases of a more general pattern in which the output frames are decimated/subsampled in a regular fashion. All such subsamplings could be supported by a unified framework in which the video is always stored with all planes fully sampled, with a header indicating the recommended subsampling for display. In such cases, the encoder can regard the transform as highly overcomplete, and simply ignore unneeded coefficients (presumably by leaving high frequency residuals coded as zero). This structure would in effect turn the codec into a motion-compensated interpolating/deinterlacing filter. Whether this approach is sensible presumably depends in part on how the transform is structured. It would be especially easy if the transform's highest-frequencies were coded by a wavelet-like layer.
* Lossless intra-ability: The ability to losslessly rewrite any frame as an intra frame (perhaps with significant bitrate overhead) in order to make frame accurate cuts possible.
** Or best handled by making sure that containers have working pre-roll, but presumably common GOP sizes will be greater than the number of references so even if losslessly reencoding the references is expensive it may be cheaper than pre-roll. Do both?
** Can be had for 'free' if lossless is supported, plus the right header flags to restuff the references from lossless copies in a packed hidden frame.
** Use of explicitly (rather than staged) super-resolution and/or deeper references may make this functionality unattractive due to increased overhead.
* Internal overlays which could be swapped without re-encoding? (e.g. advertising, station ID). Could also be automatically generated by a Sufficiently Advanced™ encoder to improve efficiencies for static sprites over moving backgrounds.
** Complicates making the complexity bounded. No Sufficiently Advanced™ encoder likely to ever exist. But perhaps the station id/advertising uses fully justify this.
** Could be done externally to the video codec, but if so it's no likely to be useful for anyone ever.
* A secondary reference implementation in OpenCL, maintained throughout development, to make sure that the codec is GPU-friendly and can be done efficiently using OpenCL primitives.
* SWAR-friendly arithmetic. For example, choosing transform coefficients so that no intermediate product overflows 16 bits (tricky for signed values) can sometimes enable (e.g.) 4 parallel operations in one uint64_t. This can allow a pure C reference implementation to run faster, which is valuable for initial adoption and ports to new platforms.
* Parametric decode-side blur.
** Symmetrical blur in regions that are smooth on scales longer than the block size. Could be signaled or derived from observed DC values.
** Motion blur so that moving objects are blurred along the motion vector. May require coding a shutter speed parameter (0..1 as a fraction of the inter-frame interval).
* Fancy block property prediction. (Not clear how these prediction interact with intra pred)
** Predict block properties (quantizer, energy, etc.) from MV. (0,0) probably means small delta. Larger MV's may correspond to larger deltas ... although at low shutter speeds large MVs may correlate with reduced overall HF energy.
** Predict delta spectral shape from source block spectral shape. HF/LF ratio of the delta may be correlated with the same ratio in its source blocks. Works well with decode-side fDCT.

==Negative results==

* Using Kurtosis for detecting text in a frame
** The idea was to detect a Bernouilli distribution but it's not robust and too noisy

Daala

2012-08-20T21:34:45Z

Rillian: /* Weekly meetings */ last week's meeting was skipped

Daala is the current working name of a next generation video codec— to be renamed once someone insists on something better. So far the best proposed alternative is PatentCake.

For now the purposes of this page is to collect notes about things which have been discussed in informal public IRC discussion about the next generation initiative. Participants in these discussions have included Timothy Terriberry, Jason Garrett-Glaser, Loren Merritt, Ben Schwartz, Greg Maxwell, and others.

See also: [https://xiph.org/daala/ https://xiph.org/daala/]

== Weekly meetings ==

We've been having weekly progress meetings on mumble.

* 2012 June 4 [https://people.xiph.org/~giles/2012/daala_20120604.txt minutes] (actually a work week)
* 2012 June 22 [https://people.xiph.org/~giles/2012/daala_20120622.txt minutes]
* 2012 June 29 [https://people.xiph.org/~giles/2012/daala_20120629.txt minutes] [https://people.xiph.org/~giles/2012/daala_20120629.opus recording]
* 2012 July 6 [https://people.xiph.org/~giles/2012/daala_20120706.txt minutes]
* 2012 July 13 [https://people.xiph.org/~giles/2012/daala_20120713.txt minutes]
* 2012 July 20 [https://people.xiph.org/~giles/2012/daala_20120720.txt minutes]
* 2012 July 27 [https://people.xiph.org/~giles/2012/daala_20120727.txt minutes]
* 2012 August 3 [https://people.xiph.org/~giles/2012/daala_20120803.txt minutes] [https://people.xiph.org/~giles/2012/daala_20120803.opus recording]
* 2012 August 10 [https://people.xiph.org/~giles/2012/daala_20120810.txt minutes]
* 2012 August 17 - no meeting

= Techniques =

The discussed overall structure so far has been a variable size lapped-DCT block based codec with lapping done via pre/post filtering with a specially structured (lifting) linear phase transform along the edges along with overlapped block motion compensation and the expected trimmings. The lapping can be optimized for energy compaction and other useful properties, including invert-ability, and yields excellent results with efficient finite precision math.

Other components which have been discussed include:

==Techniques applicable to all frame types==
* Multisymbol arithmetic coding
** Timothy has some trial code showing speed-up proportional to the number of bits coded at once. (ec_test.c)
* Mode prediction using the previously decoded data, e.g. coding the mode using a probability function derived from trained predictors on the surrounding blocks.
** This will be terrible for robustness but may significantly reduce signalling overhead, allowing many more modes, and provide continuous adaptation between signalling free and fully signalled modes.
* Explore legendre polynomial basis transforms instead of DCT
** May have better perceptual properties and/or result in 'less compromised' efficient implementations.
* Coefficient domain prediction to allow efficient energy preserving quantization.
* Variable partition size/shape and the use of good predictors appears to remove most of the benefit of directional transforms.
** Perhaps 45deg is still useful?
** How does this change with partition sizes? Directional transforms are clearly not that useful with 4x4.
* Transform-post filtering to allow merging smaller transform blocks (like TF merging in CELT) may allow more flexible partitioning then outright using mixed block sizes.
* Perturbed quantization mode-signalling has been discussed but mostly laughed at. ;)
* Special block modes well suited to solid color/cartoon like content— avoiding ringing.
** Are pixel prediction modes too slow?
* In general— what markov random field techniques can be applied with acceptable performance. Any?
* Designed for parallel encode and decode within each frame
** Important because
*** the proposed techniques need a lot more CPU than H.264 and VP8 for both encode and decode
*** Moore's law for single-threaded throughput is dead. Future hardware is all multicore/GPU.
** Implies
*** Getting the order of application right for the lapping filters.
*** Mandatory slicing? Maybe some kind of multilevel entropy coding to reduce redundancy between slices while minimizing the single-threaded portion of decode.

* Using PVQ and energy conservation: see http://jmvalin.ca/video/video_pvq_v3.pdf

==Techniques applicable to inter frames==
* Using x264 as a test-bed Jason and Loren demonstrated 15% rate/distortion improvements from using 10-bit intermediaries and references, estimated as being 1/3rd from quality calculation in the 10-bit space, 1/3rd from the higher precision references, and 1/3rd from higher intermediate precision in calculations (e.g. MC filter processing).
** Increased reference precision competes for memory with increased number of references. The improvements demonstrated appear to be a greater win than increasing the reference count once there are four references or so.
* Super-resolution techniques for motion-compensation references have been discussed— in particular it appears that the half-pel location is where intelligent filtering matters the most so staged computation could be effectively used to allow more expensive filtering at that level.
** Edge-directed interpolation techniques might be effectively applied to increase motion compensation accuracy, but most of the techniques known to be very effective are too slow.
** Speculation has been offered that a significant part of MC inaccuracy may be due to blending in a physically incorrect (gamma-corrected) space, though no real conclusions were made. Academic papers on motion compensation accuracy seem to have ignored this issue.
* Timothy has an example code base for a variable partition size blocking-free motion compensation scheme which merges OBMC (overlapped block motion compensation) and CGI (control-grid interpolation) with an interesting prediction/sub-division scheme and whole-frame trellis optimization of motion vectors. (daala-exp)

==Basic features==
* YUV 4:4:4, 4:2:2 , 4:2:0 subsamplings, 8-bit, 10bit.
* Alpha channel — need testing material!
* 8-bit RGB compatible mode? (e.g. YCoCg, internally or at least flagging for it)
* Efficient 3D? — need testing material!
* Lossless?
** The value of this is disputable. If nothing else it's arguable that stuffing lossless into a lossy format may be the only way to get lossless into many people's hands. Also, see below
* Good support for decode side droppable frames?
** Hopefully the referencing structure will be flexible enough to enable this even if it's not an intentional feature.

==Frills==
* Optionally storing a checksum of the expected decoded frame for decoder/encoder mismatch detection.
* Expose the number of referential descendants of a given frame (or even the whole reference DAG) for most efficient allocation of FEC.

==Wingdings==
Crazy crap that might be interesting or at least fun to make fun of...
* >10bit?
** Use cases don't seem well enough defined yet. Significant complexity. Any prospective hardware developer may hire assassins.
** Possible compromise: the video reference structure contains a backbone that can be decoded at only N bits of depth (e.g. 10), and higher precisions are only supported outside of this reference chain.
**# Precision by truncation: decode is performed twice on each frame, identically, at low and high precision. The only difference between them is the bit-depth of the transform, or possibly of the transform and MC filters. Only low-precision outputs can be referenced by subsequent frames. Useful if high-precision content is still worth watching at low precision.
**# Precision by gamma: decode is performed once at low precision as normal. Then the output frame is converted to linear-light at high precision, after which another layer of residuals is added. The second layer can be permitted to reference previous high-precision frames... tricky to use both sets of references though. Useful if high precision is used for storing linear data, but people still want to watch it on "low-end" hardware.
* Some high end digital cameras are operating jpeg-derivatives in a special mode that keeps the image in the native linear RGB bayer format in order to avoid lossy/slow demosaicing on the camera. In particular this allows white balancing in post without excessive loss. Probably out of scope for Daala itself.
** Bayer, 4:2:0, 4:2:2, and Interlacing are all special cases of a more general pattern in which the output frames are decimated/subsampled in a regular fashion. All such subsamplings could be supported by a unified framework in which the video is always stored with all planes fully sampled, with a header indicating the recommended subsampling for display. In such cases, the encoder can regard the transform as highly overcomplete, and simply ignore unneeded coefficients (presumably by leaving high frequency residuals coded as zero). This structure would in effect turn the codec into a motion-compensated interpolating/deinterlacing filter. Whether this approach is sensible presumably depends in part on how the transform is structured. It would be especially easy if the transform's highest-frequencies were coded by a wavelet-like layer.
* Lossless intra-ability: The ability to losslessly rewrite any frame as an intra frame (perhaps with significant bitrate overhead) in order to make frame accurate cuts possible.
** Or best handled by making sure that containers have working pre-roll, but presumably common GOP sizes will be greater than the number of references so even if losslessly reencoding the references is expensive it may be cheaper than pre-roll. Do both?
** Can be had for 'free' if lossless is supported, plus the right header flags to restuff the references from lossless copies in a packed hidden frame.
** Use of explicitly (rather than staged) super-resolution and/or deeper references may make this functionality unattractive due to increased overhead.
* Internal overlays which could be swapped without re-encoding? (e.g. advertising, station ID). Could also be automatically generated by a Sufficiently Advanced™ encoder to improve efficiencies for static sprites over moving backgrounds.
** Complicates making the complexity bounded. No Sufficiently Advanced™ encoder likely to ever exist. But perhaps the station id/advertising uses fully justify this.
** Could be done externally to the video codec, but if so it's no likely to be useful for anyone ever.
* A secondary reference implementation in OpenCL, maintained throughout development, to make sure that the codec is GPU-friendly and can be done efficiently using OpenCL primitives.
* SWAR-friendly arithmetic. For example, choosing transform coefficients so that no intermediate product overflows 16 bits (tricky for signed values) can sometimes enable (e.g.) 4 parallel operations in one uint64_t. This can allow a pure C reference implementation to run faster, which is valuable for initial adoption and ports to new platforms.
* Parametric decode-side blur.
** Symmetrical blur in regions that are smooth on scales longer than the block size. Could be signaled or derived from observed DC values.
** Motion blur so that moving objects are blurred along the motion vector. May require coding a shutter speed parameter (0..1 as a fraction of the inter-frame interval).
* Fancy block property prediction. (Not clear how these prediction interact with intra pred)
** Predict block properties (quantizer, energy, etc.) from MV. (0,0) probably means small delta. Larger MV's may correspond to larger deltas ... although at low shutter speeds large MVs may correlate with reduced overall HF energy.
** Predict delta spectral shape from source block spectral shape. HF/LF ratio of the delta may be correlated with the same ratio in its source blocks. Works well with decode-side fDCT.

==Negative results==

* Using Kurtosis for detecting text in a frame
** The idea was to detect a Bernouilli distribution but it's not robust and too noisy

Daala

2012-08-13T17:09:44Z

Rillian: /* Weekly meetings */ upload friday's meeting minutes

Daala is the current working name of a next generation video codec— to be renamed once someone insists on something better. So far the best proposed alternative is PatentCake.

For now the purposes of this page is to collect notes about things which have been discussed in informal public IRC discussion about the next generation initiative. Participants in these discussions have included Timothy Terriberry, Jason Garrett-Glaser, Loren Merritt, Ben Schwartz, Greg Maxwell, and others.

See also: [https://xiph.org/daala/ https://xiph.org/daala/]

== Weekly meetings ==

We've been having weekly progress meetings on mumble.

* 2012 June 4 [https://people.xiph.org/~giles/2012/daala_20120604.txt minutes] (actually a work week)
* 2012 June 22 [https://people.xiph.org/~giles/2012/daala_20120622.txt minutes]
* 2012 June 29 [https://people.xiph.org/~giles/2012/daala_20120629.txt minutes] [https://people.xiph.org/~giles/2012/daala_20120629.opus recording]
* 2012 July 6 [https://people.xiph.org/~giles/2012/daala_20120706.txt minutes]
* 2012 July 13 [https://people.xiph.org/~giles/2012/daala_20120713.txt minutes]
* 2012 July 20 [https://people.xiph.org/~giles/2012/daala_20120720.txt minutes]
* 2012 July 27 [https://people.xiph.org/~giles/2012/daala_20120727.txt minutes]
* 2012 August 3 [https://people.xiph.org/~giles/2012/daala_20120803.txt minutes] [https://people.xiph.org/~giles/2012/daala_20120803.opus recording]
* 2012 August 10 [https://people.xiph.org/~giles/2012/daala_20120810.txt minutes]

= Techniques =

The discussed overall structure so far has been a variable size lapped-DCT block based codec with lapping done via pre/post filtering with a specially structured (lifting) linear phase transform along the edges along with overlapped block motion compensation and the expected trimmings. The lapping can be optimized for energy compaction and other useful properties, including invert-ability, and yields excellent results with efficient finite precision math.

Other components which have been discussed include:

==Techniques applicable to all frame types==
* Multisymbol arithmetic coding
** Timothy has some trial code showing speed-up proportional to the number of bits coded at once. (ec_test.c)
* Mode prediction using the previously decoded data, e.g. coding the mode using a probability function derived from trained predictors on the surrounding blocks.
** This will be terrible for robustness but may significantly reduce signalling overhead, allowing many more modes, and provide continuous adaptation between signalling free and fully signalled modes.
* Explore legendre polynomial basis transforms instead of DCT
** May have better perceptual properties and/or result in 'less compromised' efficient implementations.
* Coefficient domain prediction to allow efficient energy preserving quantization.
* Variable partition size/shape and the use of good predictors appears to remove most of the benefit of directional transforms.
** Perhaps 45deg is still useful?
** How does this change with partition sizes? Directional transforms are clearly not that useful with 4x4.
* Transform-post filtering to allow merging smaller transform blocks (like TF merging in CELT) may allow more flexible partitioning then outright using mixed block sizes.
* Perturbed quantization mode-signalling has been discussed but mostly laughed at. ;)
* Special block modes well suited to solid color/cartoon like content— avoiding ringing.
** Are pixel prediction modes too slow?
* In general— what markov random field techniques can be applied with acceptable performance. Any?
* Designed for parallel encode and decode within each frame
** Important because
*** the proposed techniques need a lot more CPU than H.264 and VP8 for both encode and decode
*** Moore's law for single-threaded throughput is dead. Future hardware is all multicore/GPU.
** Implies
*** Getting the order of application right for the lapping filters.
*** Mandatory slicing? Maybe some kind of multilevel entropy coding to reduce redundancy between slices while minimizing the single-threaded portion of decode.

* Using PVQ and energy conservation: see http://jmvalin.ca/video/video_pvq_v3.pdf

==Techniques applicable to inter frames==
* Using x264 as a test-bed Jason and Loren demonstrated 15% rate/distortion improvements from using 10-bit intermediaries and references, estimated as being 1/3rd from quality calculation in the 10-bit space, 1/3rd from the higher precision references, and 1/3rd from higher intermediate precision in calculations (e.g. MC filter processing).
** Increased reference precision competes for memory with increased number of references. The improvements demonstrated appear to be a greater win than increasing the reference count once there are four references or so.
* Super-resolution techniques for motion-compensation references have been discussed— in particular it appears that the half-pel location is where intelligent filtering matters the most so staged computation could be effectively used to allow more expensive filtering at that level.
** Edge-directed interpolation techniques might be effectively applied to increase motion compensation accuracy, but most of the techniques known to be very effective are too slow.
** Speculation has been offered that a significant part of MC inaccuracy may be due to blending in a physically incorrect (gamma-corrected) space, though no real conclusions were made. Academic papers on motion compensation accuracy seem to have ignored this issue.
* Timothy has an example code base for a variable partition size blocking-free motion compensation scheme which merges OBMC (overlapped block motion compensation) and CGI (control-grid interpolation) with an interesting prediction/sub-division scheme and whole-frame trellis optimization of motion vectors. (daala-exp)

==Basic features==
* YUV 4:4:4, 4:2:2 , 4:2:0 subsamplings, 8-bit, 10bit.
* Alpha channel — need testing material!
* 8-bit RGB compatible mode? (e.g. YCoCg, internally or at least flagging for it)
* Efficient 3D? — need testing material!
* Lossless?
** The value of this is disputable. If nothing else it's arguable that stuffing lossless into a lossy format may be the only way to get lossless into many people's hands. Also, see below
* Good support for decode side droppable frames?
** Hopefully the referencing structure will be flexible enough to enable this even if it's not an intentional feature.

==Frills==
* Optionally storing a checksum of the expected decoded frame for decoder/encoder mismatch detection.
* Expose the number of referential descendants of a given frame (or even the whole reference DAG) for most efficient allocation of FEC.

==Wingdings==
Crazy crap that might be interesting or at least fun to make fun of...
* >10bit?
** Use cases don't seem well enough defined yet. Significant complexity. Any prospective hardware developer may hire assassins.
** Possible compromise: the video reference structure contains a backbone that can be decoded at only N bits of depth (e.g. 10), and higher precisions are only supported outside of this reference chain.
**# Precision by truncation: decode is performed twice on each frame, identically, at low and high precision. The only difference between them is the bit-depth of the transform, or possibly of the transform and MC filters. Only low-precision outputs can be referenced by subsequent frames. Useful if high-precision content is still worth watching at low precision.
**# Precision by gamma: decode is performed once at low precision as normal. Then the output frame is converted to linear-light at high precision, after which another layer of residuals is added. The second layer can be permitted to reference previous high-precision frames... tricky to use both sets of references though. Useful if high precision is used for storing linear data, but people still want to watch it on "low-end" hardware.
* Some high end digital cameras are operating jpeg-derivatives in a special mode that keeps the image in the native linear RGB bayer format in order to avoid lossy/slow demosaicing on the camera. In particular this allows white balancing in post without excessive loss. Probably out of scope for Daala itself.
** Bayer, 4:2:0, 4:2:2, and Interlacing are all special cases of a more general pattern in which the output frames are decimated/subsampled in a regular fashion. All such subsamplings could be supported by a unified framework in which the video is always stored with all planes fully sampled, with a header indicating the recommended subsampling for display. In such cases, the encoder can regard the transform as highly overcomplete, and simply ignore unneeded coefficients (presumably by leaving high frequency residuals coded as zero). This structure would in effect turn the codec into a motion-compensated interpolating/deinterlacing filter. Whether this approach is sensible presumably depends in part on how the transform is structured. It would be especially easy if the transform's highest-frequencies were coded by a wavelet-like layer.
* Lossless intra-ability: The ability to losslessly rewrite any frame as an intra frame (perhaps with significant bitrate overhead) in order to make frame accurate cuts possible.
** Or best handled by making sure that containers have working pre-roll, but presumably common GOP sizes will be greater than the number of references so even if losslessly reencoding the references is expensive it may be cheaper than pre-roll. Do both?
** Can be had for 'free' if lossless is supported, plus the right header flags to restuff the references from lossless copies in a packed hidden frame.
** Use of explicitly (rather than staged) super-resolution and/or deeper references may make this functionality unattractive due to increased overhead.
* Internal overlays which could be swapped without re-encoding? (e.g. advertising, station ID). Could also be automatically generated by a Sufficiently Advanced™ encoder to improve efficiencies for static sprites over moving backgrounds.
** Complicates making the complexity bounded. No Sufficiently Advanced™ encoder likely to ever exist. But perhaps the station id/advertising uses fully justify this.
** Could be done externally to the video codec, but if so it's no likely to be useful for anyone ever.
* A secondary reference implementation in OpenCL, maintained throughout development, to make sure that the codec is GPU-friendly and can be done efficiently using OpenCL primitives.
* SWAR-friendly arithmetic. For example, choosing transform coefficients so that no intermediate product overflows 16 bits (tricky for signed values) can sometimes enable (e.g.) 4 parallel operations in one uint64_t. This can allow a pure C reference implementation to run faster, which is valuable for initial adoption and ports to new platforms.
* Parametric decode-side blur.
** Symmetrical blur in regions that are smooth on scales longer than the block size. Could be signaled or derived from observed DC values.
** Motion blur so that moving objects are blurred along the motion vector. May require coding a shutter speed parameter (0..1 as a fraction of the inter-frame interval).
* Fancy block property prediction. (Not clear how these prediction interact with intra pred)
** Predict block properties (quantizer, energy, etc.) from MV. (0,0) probably means small delta. Larger MV's may correspond to larger deltas ... although at low shutter speeds large MVs may correlate with reduced overall HF energy.
** Predict delta spectral shape from source block spectral shape. HF/LF ratio of the delta may be correlated with the same ratio in its source blocks. Works well with decode-side fDCT.

==Negative results==

* Using Kurtosis for detecting text in a frame
** The idea was to detect a Bernouilli distribution but it's not robust and too noisy

Daala

2012-08-03T22:46:19Z

Rillian: /* Weekly meetings */ Link to this week's recording

Daala is the current working name of a next generation video codec— to be renamed once someone insists on something better. So far the best proposed alternative is PatentCake.

For now the purposes of this page is to collect notes about things which have been discussed in informal public IRC discussion about the next generation initiative. Participants in these discussions have included Timothy Terriberry, Jason Garrett-Glaser, Loren Merritt, Ben Schwartz, Greg Maxwell, and others.

See also: [https://xiph.org/daala/ https://xiph.org/daala/]

== Weekly meetings ==

We've been having weekly progress meetings on mumble.

* 2012 June 4 [https://people.xiph.org/~giles/2012/daala_20120604.txt minutes] (actually a work week)
* 2012 June 22 [https://people.xiph.org/~giles/2012/daala_20120622.txt minutes]
* 2012 June 29 [https://people.xiph.org/~giles/2012/daala_20120629.txt minutes] [https://people.xiph.org/~giles/2012/daala_20120629.opus recording]
* 2012 July 6 [https://people.xiph.org/~giles/2012/daala_20120706.txt minutes]
* 2012 July 13 [https://people.xiph.org/~giles/2012/daala_20120713.txt minutes]
* 2012 July 20 [https://people.xiph.org/~giles/2012/daala_20120720.txt minutes]
* 2012 July 27 [https://people.xiph.org/~giles/2012/daala_20120727.txt minutes]
* 2012 August 3 [https://people.xiph.org/~giles/2012/daala_20120803.txt minutes] [https://people.xiph.org/~giles/2012/daala_20120803.opus recording]

= Techniques =

The discussed overall structure so far has been a variable size lapped-DCT block based codec with lapping done via pre/post filtering with a specially structured (lifting) linear phase transform along the edges along with overlapped block motion compensation and the expected trimmings. The lapping can be optimized for energy compaction and other useful properties, including invert-ability, and yields excellent results with efficient finite precision math.

Other components which have been discussed include:

==Techniques applicable to all frame types==
* Multisymbol arithmetic coding
** Timothy has some trial code showing speed-up proportional to the number of bits coded at once. (ec_test.c)
* Mode prediction using the previously decoded data, e.g. coding the mode using a probability function derived from trained predictors on the surrounding blocks.
** This will be terrible for robustness but may significantly reduce signalling overhead, allowing many more modes, and provide continuous adaptation between signalling free and fully signalled modes.
* Explore legendre polynomial basis transforms instead of DCT
** May have better perceptual properties and/or result in 'less compromised' efficient implementations.
* Coefficient domain prediction to allow efficient energy preserving quantization.
* Variable partition size/shape and the use of good predictors appears to remove most of the benefit of directional transforms.
** Perhaps 45deg is still useful?
** How does this change with partition sizes? Directional transforms are clearly not that useful with 4x4.
* Transform-post filtering to allow merging smaller transform blocks (like TF merging in CELT) may allow more flexible partitioning then outright using mixed block sizes.
* Perturbed quantization mode-signalling has been discussed but mostly laughed at. ;)
* Special block modes well suited to solid color/cartoon like content— avoiding ringing.
** Are pixel prediction modes too slow?
* In general— what markov random field techniques can be applied with acceptable performance. Any?
* Designed for parallel encode and decode within each frame
** Important because
*** the proposed techniques need a lot more CPU than H.264 and VP8 for both encode and decode
*** Moore's law for single-threaded throughput is dead. Future hardware is all multicore/GPU.
** Implies
*** Getting the order of application right for the lapping filters.
*** Mandatory slicing? Maybe some kind of multilevel entropy coding to reduce redundancy between slices while minimizing the single-threaded portion of decode.

* Using PVQ and energy conservation: see http://jmvalin.ca/video/video_pvq_v3.pdf

==Techniques applicable to inter frames==
* Using x264 as a test-bed Jason and Loren demonstrated 15% rate/distortion improvements from using 10-bit intermediaries and references, estimated as being 1/3rd from quality calculation in the 10-bit space, 1/3rd from the higher precision references, and 1/3rd from higher intermediate precision in calculations (e.g. MC filter processing).
** Increased reference precision competes for memory with increased number of references. The improvements demonstrated appear to be a greater win than increasing the reference count once there are four references or so.
* Super-resolution techniques for motion-compensation references have been discussed— in particular it appears that the half-pel location is where intelligent filtering matters the most so staged computation could be effectively used to allow more expensive filtering at that level.
** Edge-directed interpolation techniques might be effectively applied to increase motion compensation accuracy, but most of the techniques known to be very effective are too slow.
** Speculation has been offered that a significant part of MC inaccuracy may be due to blending in a physically incorrect (gamma-corrected) space, though no real conclusions were made. Academic papers on motion compensation accuracy seem to have ignored this issue.
* Timothy has an example code base for a variable partition size blocking-free motion compensation scheme which merges OBMC (overlapped block motion compensation) and CGI (control-grid interpolation) with an interesting prediction/sub-division scheme and whole-frame trellis optimization of motion vectors. (daala-exp)

==Basic features==
* YUV 4:4:4, 4:2:2 , 4:2:0 subsamplings, 8-bit, 10bit.
* Alpha channel — need testing material!
* 8-bit RGB compatible mode? (e.g. YCoCg, internally or at least flagging for it)
* Efficient 3D? — need testing material!
* Lossless?
** The value of this is disputable. If nothing else it's arguable that stuffing lossless into a lossy format may be the only way to get lossless into many people's hands. Also, see below
* Good support for decode side droppable frames?
** Hopefully the referencing structure will be flexible enough to enable this even if it's not an intentional feature.

==Frills==
* Optionally storing a checksum of the expected decoded frame for decoder/encoder mismatch detection.
* Expose the number of referential descendants of a given frame (or even the whole reference DAG) for most efficient allocation of FEC.

==Wingdings==
Crazy crap that might be interesting or at least fun to make fun of...
* >10bit?
** Use cases don't seem well enough defined yet. Significant complexity. Any prospective hardware developer may hire assassins.
** Possible compromise: the video reference structure contains a backbone that can be decoded at only N bits of depth (e.g. 10), and higher precisions are only supported outside of this reference chain.
**# Precision by truncation: decode is performed twice on each frame, identically, at low and high precision. The only difference between them is the bit-depth of the transform, or possibly of the transform and MC filters. Only low-precision outputs can be referenced by subsequent frames. Useful if high-precision content is still worth watching at low precision.
**# Precision by gamma: decode is performed once at low precision as normal. Then the output frame is converted to linear-light at high precision, after which another layer of residuals is added. The second layer can be permitted to reference previous high-precision frames... tricky to use both sets of references though. Useful if high precision is used for storing linear data, but people still want to watch it on "low-end" hardware.
* Some high end digital cameras are operating jpeg-derivatives in a special mode that keeps the image in the native linear RGB bayer format in order to avoid lossy/slow demosaicing on the camera. In particular this allows white balancing in post without excessive loss. Probably out of scope for Daala itself.
** Bayer, 4:2:0, 4:2:2, and Interlacing are all special cases of a more general pattern in which the output frames are decimated/subsampled in a regular fashion. All such subsamplings could be supported by a unified framework in which the video is always stored with all planes fully sampled, with a header indicating the recommended subsampling for display. In such cases, the encoder can regard the transform as highly overcomplete, and simply ignore unneeded coefficients (presumably by leaving high frequency residuals coded as zero). This structure would in effect turn the codec into a motion-compensated interpolating/deinterlacing filter. Whether this approach is sensible presumably depends in part on how the transform is structured. It would be especially easy if the transform's highest-frequencies were coded by a wavelet-like layer.
* Lossless intra-ability: The ability to losslessly rewrite any frame as an intra frame (perhaps with significant bitrate overhead) in order to make frame accurate cuts possible.
** Or best handled by making sure that containers have working pre-roll, but presumably common GOP sizes will be greater than the number of references so even if losslessly reencoding the references is expensive it may be cheaper than pre-roll. Do both?
** Can be had for 'free' if lossless is supported, plus the right header flags to restuff the references from lossless copies in a packed hidden frame.
** Use of explicitly (rather than staged) super-resolution and/or deeper references may make this functionality unattractive due to increased overhead.
* Internal overlays which could be swapped without re-encoding? (e.g. advertising, station ID). Could also be automatically generated by a Sufficiently Advanced™ encoder to improve efficiencies for static sprites over moving backgrounds.
** Complicates making the complexity bounded. No Sufficiently Advanced™ encoder likely to ever exist. But perhaps the station id/advertising uses fully justify this.
** Could be done externally to the video codec, but if so it's no likely to be useful for anyone ever.
* A secondary reference implementation in OpenCL, maintained throughout development, to make sure that the codec is GPU-friendly and can be done efficiently using OpenCL primitives.
* SWAR-friendly arithmetic. For example, choosing transform coefficients so that no intermediate product overflows 16 bits (tricky for signed values) can sometimes enable (e.g.) 4 parallel operations in one uint64_t. This can allow a pure C reference implementation to run faster, which is valuable for initial adoption and ports to new platforms.
* Parametric decode-side blur.
** Symmetrical blur in regions that are smooth on scales longer than the block size. Could be signaled or derived from observed DC values.
** Motion blur so that moving objects are blurred along the motion vector. May require coding a shutter speed parameter (0..1 as a fraction of the inter-frame interval).
* Fancy block property prediction. (Not clear how these prediction interact with intra pred)
** Predict block properties (quantizer, energy, etc.) from MV. (0,0) probably means small delta. Larger MV's may correspond to larger deltas ... although at low shutter speeds large MVs may correlate with reduced overall HF energy.
** Predict delta spectral shape from source block spectral shape. HF/LF ratio of the delta may be correlated with the same ratio in its source blocks. Works well with decode-side fDCT.

==Negative results==

* Using Kurtosis for detecting text in a frame
** The idea was to detect a Bernouilli distribution but it's not robust and too noisy

Daala

2012-08-03T21:46:43Z

Rillian: /* Weekly meetings */ post today's meeting minutes

Daala is the current working name of a next generation video codec— to be renamed once someone insists on something better. So far the best proposed alternative is PatentCake.

For now the purposes of this page is to collect notes about things which have been discussed in informal public IRC discussion about the next generation initiative. Participants in these discussions have included Timothy Terriberry, Jason Garrett-Glaser, Loren Merritt, Ben Schwartz, Greg Maxwell, and others.

See also: [https://xiph.org/daala/ https://xiph.org/daala/]

== Weekly meetings ==

We've been having weekly progress meetings on mumble.

* 2012 June 4 [https://people.xiph.org/~giles/2012/daala_20120604.txt minutes] (actually a work week)
* 2012 June 22 [https://people.xiph.org/~giles/2012/daala_20120622.txt minutes]
* 2012 June 29 [https://people.xiph.org/~giles/2012/daala_20120629.txt minutes] [https://people.xiph.org/~giles/2012/daala_20120629.opus recording]
* 2012 July 6 [https://people.xiph.org/~giles/2012/daala_20120706.txt minutes]
* 2012 July 13 [https://people.xiph.org/~giles/2012/daala_20120713.txt minutes]
* 2012 July 20 [https://people.xiph.org/~giles/2012/daala_20120720.txt minutes]
* 2012 July 27 [https://people.xiph.org/~giles/2012/daala_20120727.txt minutes]
* 2012 August 3 [https://people.xiph.org/~giles/2012/daala_20120803.txt minutes]

= Techniques =

The discussed overall structure so far has been a variable size lapped-DCT block based codec with lapping done via pre/post filtering with a specially structured (lifting) linear phase transform along the edges along with overlapped block motion compensation and the expected trimmings. The lapping can be optimized for energy compaction and other useful properties, including invert-ability, and yields excellent results with efficient finite precision math.

Other components which have been discussed include:

==Techniques applicable to all frame types==
* Multisymbol arithmetic coding
** Timothy has some trial code showing speed-up proportional to the number of bits coded at once. (ec_test.c)
* Mode prediction using the previously decoded data, e.g. coding the mode using a probability function derived from trained predictors on the surrounding blocks.
** This will be terrible for robustness but may significantly reduce signalling overhead, allowing many more modes, and provide continuous adaptation between signalling free and fully signalled modes.
* Explore legendre polynomial basis transforms instead of DCT
** May have better perceptual properties and/or result in 'less compromised' efficient implementations.
* Coefficient domain prediction to allow efficient energy preserving quantization.
* Variable partition size/shape and the use of good predictors appears to remove most of the benefit of directional transforms.
** Perhaps 45deg is still useful?
** How does this change with partition sizes? Directional transforms are clearly not that useful with 4x4.
* Transform-post filtering to allow merging smaller transform blocks (like TF merging in CELT) may allow more flexible partitioning then outright using mixed block sizes.
* Perturbed quantization mode-signalling has been discussed but mostly laughed at. ;)
* Special block modes well suited to solid color/cartoon like content— avoiding ringing.
** Are pixel prediction modes too slow?
* In general— what markov random field techniques can be applied with acceptable performance. Any?
* Designed for parallel encode and decode within each frame
** Important because
*** the proposed techniques need a lot more CPU than H.264 and VP8 for both encode and decode
*** Moore's law for single-threaded throughput is dead. Future hardware is all multicore/GPU.
** Implies
*** Getting the order of application right for the lapping filters.
*** Mandatory slicing? Maybe some kind of multilevel entropy coding to reduce redundancy between slices while minimizing the single-threaded portion of decode.

* Using PVQ and energy conservation: see http://jmvalin.ca/video/video_pvq_v3.pdf

==Techniques applicable to inter frames==
* Using x264 as a test-bed Jason and Loren demonstrated 15% rate/distortion improvements from using 10-bit intermediaries and references, estimated as being 1/3rd from quality calculation in the 10-bit space, 1/3rd from the higher precision references, and 1/3rd from higher intermediate precision in calculations (e.g. MC filter processing).
** Increased reference precision competes for memory with increased number of references. The improvements demonstrated appear to be a greater win than increasing the reference count once there are four references or so.
* Super-resolution techniques for motion-compensation references have been discussed— in particular it appears that the half-pel location is where intelligent filtering matters the most so staged computation could be effectively used to allow more expensive filtering at that level.
** Edge-directed interpolation techniques might be effectively applied to increase motion compensation accuracy, but most of the techniques known to be very effective are too slow.
** Speculation has been offered that a significant part of MC inaccuracy may be due to blending in a physically incorrect (gamma-corrected) space, though no real conclusions were made. Academic papers on motion compensation accuracy seem to have ignored this issue.
* Timothy has an example code base for a variable partition size blocking-free motion compensation scheme which merges OBMC (overlapped block motion compensation) and CGI (control-grid interpolation) with an interesting prediction/sub-division scheme and whole-frame trellis optimization of motion vectors. (daala-exp)

==Basic features==
* YUV 4:4:4, 4:2:2 , 4:2:0 subsamplings, 8-bit, 10bit.
* Alpha channel — need testing material!
* 8-bit RGB compatible mode? (e.g. YCoCg, internally or at least flagging for it)
* Efficient 3D? — need testing material!
* Lossless?
** The value of this is disputable. If nothing else it's arguable that stuffing lossless into a lossy format may be the only way to get lossless into many people's hands. Also, see below
* Good support for decode side droppable frames?
** Hopefully the referencing structure will be flexible enough to enable this even if it's not an intentional feature.

==Frills==
* Optionally storing a checksum of the expected decoded frame for decoder/encoder mismatch detection.
* Expose the number of referential descendants of a given frame (or even the whole reference DAG) for most efficient allocation of FEC.

==Wingdings==
Crazy crap that might be interesting or at least fun to make fun of...
* >10bit?
** Use cases don't seem well enough defined yet. Significant complexity. Any prospective hardware developer may hire assassins.
** Possible compromise: the video reference structure contains a backbone that can be decoded at only N bits of depth (e.g. 10), and higher precisions are only supported outside of this reference chain.
**# Precision by truncation: decode is performed twice on each frame, identically, at low and high precision. The only difference between them is the bit-depth of the transform, or possibly of the transform and MC filters. Only low-precision outputs can be referenced by subsequent frames. Useful if high-precision content is still worth watching at low precision.
**# Precision by gamma: decode is performed once at low precision as normal. Then the output frame is converted to linear-light at high precision, after which another layer of residuals is added. The second layer can be permitted to reference previous high-precision frames... tricky to use both sets of references though. Useful if high precision is used for storing linear data, but people still want to watch it on "low-end" hardware.
* Some high end digital cameras are operating jpeg-derivatives in a special mode that keeps the image in the native linear RGB bayer format in order to avoid lossy/slow demosaicing on the camera. In particular this allows white balancing in post without excessive loss. Probably out of scope for Daala itself.
** Bayer, 4:2:0, 4:2:2, and Interlacing are all special cases of a more general pattern in which the output frames are decimated/subsampled in a regular fashion. All such subsamplings could be supported by a unified framework in which the video is always stored with all planes fully sampled, with a header indicating the recommended subsampling for display. In such cases, the encoder can regard the transform as highly overcomplete, and simply ignore unneeded coefficients (presumably by leaving high frequency residuals coded as zero). This structure would in effect turn the codec into a motion-compensated interpolating/deinterlacing filter. Whether this approach is sensible presumably depends in part on how the transform is structured. It would be especially easy if the transform's highest-frequencies were coded by a wavelet-like layer.
* Lossless intra-ability: The ability to losslessly rewrite any frame as an intra frame (perhaps with significant bitrate overhead) in order to make frame accurate cuts possible.
** Or best handled by making sure that containers have working pre-roll, but presumably common GOP sizes will be greater than the number of references so even if losslessly reencoding the references is expensive it may be cheaper than pre-roll. Do both?
** Can be had for 'free' if lossless is supported, plus the right header flags to restuff the references from lossless copies in a packed hidden frame.
** Use of explicitly (rather than staged) super-resolution and/or deeper references may make this functionality unattractive due to increased overhead.
* Internal overlays which could be swapped without re-encoding? (e.g. advertising, station ID). Could also be automatically generated by a Sufficiently Advanced™ encoder to improve efficiencies for static sprites over moving backgrounds.
** Complicates making the complexity bounded. No Sufficiently Advanced™ encoder likely to ever exist. But perhaps the station id/advertising uses fully justify this.
** Could be done externally to the video codec, but if so it's no likely to be useful for anyone ever.
* A secondary reference implementation in OpenCL, maintained throughout development, to make sure that the codec is GPU-friendly and can be done efficiently using OpenCL primitives.
* SWAR-friendly arithmetic. For example, choosing transform coefficients so that no intermediate product overflows 16 bits (tricky for signed values) can sometimes enable (e.g.) 4 parallel operations in one uint64_t. This can allow a pure C reference implementation to run faster, which is valuable for initial adoption and ports to new platforms.
* Parametric decode-side blur.
** Symmetrical blur in regions that are smooth on scales longer than the block size. Could be signaled or derived from observed DC values.
** Motion blur so that moving objects are blurred along the motion vector. May require coding a shutter speed parameter (0..1 as a fraction of the inter-frame interval).
* Fancy block property prediction. (Not clear how these prediction interact with intra pred)
** Predict block properties (quantizer, energy, etc.) from MV. (0,0) probably means small delta. Larger MV's may correspond to larger deltas ... although at low shutter speeds large MVs may correlate with reduced overall HF energy.
** Predict delta spectral shape from source block spectral shape. HF/LF ratio of the delta may be correlated with the same ratio in its source blocks. Works well with decode-side fDCT.

==Negative results==

* Using Kurtosis for detecting text in a frame
** The idea was to detect a Bernouilli distribution but it's not robust and too noisy

Daala

2012-07-29T15:44:18Z

Rillian: /* Weekly meetings */ link minutes from this week's meeting

Daala is the current working name of a next generation video codec— to be renamed once someone insists on something better. So far the best proposed alternative is PatentCake.

For now the purposes of this page is to collect notes about things which have been discussed in informal public IRC discussion about the next generation initiative. Participants in these discussions have included Timothy Terriberry, Jason Garrett-Glaser, Loren Merritt, Ben Schwartz, Greg Maxwell, and others.

See also: [https://xiph.org/daala/ https://xiph.org/daala/]

== Weekly meetings ==

We've been having weekly progress meetings on mumble.

* 2012 June 4 [https://people.xiph.org/~giles/2012/daala_20120604.txt minutes] (actually a work week)
* 2012 June 22 [https://people.xiph.org/~giles/2012/daala_20120622.txt minutes]
* 2012 June 29 [https://people.xiph.org/~giles/2012/daala_20120629.txt minutes] [https://people.xiph.org/~giles/2012/daala_20120629.opus recording]
* 2012 July 6 [https://people.xiph.org/~giles/2012/daala_20120706.txt minutes]
* 2012 July 13 [https://people.xiph.org/~giles/2012/daala_20120713.txt minutes]
* 2012 July 20 [https://people.xiph.org/~giles/2012/daala_20120720.txt minutes]
* 2012 July 27 [https://people.xiph.org/~giles/2012/daala_20120727.txt minutes]

= Techniques =

The discussed overall structure so far has been a variable size lapped-DCT block based codec with lapping done via pre/post filtering with a specially structured (lifting) linear phase transform along the edges along with overlapped block motion compensation and the expected trimmings. The lapping can be optimized for energy compaction and other useful properties, including invert-ability, and yields excellent results with efficient finite precision math.

Other components which have been discussed include:

==Techniques applicable to all frame types==
* Multisymbol arithmetic coding
** Timothy has some trial code showing speed-up proportional to the number of bits coded at once. (ec_test.c)
* Mode prediction using the previously decoded data, e.g. coding the mode using a probability function derived from trained predictors on the surrounding blocks.
** This will be terrible for robustness but may significantly reduce signalling overhead, allowing many more modes, and provide continuous adaptation between signalling free and fully signalled modes.
* Explore legendre polynomial basis transforms instead of DCT
** May have better perceptual properties and/or result in 'less compromised' efficient implementations.
* Coefficient domain prediction to allow efficient energy preserving quantization.
* Variable partition size/shape and the use of good predictors appears to remove most of the benefit of directional transforms.
** Perhaps 45deg is still useful?
** How does this change with partition sizes? Directional transforms are clearly not that useful with 4x4.
* Transform-post filtering to allow merging smaller transform blocks (like TF merging in CELT) may allow more flexible partitioning then outright using mixed block sizes.
* Perturbed quantization mode-signalling has been discussed but mostly laughed at. ;)
* Special block modes well suited to solid color/cartoon like content— avoiding ringing.
** Are pixel prediction modes too slow?
* In general— what markov random field techniques can be applied with acceptable performance. Any?
* Designed for parallel encode and decode within each frame
** Important because
*** the proposed techniques need a lot more CPU than H.264 and VP8 for both encode and decode
*** Moore's law for single-threaded throughput is dead. Future hardware is all multicore/GPU.
** Implies
*** Getting the order of application right for the lapping filters.
*** Mandatory slicing? Maybe some kind of multilevel entropy coding to reduce redundancy between slices while minimizing the single-threaded portion of decode.

* Using PVQ and energy conservation: see http://jmvalin.ca/video/video_pvq_v3.pdf

==Techniques applicable to inter frames==
* Using x264 as a test-bed Jason and Loren demonstrated 15% rate/distortion improvements from using 10-bit intermediaries and references, estimated as being 1/3rd from quality calculation in the 10-bit space, 1/3rd from the higher precision references, and 1/3rd from higher intermediate precision in calculations (e.g. MC filter processing).
** Increased reference precision competes for memory with increased number of references. The improvements demonstrated appear to be a greater win than increasing the reference count once there are four references or so.
* Super-resolution techniques for motion-compensation references have been discussed— in particular it appears that the half-pel location is where intelligent filtering matters the most so staged computation could be effectively used to allow more expensive filtering at that level.
** Edge-directed interpolation techniques might be effectively applied to increase motion compensation accuracy, but most of the techniques known to be very effective are too slow.
** Speculation has been offered that a significant part of MC inaccuracy may be due to blending in a physically incorrect (gamma-corrected) space, though no real conclusions were made. Academic papers on motion compensation accuracy seem to have ignored this issue.
* Timothy has an example code base for a variable partition size blocking-free motion compensation scheme which merges OBMC (overlapped block motion compensation) and CGI (control-grid interpolation) with an interesting prediction/sub-division scheme and whole-frame trellis optimization of motion vectors. (daala-exp)

==Basic features==
* YUV 4:4:4, 4:2:2 , 4:2:0 subsamplings, 8-bit, 10bit.
* Alpha channel — need testing material!
* 8-bit RGB compatible mode? (e.g. YCoCg, internally or at least flagging for it)
* Efficient 3D? — need testing material!
* Lossless?
** The value of this is disputable. If nothing else it's arguable that stuffing lossless into a lossy format may be the only way to get lossless into many people's hands. Also, see below
* Good support for decode side droppable frames?
** Hopefully the referencing structure will be flexible enough to enable this even if it's not an intentional feature.

==Frills==
* Optionally storing a checksum of the expected decoded frame for decoder/encoder mismatch detection.
* Expose the number of referential descendants of a given frame (or even the whole reference DAG) for most efficient allocation of FEC.

==Wingdings==
Crazy crap that might be interesting or at least fun to make fun of...
* >10bit?
** Use cases don't seem well enough defined yet. Significant complexity. Any prospective hardware developer may hire assassins.
** Possible compromise: the video reference structure contains a backbone that can be decoded at only N bits of depth (e.g. 10), and higher precisions are only supported outside of this reference chain.
**# Precision by truncation: decode is performed twice on each frame, identically, at low and high precision. The only difference between them is the bit-depth of the transform, or possibly of the transform and MC filters. Only low-precision outputs can be referenced by subsequent frames. Useful if high-precision content is still worth watching at low precision.
**# Precision by gamma: decode is performed once at low precision as normal. Then the output frame is converted to linear-light at high precision, after which another layer of residuals is added. The second layer can be permitted to reference previous high-precision frames... tricky to use both sets of references though. Useful if high precision is used for storing linear data, but people still want to watch it on "low-end" hardware.
* Some high end digital cameras are operating jpeg-derivatives in a special mode that keeps the image in the native linear RGB bayer format in order to avoid lossy/slow demosaicing on the camera. In particular this allows white balancing in post without excessive loss. Probably out of scope for Daala itself.
** Bayer, 4:2:0, 4:2:2, and Interlacing are all special cases of a more general pattern in which the output frames are decimated/subsampled in a regular fashion. All such subsamplings could be supported by a unified framework in which the video is always stored with all planes fully sampled, with a header indicating the recommended subsampling for display. In such cases, the encoder can regard the transform as highly overcomplete, and simply ignore unneeded coefficients (presumably by leaving high frequency residuals coded as zero). This structure would in effect turn the codec into a motion-compensated interpolating/deinterlacing filter. Whether this approach is sensible presumably depends in part on how the transform is structured. It would be especially easy if the transform's highest-frequencies were coded by a wavelet-like layer.
* Lossless intra-ability: The ability to losslessly rewrite any frame as an intra frame (perhaps with significant bitrate overhead) in order to make frame accurate cuts possible.
** Or best handled by making sure that containers have working pre-roll, but presumably common GOP sizes will be greater than the number of references so even if losslessly reencoding the references is expensive it may be cheaper than pre-roll. Do both?
** Can be had for 'free' if lossless is supported, plus the right header flags to restuff the references from lossless copies in a packed hidden frame.
** Use of explicitly (rather than staged) super-resolution and/or deeper references may make this functionality unattractive due to increased overhead.
* Internal overlays which could be swapped without re-encoding? (e.g. advertising, station ID). Could also be automatically generated by a Sufficiently Advanced™ encoder to improve efficiencies for static sprites over moving backgrounds.
** Complicates making the complexity bounded. No Sufficiently Advanced™ encoder likely to ever exist. But perhaps the station id/advertising uses fully justify this.
** Could be done externally to the video codec, but if so it's no likely to be useful for anyone ever.
* A secondary reference implementation in OpenCL, maintained throughout development, to make sure that the codec is GPU-friendly and can be done efficiently using OpenCL primitives.
* SWAR-friendly arithmetic. For example, choosing transform coefficients so that no intermediate product overflows 16 bits (tricky for signed values) can sometimes enable (e.g.) 4 parallel operations in one uint64_t. This can allow a pure C reference implementation to run faster, which is valuable for initial adoption and ports to new platforms.
* Parametric decode-side blur.
** Symmetrical blur in regions that are smooth on scales longer than the block size. Could be signaled or derived from observed DC values.
** Motion blur so that moving objects are blurred along the motion vector. May require coding a shutter speed parameter (0..1 as a fraction of the inter-frame interval).
* Fancy block property prediction. (Not clear how these prediction interact with intra pred)
** Predict block properties (quantizer, energy, etc.) from MV. (0,0) probably means small delta. Larger MV's may correspond to larger deltas ... although at low shutter speeds large MVs may correlate with reduced overall HF energy.
** Predict delta spectral shape from source block spectral shape. HF/LF ratio of the delta may be correlated with the same ratio in its source blocks. Works well with decode-side fDCT.

==Negative results==

* Using Kurtosis for detecting text in a frame
** The idea was to detect a Bernouilli distribution but it's not robust and too noisy

Daala

2012-07-21T14:19:11Z

Rillian: /* Weekly meetings */ Fix date on the latest minutes.

Daala is the current working name of a next generation video codec— to be renamed once someone insists on something better. So far the best proposed alternative is PatentCake.

For now the purposes of this page is to collect notes about things which have been discussed in informal public IRC discussion about the next generation initiative. Participants in these discussions have included Timothy Terriberry, Jason Garrett-Glaser, Loren Merritt, Ben Schwartz, Greg Maxwell, and others.

See also: [https://xiph.org/daala/ https://xiph.org/daala/]

== Weekly meetings ==

We've been having weekly progress meetings on mumble.

* 2012 June 4 [https://people.xiph.org/~giles/2012/daala_20120604.txt minutes] (actually a work week)
* 2012 June 22 [https://people.xiph.org/~giles/2012/daala_20120622.txt minutes]
* 2012 June 29 [https://people.xiph.org/~giles/2012/daala_20120629.txt minutes] [https://people.xiph.org/~giles/2012/daala_20120629.opus recording]
* 2012 July 6 [https://people.xiph.org/~giles/2012/daala_20120706.txt minutes]
* 2012 July 13 [https://people.xiph.org/~giles/2012/daala_20120713.txt minutes]
* 2012 July 20 [https://people.xiph.org/~giles/2012/daala_20120720.txt minutes]

= Techniques =

The discussed overall structure so far has been a variable size lapped-DCT block based codec with lapping done via pre/post filtering with a specially structured (lifting) linear phase transform along the edges along with overlapped block motion compensation and the expected trimmings. The lapping can be optimized for energy compaction and other useful properties, including invert-ability, and yields excellent results with efficient finite precision math.

Other components which have been discussed include:

==Techniques applicable to all frame types==
* Multisymbol arithmetic coding
** Timothy has some trial code showing speed-up proportional to the number of bits coded at once. (ec_test.c)
* Mode prediction using the previously decoded data, e.g. coding the mode using a probability function derived from trained predictors on the surrounding blocks.
** This will be terrible for robustness but may significantly reduce signalling overhead, allowing many more modes, and provide continuous adaptation between signalling free and fully signalled modes.
* Explore legendre polynomial basis transforms instead of DCT
** May have better perceptual properties and/or result in 'less compromised' efficient implementations.
* Coefficient domain prediction to allow efficient energy preserving quantization.
* Variable partition size/shape and the use of good predictors appears to remove most of the benefit of directional transforms.
** Perhaps 45deg is still useful?
** How does this change with partition sizes? Directional transforms are clearly not that useful with 4x4.
* Transform-post filtering to allow merging smaller transform blocks (like TF merging in CELT) may allow more flexible partitioning then outright using mixed block sizes.
* Perturbed quantization mode-signalling has been discussed but mostly laughed at. ;)
* Special block modes well suited to solid color/cartoon like content— avoiding ringing.
** Are pixel prediction modes too slow?
* In general— what markov random field techniques can be applied with acceptable performance. Any?
* Designed for parallel encode and decode within each frame
** Important because
*** the proposed techniques need a lot more CPU than H.264 and VP8 for both encode and decode
*** Moore's law for single-threaded throughput is dead. Future hardware is all multicore/GPU.
** Implies
*** Getting the order of application right for the lapping filters.
*** Mandatory slicing? Maybe some kind of multilevel entropy coding to reduce redundancy between slices while minimizing the single-threaded portion of decode.

* Using PVQ and energy conservation: see http://jmvalin.ca/video/video_pvq_v3.pdf

==Techniques applicable to inter frames==
* Using x264 as a test-bed Jason and Loren demonstrated 15% rate/distortion improvements from using 10-bit intermediaries and references, estimated as being 1/3rd from quality calculation in the 10-bit space, 1/3rd from the higher precision references, and 1/3rd from higher intermediate precision in calculations (e.g. MC filter processing).
** Increased reference precision competes for memory with increased number of references. The improvements demonstrated appear to be a greater win than increasing the reference count once there are four references or so.
* Super-resolution techniques for motion-compensation references have been discussed— in particular it appears that the half-pel location is where intelligent filtering matters the most so staged computation could be effectively used to allow more expensive filtering at that level.
** Edge-directed interpolation techniques might be effectively applied to increase motion compensation accuracy, but most of the techniques known to be very effective are too slow.
** Speculation has been offered that a significant part of MC inaccuracy may be due to blending in a physically incorrect (gamma-corrected) space, though no real conclusions were made. Academic papers on motion compensation accuracy seem to have ignored this issue.
* Timothy has an example code base for a variable partition size blocking-free motion compensation scheme which merges OBMC (overlapped block motion compensation) and CGI (control-grid interpolation) with an interesting prediction/sub-division scheme and whole-frame trellis optimization of motion vectors. (daala-exp)

==Basic features==
* YUV 4:4:4, 4:2:2 , 4:2:0 subsamplings, 8-bit, 10bit.
* Alpha channel — need testing material!
* 8-bit RGB compatible mode? (e.g. YCoCg, internally or at least flagging for it)
* Efficient 3D? — need testing material!
* Lossless?
** The value of this is disputable. If nothing else it's arguable that stuffing lossless into a lossy format may be the only way to get lossless into many people's hands. Also, see below
* Good support for decode side droppable frames?
** Hopefully the referencing structure will be flexible enough to enable this even if it's not an intentional feature.

==Frills==
* Optionally storing a checksum of the expected decoded frame for decoder/encoder mismatch detection.
* Expose the number of referential descendants of a given frame (or even the whole reference DAG) for most efficient allocation of FEC.

==Wingdings==
Crazy crap that might be interesting or at least fun to make fun of...
* >10bit?
** Use cases don't seem well enough defined yet. Significant complexity. Any prospective hardware developer may hire assassins.
** Possible compromise: the video reference structure contains a backbone that can be decoded at only N bits of depth (e.g. 10), and higher precisions are only supported outside of this reference chain.
**# Precision by truncation: decode is performed twice on each frame, identically, at low and high precision. The only difference between them is the bit-depth of the transform, or possibly of the transform and MC filters. Only low-precision outputs can be referenced by subsequent frames. Useful if high-precision content is still worth watching at low precision.
**# Precision by gamma: decode is performed once at low precision as normal. Then the output frame is converted to linear-light at high precision, after which another layer of residuals is added. The second layer can be permitted to reference previous high-precision frames... tricky to use both sets of references though. Useful if high precision is used for storing linear data, but people still want to watch it on "low-end" hardware.
* Some high end digital cameras are operating jpeg-derivatives in a special mode that keeps the image in the native linear RGB bayer format in order to avoid lossy/slow demosaicing on the camera. In particular this allows white balancing in post without excessive loss. Probably out of scope for Daala itself.
** Bayer, 4:2:0, 4:2:2, and Interlacing are all special cases of a more general pattern in which the output frames are decimated/subsampled in a regular fashion. All such subsamplings could be supported by a unified framework in which the video is always stored with all planes fully sampled, with a header indicating the recommended subsampling for display. In such cases, the encoder can regard the transform as highly overcomplete, and simply ignore unneeded coefficients (presumably by leaving high frequency residuals coded as zero). This structure would in effect turn the codec into a motion-compensated interpolating/deinterlacing filter. Whether this approach is sensible presumably depends in part on how the transform is structured. It would be especially easy if the transform's highest-frequencies were coded by a wavelet-like layer.
* Lossless intra-ability: The ability to losslessly rewrite any frame as an intra frame (perhaps with significant bitrate overhead) in order to make frame accurate cuts possible.
** Or best handled by making sure that containers have working pre-roll, but presumably common GOP sizes will be greater than the number of references so even if losslessly reencoding the references is expensive it may be cheaper than pre-roll. Do both?
** Can be had for 'free' if lossless is supported, plus the right header flags to restuff the references from lossless copies in a packed hidden frame.
** Use of explicitly (rather than staged) super-resolution and/or deeper references may make this functionality unattractive due to increased overhead.
* Internal overlays which could be swapped without re-encoding? (e.g. advertising, station ID). Could also be automatically generated by a Sufficiently Advanced™ encoder to improve efficiencies for static sprites over moving backgrounds.
** Complicates making the complexity bounded. No Sufficiently Advanced™ encoder likely to ever exist. But perhaps the station id/advertising uses fully justify this.
** Could be done externally to the video codec, but if so it's no likely to be useful for anyone ever.
* A secondary reference implementation in OpenCL, maintained throughout development, to make sure that the codec is GPU-friendly and can be done efficiently using OpenCL primitives.
* SWAR-friendly arithmetic. For example, choosing transform coefficients so that no intermediate product overflows 16 bits (tricky for signed values) can sometimes enable (e.g.) 4 parallel operations in one uint64_t. This can allow a pure C reference implementation to run faster, which is valuable for initial adoption and ports to new platforms.
* Parametric decode-side blur.
** Symmetrical blur in regions that are smooth on scales longer than the block size. Could be signaled or derived from observed DC values.
** Motion blur so that moving objects are blurred along the motion vector. May require coding a shutter speed parameter (0..1 as a fraction of the inter-frame interval).
* Fancy block property prediction. (Not clear how these prediction interact with intra pred)
** Predict block properties (quantizer, energy, etc.) from MV. (0,0) probably means small delta. Larger MV's may correspond to larger deltas ... although at low shutter speeds large MVs may correlate with reduced overall HF energy.
** Predict delta spectral shape from source block spectral shape. HF/LF ratio of the delta may be correlated with the same ratio in its source blocks. Works well with decode-side fDCT.

==Negative results==

* Using Kurtosis for detecting text in a frame
** The idea was to detect a Bernouilli distribution but it's not robust and too noisy

Daala

2012-07-20T22:11:30Z

Rillian: /* Weekly meetings */ link minutes from this week's meeting

Daala is the current working name of a next generation video codec— to be renamed once someone insists on something better. So far the best proposed alternative is PatentCake.

For now the purposes of this page is to collect notes about things which have been discussed in informal public IRC discussion about the next generation initiative. Participants in these discussions have included Timothy Terriberry, Jason Garrett-Glaser, Loren Merritt, Ben Schwartz, Greg Maxwell, and others.

See also: [https://xiph.org/daala/ https://xiph.org/daala/]

== Weekly meetings ==

We've been having weekly progress meetings on mumble.

* 2012 June 4 [https://people.xiph.org/~giles/2012/daala_20120604.txt minutes] (actually a work week)
* 2012 June 22 [https://people.xiph.org/~giles/2012/daala_20120622.txt minutes]
* 2012 June 29 [https://people.xiph.org/~giles/2012/daala_20120629.txt minutes] [https://people.xiph.org/~giles/2012/daala_20120629.opus recording]
* 2012 July 6 [https://people.xiph.org/~giles/2012/daala_20120706.txt minutes]
* 2012 July 13 [https://people.xiph.org/~giles/2012/daala_20120713.txt minutes]
* 2012 July 13 [https://people.xiph.org/~giles/2012/daala_20120720.txt minutes]

= Techniques =

The discussed overall structure so far has been a variable size lapped-DCT block based codec with lapping done via pre/post filtering with a specially structured (lifting) linear phase transform along the edges along with overlapped block motion compensation and the expected trimmings. The lapping can be optimized for energy compaction and other useful properties, including invert-ability, and yields excellent results with efficient finite precision math.

Other components which have been discussed include:

==Techniques applicable to all frame types==
* Multisymbol arithmetic coding
** Timothy has some trial code showing speed-up proportional to the number of bits coded at once. (ec_test.c)
* Mode prediction using the previously decoded data, e.g. coding the mode using a probability function derived from trained predictors on the surrounding blocks.
** This will be terrible for robustness but may significantly reduce signalling overhead, allowing many more modes, and provide continuous adaptation between signalling free and fully signalled modes.
* Explore legendre polynomial basis transforms instead of DCT
** May have better perceptual properties and/or result in 'less compromised' efficient implementations.
* Coefficient domain prediction to allow efficient energy preserving quantization.
* Variable partition size/shape and the use of good predictors appears to remove most of the benefit of directional transforms.
** Perhaps 45deg is still useful?
** How does this change with partition sizes? Directional transforms are clearly not that useful with 4x4.
* Transform-post filtering to allow merging smaller transform blocks (like TF merging in CELT) may allow more flexible partitioning then outright using mixed block sizes.
* Perturbed quantization mode-signalling has been discussed but mostly laughed at. ;)
* Special block modes well suited to solid color/cartoon like content— avoiding ringing.
** Are pixel prediction modes too slow?
* In general— what markov random field techniques can be applied with acceptable performance. Any?
* Designed for parallel encode and decode within each frame
** Important because
*** the proposed techniques need a lot more CPU than H.264 and VP8 for both encode and decode
*** Moore's law for single-threaded throughput is dead. Future hardware is all multicore/GPU.
** Implies
*** Getting the order of application right for the lapping filters.
*** Mandatory slicing? Maybe some kind of multilevel entropy coding to reduce redundancy between slices while minimizing the single-threaded portion of decode.

* Using PVQ and energy conservation: see http://jmvalin.ca/video/video_pvq_v3.pdf

==Techniques applicable to inter frames==
* Using x264 as a test-bed Jason and Loren demonstrated 15% rate/distortion improvements from using 10-bit intermediaries and references, estimated as being 1/3rd from quality calculation in the 10-bit space, 1/3rd from the higher precision references, and 1/3rd from higher intermediate precision in calculations (e.g. MC filter processing).
** Increased reference precision competes for memory with increased number of references. The improvements demonstrated appear to be a greater win than increasing the reference count once there are four references or so.
* Super-resolution techniques for motion-compensation references have been discussed— in particular it appears that the half-pel location is where intelligent filtering matters the most so staged computation could be effectively used to allow more expensive filtering at that level.
** Edge-directed interpolation techniques might be effectively applied to increase motion compensation accuracy, but most of the techniques known to be very effective are too slow.
** Speculation has been offered that a significant part of MC inaccuracy may be due to blending in a physically incorrect (gamma-corrected) space, though no real conclusions were made. Academic papers on motion compensation accuracy seem to have ignored this issue.
* Timothy has an example code base for a variable partition size blocking-free motion compensation scheme which merges OBMC (overlapped block motion compensation) and CGI (control-grid interpolation) with an interesting prediction/sub-division scheme and whole-frame trellis optimization of motion vectors. (daala-exp)

==Basic features==
* YUV 4:4:4, 4:2:2 , 4:2:0 subsamplings, 8-bit, 10bit.
* Alpha channel — need testing material!
* 8-bit RGB compatible mode? (e.g. YCoCg, internally or at least flagging for it)
* Efficient 3D? — need testing material!
* Lossless?
** The value of this is disputable. If nothing else it's arguable that stuffing lossless into a lossy format may be the only way to get lossless into many people's hands. Also, see below
* Good support for decode side droppable frames?
** Hopefully the referencing structure will be flexible enough to enable this even if it's not an intentional feature.

==Frills==
* Optionally storing a checksum of the expected decoded frame for decoder/encoder mismatch detection.
* Expose the number of referential descendants of a given frame (or even the whole reference DAG) for most efficient allocation of FEC.

==Wingdings==
Crazy crap that might be interesting or at least fun to make fun of...
* >10bit?
** Use cases don't seem well enough defined yet. Significant complexity. Any prospective hardware developer may hire assassins.
** Possible compromise: the video reference structure contains a backbone that can be decoded at only N bits of depth (e.g. 10), and higher precisions are only supported outside of this reference chain.
**# Precision by truncation: decode is performed twice on each frame, identically, at low and high precision. The only difference between them is the bit-depth of the transform, or possibly of the transform and MC filters. Only low-precision outputs can be referenced by subsequent frames. Useful if high-precision content is still worth watching at low precision.
**# Precision by gamma: decode is performed once at low precision as normal. Then the output frame is converted to linear-light at high precision, after which another layer of residuals is added. The second layer can be permitted to reference previous high-precision frames... tricky to use both sets of references though. Useful if high precision is used for storing linear data, but people still want to watch it on "low-end" hardware.
* Some high end digital cameras are operating jpeg-derivatives in a special mode that keeps the image in the native linear RGB bayer format in order to avoid lossy/slow demosaicing on the camera. In particular this allows white balancing in post without excessive loss. Probably out of scope for Daala itself.
** Bayer, 4:2:0, 4:2:2, and Interlacing are all special cases of a more general pattern in which the output frames are decimated/subsampled in a regular fashion. All such subsamplings could be supported by a unified framework in which the video is always stored with all planes fully sampled, with a header indicating the recommended subsampling for display. In such cases, the encoder can regard the transform as highly overcomplete, and simply ignore unneeded coefficients (presumably by leaving high frequency residuals coded as zero). This structure would in effect turn the codec into a motion-compensated interpolating/deinterlacing filter. Whether this approach is sensible presumably depends in part on how the transform is structured. It would be especially easy if the transform's highest-frequencies were coded by a wavelet-like layer.
* Lossless intra-ability: The ability to losslessly rewrite any frame as an intra frame (perhaps with significant bitrate overhead) in order to make frame accurate cuts possible.
** Or best handled by making sure that containers have working pre-roll, but presumably common GOP sizes will be greater than the number of references so even if losslessly reencoding the references is expensive it may be cheaper than pre-roll. Do both?
** Can be had for 'free' if lossless is supported, plus the right header flags to restuff the references from lossless copies in a packed hidden frame.
** Use of explicitly (rather than staged) super-resolution and/or deeper references may make this functionality unattractive due to increased overhead.
* Internal overlays which could be swapped without re-encoding? (e.g. advertising, station ID). Could also be automatically generated by a Sufficiently Advanced™ encoder to improve efficiencies for static sprites over moving backgrounds.
** Complicates making the complexity bounded. No Sufficiently Advanced™ encoder likely to ever exist. But perhaps the station id/advertising uses fully justify this.
** Could be done externally to the video codec, but if so it's no likely to be useful for anyone ever.
* A secondary reference implementation in OpenCL, maintained throughout development, to make sure that the codec is GPU-friendly and can be done efficiently using OpenCL primitives.
* SWAR-friendly arithmetic. For example, choosing transform coefficients so that no intermediate product overflows 16 bits (tricky for signed values) can sometimes enable (e.g.) 4 parallel operations in one uint64_t. This can allow a pure C reference implementation to run faster, which is valuable for initial adoption and ports to new platforms.
* Parametric decode-side blur.
** Symmetrical blur in regions that are smooth on scales longer than the block size. Could be signaled or derived from observed DC values.
** Motion blur so that moving objects are blurred along the motion vector. May require coding a shutter speed parameter (0..1 as a fraction of the inter-frame interval).
* Fancy block property prediction. (Not clear how these prediction interact with intra pred)
** Predict block properties (quantizer, energy, etc.) from MV. (0,0) probably means small delta. Larger MV's may correspond to larger deltas ... although at low shutter speeds large MVs may correlate with reduced overall HF energy.
** Predict delta spectral shape from source block spectral shape. HF/LF ratio of the delta may be correlated with the same ratio in its source blocks. Works well with decode-side fDCT.

==Negative results==

* Using Kurtosis for detecting text in a frame
** The idea was to detect a Bernouilli distribution but it's not robust and too noisy

Daala

2012-07-20T19:19:53Z

Rillian: /* Weekly meetings */ upload minutes from last week's meeting. I didn't make a recording.

Daala is the current working name of a next generation video codec— to be renamed once someone insists on something better. So far the best proposed alternative is PatentCake.

For now the purposes of this page is to collect notes about things which have been discussed in informal public IRC discussion about the next generation initiative. Participants in these discussions have included Timothy Terriberry, Jason Garrett-Glaser, Loren Merritt, Ben Schwartz, Greg Maxwell, and others.

See also: [https://xiph.org/daala/ https://xiph.org/daala/]

== Weekly meetings ==

We've been having weekly progress meetings on mumble.

* 2012 June 4 [https://people.xiph.org/~giles/2012/daala_20120604.txt minutes] (actually a work week)
* 2012 June 22 [https://people.xiph.org/~giles/2012/daala_20120622.txt minutes]
* 2012 June 29 [https://people.xiph.org/~giles/2012/daala_20120629.txt minutes] [https://people.xiph.org/~giles/2012/daala_20120629.opus recording]
* 2012 July 6 [https://people.xiph.org/~giles/2012/daala_20120706.txt minutes]
* 2012 July 13 [https://people.xiph.org/~giles/2012/daala_20120713.txt minutes]

= Techniques =

The discussed overall structure so far has been a variable size lapped-DCT block based codec with lapping done via pre/post filtering with a specially structured (lifting) linear phase transform along the edges along with overlapped block motion compensation and the expected trimmings. The lapping can be optimized for energy compaction and other useful properties, including invert-ability, and yields excellent results with efficient finite precision math.

Other components which have been discussed include:

==Techniques applicable to all frame types==
* Multisymbol arithmetic coding
** Timothy has some trial code showing speed-up proportional to the number of bits coded at once. (ec_test.c)
* Mode prediction using the previously decoded data, e.g. coding the mode using a probability function derived from trained predictors on the surrounding blocks.
** This will be terrible for robustness but may significantly reduce signalling overhead, allowing many more modes, and provide continuous adaptation between signalling free and fully signalled modes.
* Explore legendre polynomial basis transforms instead of DCT
** May have better perceptual properties and/or result in 'less compromised' efficient implementations.
* Coefficient domain prediction to allow efficient energy preserving quantization.
* Variable partition size/shape and the use of good predictors appears to remove most of the benefit of directional transforms.
** Perhaps 45deg is still useful?
** How does this change with partition sizes? Directional transforms are clearly not that useful with 4x4.
* Transform-post filtering to allow merging smaller transform blocks (like TF merging in CELT) may allow more flexible partitioning then outright using mixed block sizes.
* Perturbed quantization mode-signalling has been discussed but mostly laughed at. ;)
* Special block modes well suited to solid color/cartoon like content— avoiding ringing.
** Are pixel prediction modes too slow?
* In general— what markov random field techniques can be applied with acceptable performance. Any?
* Designed for parallel encode and decode within each frame
** Important because
*** the proposed techniques need a lot more CPU than H.264 and VP8 for both encode and decode
*** Moore's law for single-threaded throughput is dead. Future hardware is all multicore/GPU.
** Implies
*** Getting the order of application right for the lapping filters.
*** Mandatory slicing? Maybe some kind of multilevel entropy coding to reduce redundancy between slices while minimizing the single-threaded portion of decode.

* Using PVQ and energy conservation: see http://jmvalin.ca/video/video_pvq_v3.pdf

==Techniques applicable to inter frames==
* Using x264 as a test-bed Jason and Loren demonstrated 15% rate/distortion improvements from using 10-bit intermediaries and references, estimated as being 1/3rd from quality calculation in the 10-bit space, 1/3rd from the higher precision references, and 1/3rd from higher intermediate precision in calculations (e.g. MC filter processing).
** Increased reference precision competes for memory with increased number of references. The improvements demonstrated appear to be a greater win than increasing the reference count once there are four references or so.
* Super-resolution techniques for motion-compensation references have been discussed— in particular it appears that the half-pel location is where intelligent filtering matters the most so staged computation could be effectively used to allow more expensive filtering at that level.
** Edge-directed interpolation techniques might be effectively applied to increase motion compensation accuracy, but most of the techniques known to be very effective are too slow.
** Speculation has been offered that a significant part of MC inaccuracy may be due to blending in a physically incorrect (gamma-corrected) space, though no real conclusions were made. Academic papers on motion compensation accuracy seem to have ignored this issue.
* Timothy has an example code base for a variable partition size blocking-free motion compensation scheme which merges OBMC (overlapped block motion compensation) and CGI (control-grid interpolation) with an interesting prediction/sub-division scheme and whole-frame trellis optimization of motion vectors. (daala-exp)

==Basic features==
* YUV 4:4:4, 4:2:2 , 4:2:0 subsamplings, 8-bit, 10bit.
* Alpha channel — need testing material!
* 8-bit RGB compatible mode? (e.g. YCoCg, internally or at least flagging for it)
* Efficient 3D? — need testing material!
* Lossless?
** The value of this is disputable. If nothing else it's arguable that stuffing lossless into a lossy format may be the only way to get lossless into many people's hands. Also, see below
* Good support for decode side droppable frames?
** Hopefully the referencing structure will be flexible enough to enable this even if it's not an intentional feature.

==Frills==
* Optionally storing a checksum of the expected decoded frame for decoder/encoder mismatch detection.
* Expose the number of referential descendants of a given frame (or even the whole reference DAG) for most efficient allocation of FEC.

==Wingdings==
Crazy crap that might be interesting or at least fun to make fun of...
* >10bit?
** Use cases don't seem well enough defined yet. Significant complexity. Any prospective hardware developer may hire assassins.
** Possible compromise: the video reference structure contains a backbone that can be decoded at only N bits of depth (e.g. 10), and higher precisions are only supported outside of this reference chain.
**# Precision by truncation: decode is performed twice on each frame, identically, at low and high precision. The only difference between them is the bit-depth of the transform, or possibly of the transform and MC filters. Only low-precision outputs can be referenced by subsequent frames. Useful if high-precision content is still worth watching at low precision.
**# Precision by gamma: decode is performed once at low precision as normal. Then the output frame is converted to linear-light at high precision, after which another layer of residuals is added. The second layer can be permitted to reference previous high-precision frames... tricky to use both sets of references though. Useful if high precision is used for storing linear data, but people still want to watch it on "low-end" hardware.
* Some high end digital cameras are operating jpeg-derivatives in a special mode that keeps the image in the native linear RGB bayer format in order to avoid lossy/slow demosaicing on the camera. In particular this allows white balancing in post without excessive loss. Probably out of scope for Daala itself.
** Bayer, 4:2:0, 4:2:2, and Interlacing are all special cases of a more general pattern in which the output frames are decimated/subsampled in a regular fashion. All such subsamplings could be supported by a unified framework in which the video is always stored with all planes fully sampled, with a header indicating the recommended subsampling for display. In such cases, the encoder can regard the transform as highly overcomplete, and simply ignore unneeded coefficients (presumably by leaving high frequency residuals coded as zero). This structure would in effect turn the codec into a motion-compensated interpolating/deinterlacing filter. Whether this approach is sensible presumably depends in part on how the transform is structured. It would be especially easy if the transform's highest-frequencies were coded by a wavelet-like layer.
* Lossless intra-ability: The ability to losslessly rewrite any frame as an intra frame (perhaps with significant bitrate overhead) in order to make frame accurate cuts possible.
** Or best handled by making sure that containers have working pre-roll, but presumably common GOP sizes will be greater than the number of references so even if losslessly reencoding the references is expensive it may be cheaper than pre-roll. Do both?
** Can be had for 'free' if lossless is supported, plus the right header flags to restuff the references from lossless copies in a packed hidden frame.
** Use of explicitly (rather than staged) super-resolution and/or deeper references may make this functionality unattractive due to increased overhead.
* Internal overlays which could be swapped without re-encoding? (e.g. advertising, station ID). Could also be automatically generated by a Sufficiently Advanced™ encoder to improve efficiencies for static sprites over moving backgrounds.
** Complicates making the complexity bounded. No Sufficiently Advanced™ encoder likely to ever exist. But perhaps the station id/advertising uses fully justify this.
** Could be done externally to the video codec, but if so it's no likely to be useful for anyone ever.
* A secondary reference implementation in OpenCL, maintained throughout development, to make sure that the codec is GPU-friendly and can be done efficiently using OpenCL primitives.
* SWAR-friendly arithmetic. For example, choosing transform coefficients so that no intermediate product overflows 16 bits (tricky for signed values) can sometimes enable (e.g.) 4 parallel operations in one uint64_t. This can allow a pure C reference implementation to run faster, which is valuable for initial adoption and ports to new platforms.
* Parametric decode-side blur.
** Symmetrical blur in regions that are smooth on scales longer than the block size. Could be signaled or derived from observed DC values.
** Motion blur so that moving objects are blurred along the motion vector. May require coding a shutter speed parameter (0..1 as a fraction of the inter-frame interval).
* Fancy block property prediction. (Not clear how these prediction interact with intra pred)
** Predict block properties (quantizer, energy, etc.) from MV. (0,0) probably means small delta. Larger MV's may correspond to larger deltas ... although at low shutter speeds large MVs may correlate with reduced overall HF energy.
** Predict delta spectral shape from source block spectral shape. HF/LF ratio of the delta may be correlated with the same ratio in its source blocks. Works well with decode-side fDCT.

==Negative results==

* Using Kurtosis for detecting text in a frame
** The idea was to detect a Bernouilli distribution but it's not robust and too noisy

Daala

2012-07-09T21:35:18Z

Rillian: /* Weekly meetings */ Minutes from the July 6 daala weekly workshop

Daala is the current working name of a next generation video codec— to be renamed once someone insists on something better. So far the best proposed alternative is PatentCake.

For now the purposes of this page is to collect notes about things which have been discussed in informal public IRC discussion about the next generation initiative. Participants in these discussions have included Timothy Terriberry, Jason Garrett-Glaser, Loren Merritt, Ben Schwartz, Greg Maxwell, and others.

See also: [https://xiph.org/daala/ https://xiph.org/daala/]

== Weekly meetings ==

We've been having weekly progress meetings on mumble.

* 2012 June 4 [https://people.xiph.org/~giles/2012/daala_20120604.txt minutes] (actually a work week)
* 2012 June 22 [https://people.xiph.org/~giles/2012/daala_20120622.txt minutes]
* 2012 June 29 [https://people.xiph.org/~giles/2012/daala_20120629.txt minutes] [https://people.xiph.org/~giles/2012/daala_20120629.opus recording]
* 2012 July 6 [https://people.xiph.org/~giles/2012/daala_20120706.txt minutes]

= Techniques =

The discussed overall structure so far has been a variable size lapped-DCT block based codec with lapping done via pre/post filtering with a specially structured (lifting) linear phase transform along the edges along with overlapped block motion compensation and the expected trimmings. The lapping can be optimized for energy compaction and other useful properties, including invert-ability, and yields excellent results with efficient finite precision math.

Other components which have been discussed include:

==Techniques applicable to all frame types==
* Multisymbol arithmetic coding
** Timothy has some trial code showing speed-up proportional to the number of bits coded at once. (ec_test.c)
* Mode prediction using the previously decoded data, e.g. coding the mode using a probability function derived from trained predictors on the surrounding blocks.
** This will be terrible for robustness but may significantly reduce signalling overhead, allowing many more modes, and provide continuous adaptation between signalling free and fully signalled modes.
* Explore legendre polynomial basis transforms instead of DCT
** May have better perceptual properties and/or result in 'less compromised' efficient implementations.
* Coefficient domain prediction to allow efficient energy preserving quantization.
* Variable partition size/shape and the use of good predictors appears to remove most of the benefit of directional transforms.
** Perhaps 45deg is still useful?
** How does this change with partition sizes? Directional transforms are clearly not that useful with 4x4.
* Transform-post filtering to allow merging smaller transform blocks (like TF merging in CELT) may allow more flexible partitioning then outright using mixed block sizes.
* Perturbed quantization mode-signalling has been discussed but mostly laughed at. ;)
* Special block modes well suited to solid color/cartoon like content— avoiding ringing.
** Are pixel prediction modes too slow?
* In general— what markov random field techniques can be applied with acceptable performance. Any?
* Designed for parallel encode and decode within each frame
** Important because
*** the proposed techniques need a lot more CPU than H.264 and VP8 for both encode and decode
*** Moore's law for single-threaded throughput is dead. Future hardware is all multicore/GPU.
** Implies
*** Getting the order of application right for the lapping filters.
*** Mandatory slicing? Maybe some kind of multilevel entropy coding to reduce redundancy between slices while minimizing the single-threaded portion of decode.

* Using PVQ and energy conservation: see http://jmvalin.ca/video/video_pvq_v3.pdf

==Techniques applicable to inter frames==
* Using x264 as a test-bed Jason and Loren demonstrated 15% rate/distortion improvements from using 10-bit intermediaries and references, estimated as being 1/3rd from quality calculation in the 10-bit space, 1/3rd from the higher precision references, and 1/3rd from higher intermediate precision in calculations (e.g. MC filter processing).
** Increased reference precision competes for memory with increased number of references. The improvements demonstrated appear to be a greater win than increasing the reference count once there are four references or so.
* Super-resolution techniques for motion-compensation references have been discussed— in particular it appears that the half-pel location is where intelligent filtering matters the most so staged computation could be effectively used to allow more expensive filtering at that level.
** Edge-directed interpolation techniques might be effectively applied to increase motion compensation accuracy, but most of the techniques known to be very effective are too slow.
** Speculation has been offered that a significant part of MC inaccuracy may be due to blending in a physically incorrect (gamma-corrected) space, though no real conclusions were made. Academic papers on motion compensation accuracy seem to have ignored this issue.
* Timothy has an example code base for a variable partition size blocking-free motion compensation scheme which merges OBMC (overlapped block motion compensation) and CGI (control-grid interpolation) with an interesting prediction/sub-division scheme and whole-frame trellis optimization of motion vectors. (daala-exp)

==Basic features==
* YUV 4:4:4, 4:2:2 , 4:2:0 subsamplings, 8-bit, 10bit.
* Alpha channel — need testing material!
* 8-bit RGB compatible mode? (e.g. YCoCg, internally or at least flagging for it)
* Efficient 3D? — need testing material!
* Lossless?
** The value of this is disputable. If nothing else it's arguable that stuffing lossless into a lossy format may be the only way to get lossless into many people's hands. Also, see below
* Good support for decode side droppable frames?
** Hopefully the referencing structure will be flexible enough to enable this even if it's not an intentional feature.

==Frills==
* Optionally storing a checksum of the expected decoded frame for decoder/encoder mismatch detection.
* Expose the number of referential descendants of a given frame (or even the whole reference DAG) for most efficient allocation of FEC.

==Wingdings==
Crazy crap that might be interesting or at least fun to make fun of...
* >10bit?
** Use cases don't seem well enough defined yet. Significant complexity. Any prospective hardware developer may hire assassins.
** Possible compromise: the video reference structure contains a backbone that can be decoded at only N bits of depth (e.g. 10), and higher precisions are only supported outside of this reference chain.
**# Precision by truncation: decode is performed twice on each frame, identically, at low and high precision. The only difference between them is the bit-depth of the transform, or possibly of the transform and MC filters. Only low-precision outputs can be referenced by subsequent frames. Useful if high-precision content is still worth watching at low precision.
**# Precision by gamma: decode is performed once at low precision as normal. Then the output frame is converted to linear-light at high precision, after which another layer of residuals is added. The second layer can be permitted to reference previous high-precision frames... tricky to use both sets of references though. Useful if high precision is used for storing linear data, but people still want to watch it on "low-end" hardware.
* Some high end digital cameras are operating jpeg-derivatives in a special mode that keeps the image in the native linear RGB bayer format in order to avoid lossy/slow demosaicing on the camera. In particular this allows white balancing in post without excessive loss. Probably out of scope for Daala itself.
** Bayer, 4:2:0, 4:2:2, and Interlacing are all special cases of a more general pattern in which the output frames are decimated/subsampled in a regular fashion. All such subsamplings could be supported by a unified framework in which the video is always stored with all planes fully sampled, with a header indicating the recommended subsampling for display. In such cases, the encoder can regard the transform as highly overcomplete, and simply ignore unneeded coefficients (presumably by leaving high frequency residuals coded as zero). This structure would in effect turn the codec into a motion-compensated interpolating/deinterlacing filter. Whether this approach is sensible presumably depends in part on how the transform is structured. It would be especially easy if the transform's highest-frequencies were coded by a wavelet-like layer.
* Lossless intra-ability: The ability to losslessly rewrite any frame as an intra frame (perhaps with significant bitrate overhead) in order to make frame accurate cuts possible.
** Or best handled by making sure that containers have working pre-roll, but presumably common GOP sizes will be greater than the number of references so even if losslessly reencoding the references is expensive it may be cheaper than pre-roll. Do both?
** Can be had for 'free' if lossless is supported, plus the right header flags to restuff the references from lossless copies in a packed hidden frame.
** Use of explicitly (rather than staged) super-resolution and/or deeper references may make this functionality unattractive due to increased overhead.
* Internal overlays which could be swapped without re-encoding? (e.g. advertising, station ID). Could also be automatically generated by a Sufficiently Advanced™ encoder to improve efficiencies for static sprites over moving backgrounds.
** Complicates making the complexity bounded. No Sufficiently Advanced™ encoder likely to ever exist. But perhaps the station id/advertising uses fully justify this.
** Could be done externally to the video codec, but if so it's no likely to be useful for anyone ever.
* A secondary reference implementation in OpenCL, maintained throughout development, to make sure that the codec is GPU-friendly and can be done efficiently using OpenCL primitives.
* SWAR-friendly arithmetic. For example, choosing transform coefficients so that no intermediate product overflows 16 bits (tricky for signed values) can sometimes enable (e.g.) 4 parallel operations in one uint64_t. This can allow a pure C reference implementation to run faster, which is valuable for initial adoption and ports to new platforms.
* Parametric decode-side blur.
** Symmetrical blur in regions that are smooth on scales longer than the block size. Could be signaled or derived from observed DC values.
** Motion blur so that moving objects are blurred along the motion vector. May require coding a shutter speed parameter (0..1 as a fraction of the inter-frame interval).
* Fancy block property prediction. (Not clear how these prediction interact with intra pred)
** Predict block properties (quantizer, energy, etc.) from MV. (0,0) probably means small delta. Larger MV's may correspond to larger deltas ... although at low shutter speeds large MVs may correlate with reduced overall HF energy.
** Predict delta spectral shape from source block spectral shape. HF/LF ratio of the delta may be correlated with the same ratio in its source blocks. Works well with decode-side fDCT.

==Negative results==

* Using Kurtosis for detecting text in a frame
** The idea was to detect a Bernouilli distribution but it's not robust and too noisy

OggOpus

2012-07-05T22:44:49Z

Rillian: Link to the ietf draft

'''Superceeded by [http://tools.ietf.org/html/draft-terriberry-oggopus the ietf draft].'''

== Ogg Mapping for Opus ==

The IETF Opus codec is a low-latency audio codec optimized for both voice and general-purpose audio. See the [http://tools.ietf.org/html/draft-ietf-codec-opus Opus Specification] for technical details.

Almost everything about Opus is either fixed or dynamically switchable, so most of the usual ID and setup header parameters in the header packets of an Ogg encapsulation aren't needed. In particular, bitrate, packet duration, mono/stereo flags, and coding modes are all dynamically switchable from packet to packet. The first one or two bytes in each data packet, the start of the 'TOC sequence' that defines the layout of the packet, specifies all of these parameters for that particular packet. See Section 3 of the Opus Specification for the exact format of the TOC sequence.

The remaining parameters that must be signaled are

* The magic number for stream identification,
* The stream count and coupling for multichannel audio, and
* Any metadata or tags.

=== Content Type ===

The recommended mime-type for Ogg Opus files is '''audio/ogg''', defined in [http://www.ietf.org/rfc/rfc5334.txt RFC 5334].

If more specificity is desired, one can distinguish Opus files as 'audio/ogg; codecs=opus'.

The recommended filename extension for Ogg Opus files is '''.opus'''.

=== Packet Organization ===

Opus is framed in a continuous logical [http://www.xiph.org/ogg/doc/framing.html Ogg stream].

There are two mandatory headers. The granule position of the pages containing these headers MUST be zero.

The first packet in the logical Ogg stream MUST contain the identification header, which uniquely identifies a stream as Opus audio. It MUST begin with the 8 bytes "OpusHead". It MUST be placed alone in the first page of the logical Ogg stream. This page MUST have the ’beginning of stream’ flag set.

The second Opus packet MUST contain the comment header. It must begin with the 8 bytes "OpusTags". It MAY span one or more pages, beginning on the second page of the logical stream. However many pages it spans, the comment header packet MUST finish the page on which it ends.

All subsequent pages are audio data pages and the packets they contain are audio data packets. The first audio page SHOULD NOT have the 'continued packet' flag set (which would indicate the first audio packet is continued from a previous page). Packets MUST be placed into Ogg pages in order until the end of stream. Audio packets MAY span page boundaries. A decoder MUST treat a zero-byte audio packet as if it were an Opus packet with an illegal TOC sequence. The last page SHOULD have the 'end of stream' flag set, but implementations should be prepared to deal with truncated streams which do not have a page marked 'end of stream'. The final packet SHOULD complete on the last page, i.e., the final lacing value should be less than 255. There MUST NOT be any more pages in an Opus logical stream after a page marked 'end of stream'.

=== Granule Position ===

The granule position of an audio page encodes the total number of PCM samples in the stream up to and including the last fully-decodable sample from the last packet ''completed'' on that page. A page that is entirely spanned by a single packet (that completes on a subsequent page) has no granule position, and the granule position field MUST be set to the special value ’-1’ in two's complement.

The granule position of an audio page is in units of PCM audio samples at a fixed rate of 48 kHz (per channel; a stereo stream’s granule position does not increment at twice the speed of a mono stream). It is possible to run a decoder at other sampling rates, but the format and this specification always count samples assuming a 48 kHz decoding rate.

The duration of an Opus packet may be any multiple of 2.5 ms, up to a maximum of 120 ms. This duration is encoded in the TOC sequence at the beginning of each packet. The number of samples returned by a decoder corresponds to this duration exactly, even for the first few packets. For example, a 20 ms packet fed to a decoder running at 48 kHz will always return 960 samples. A demuxer can parse these TOC sequences to work backwards or forwards from a packet with a known granule position (i.e., the last packet completed on some page) in order to assign granule positions to every packet, or even every individual sample. The one exception is the last page in the stream, as described below.

All other pages with completed packets after the first MUST have a granule position equal to the number of samples contained in packets that complete on that page plus the granule position of the most recent page with completed packets. This guarantees that a demuxer can assign individual packets the same granule position when working forwards as when working backwards. There must not be any gaps. In order to support capturing a stream that uses discontinuous transmission (DTX), an encoder SHOULD emit packets that explicitly request the use of Packet Loss Concealment (PLC) (i.e., with a frame length of 0, as defined in Section 3.2.1 of the Opus Specification) in place of the packets that were not transmitted.

There is some amount of latency introduced during the decoding process, to allow for overlap in the MDCT modes, stereo mixing in the LP modes, and resampling, and the encoder will introduce even more latency (though the exact amount is not specified). Therefore the first few samples produced by the decoder do not correspond to any real, input audio, but are instead composed of padding inserted by the encoder to compensate for this latency. These samples must be stored and decoded, as Opus is an asymptotically convergent predictive codec, meaning the decoded contents of each frame depend on the recent history of decoder inputs. A 'pre-skip' field in the ID header signals the number of samples which should be skipped at the beginning of the stream. This provides sufficient history to the decoder so that it has already converged before the stream's output begins. It may also be used to perform sample-accurate cropping of existing encoded streams. This amount need not be a multiple of 2.5 ms, may be smaller than a single packet, or may span the contents of several packets.

The PCM sample position is determined from the granule position using the formula

'PCM sample position' = 'granule position' - 'pre-skip' .

For example, if the granule position of the first page is 59971, and the pre-skip is 11971, then the PCM sample position of the last decoded sample from the first page is 48000. This may be converted into a playback time using the formula

'PCM sample position'
'playback time' = --------------------- .
48000.0

The initial PCM sample position before any samples are played is normally '0'. In this case, the PCM sample position of the first audio sample to be played starts at '1', because it marks the time on the clock ''after'' that sample has been played, and a stream that is exactly one second long has a final PCM sample position of '48000', as in the example here.

Vorbis streams use a granule position smaller than the number of audio samples contained in the first page to indicate that some of those samples must be trimmed from the output. However, to do so it requires that the first page contains exactly two packets, in order to allow the decoder to perform PCM position adjustments before needing to return any PCM data. Opus uses the pre-skip mechanism for this purpose instead, since the encoder may introduce more than a single packet's worth of latency, and since very large packets in streams with a very large number of channels may not fit on a single page.

The page with the 'end of stream' flag set MAY have a granule position that indicates the page contains less audio data than would normally be returned by decoding up through the final packet. This is used to end the stream somewhere other than an even frame boundary. The granule position of the most recent audio page with completed packets is used to make this determination, or '0' is used if there were no previous audio pages with a completed packet. The difference between these granule positions indicates how many samples to keep after decoding the packets that completed on the final page. The remaining samples are discarded. The number of discarded samples SHOULD be smaller than the number decoded from the last packet.

The granule position of the first audio page with a completed packet MAY be larger than the number of samples contained in packets that complete on that page, however it MUST NOT be smaller, unless that page has the 'end of stream' flag set. Allowing a granule position larger than the number of samples allows the beginning of a stream to be cropped without rewriting the granule position of all the remaining pages. This means that the PCM sample position just before the first sample to be played may be larger than '0', but the PCM sample position relative to '0' should still be used for the purposes of synchronization when multiplexing with other logical streams. This does not affect the behavior of pre-skip: exactly 'pre-skip' samples should be skipped from the beginning of the decoded output, even if the initial PCM sample position is greater than zero.

On the other hand, a granule position that is smaller than the number of decoded samples prevents a demuxer from working backwards to assign each packet or each individual sample a valid granule position, since granule positions must be non-negative. A decoder MUST reject as invalid any stream where the granule position is smaller than the number of samples contained in packets that complete on the first page with a completed packet, unless that page has the 'end of stream' flag set. It MAY defer this action until it decodes the last packet completed on that page. If that page has the 'end of stream' flag set, a demuxer can work forwards from the granule position '0', but MUST reject as invalid any stream where the granule position is smaller than the 'pre-skip' amount. This would indicate that more samples should be skipped from the initial decoded output than exist in the stream.

==== ID Header ====

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 'O' | 'p' | 'u' | 's' |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 'H' | 'e' | 'a' | 'd' |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| version = 1 | channel count | pre-skip |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| original input sample rate in Hz |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| output gain Q7.8 in dB | channel map | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ :
| |
: optional channel mapping table... :
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Brief description of each field:

- Magic signature: "OpusHead" (64 bits)
- Version number (8 bits unsigned): 0x01 for this spec
- Channel count 'c' (8 bits unsigned): MUST be > 0
- Pre-skip (16 bits unsigned, little endian)
- Input sample rate (32 bits unsigned, little endian): informational only
- Output gain (16 bits, little endian, signed Q7.8 in dB) to apply when
decoding
- Channel mapping family (8 bits unsigned)
-- 0 = one stream: mono or L,R stereo
-- 1 = channels in vorbis spec order: mono or L,R stereo or ... or FL,C,FR,RL,RR,LFE, ...
-- 2..254 = reserved (treat as 255)
-- 255 = no defined channel meaning
If channel mapping family > 0
- Stream count 'N' (8 bits unsigned): MUST be > 0
- Two-channel stream count 'M' (8 bits unsigned): MUST satisfy M <= N, M+N <= 255
- Channel mapping (8*c bits)
-- one stream index (8 bits unsigned) per channel (255 means silent throughout the file)

Detailed definition of each field:

* '''Magic signature'''
The magic signature "OpusHead" allows codec identification and is human readable. Starting with 'Op' helps distinguish it from data packets, as this is an invalid TOC sequence.

* '''Version'''
The version number MUST always be '1' for this version of the encapsulation specification.

Implementations SHOULD treat streams where the upper four bits of the version number match a recognized specification as backwards-compatible with that specification. That is, the version number can be considered split into "major" and "minor" version sub-fields, with changes to the "minor" sub-field in the lower four bits signaling compatible changes. For example, a decoder implementing this specification SHOULD accept any stream with a version number 15 or less, and SHOULD assume any stream with a version number 16 or greater is incompatible. The initial version '1' was chosen to keep implementations from relying on this byte as a null terminator for the OpusHead string.

* '''Channel count''' 'c'
The number of channels byte specifies the number of output channels (1...255) for this Ogg Opus stream.

* '''Pre-skip'''
This is the number of samples (at 48 kHz) to discard from the decoder output when starting playback, and also the number to subtract from a page's granule position to calculate its PCM sample position.

When constructing cropped Ogg Opus streams, a pre-skip of at least 3840 samples (80 ms) is RECOMMENDED to ensure complete convergence.

* '''Input sample rate'''
This is ''not'' the sample rate to use for playback of the encoded data.

Opus has a handful of coding modes, with internal audio bandwidths of 4, 6, 8, 12, and 20 kHz. Each packet in the stream may have a different audio bandwidth. Regardless of the audio bandwidth, the reference decoder supports decoding any stream at a sample rate of 8, 12, 16, 24, or 48 kHz. The original sample rate of the encoder input is not preserved by the lossy compression.

An Ogg Opus player SHOULD select the playback sample rate according to the following procedure:
* If the hardware supports 48 kHz playback, decode at 48 kHz,
* else if the hardware's highest available sample rate is a supported rate, decode at this sample rate,
* else if the hardware's highest available sample rate is less than 48 kHz, decode at the next higher supported rate and resample,
* else decode at 48 kHz and resample.

However, the 'input sample rate' field allows the encoder to pass the sample rate of the original input stream as metadata. This may be useful when the user requires the output sample rate to match the input sample rate. For example, a non-player decoder writing PCM format to disk might choose to resample the output audio back to the original input rate to reduce surprise to the user, who might reasonably expect to get back a file with the same sample rate as the one they fed to the encoder.

A value of zero indicates 'unspecified'. Encoders SHOULD write the actual input rate or zero, but decoder implementations which do something with this field SHOULD take care to behave sanely if given crazy values (e.g. don't
actually upsample the output to 10 MHz if requested).

* '''Output gain'''
This is a gain to be applied by the decoder. Virtually all players and media frameworks should apply it by default. If a player chooses to apply any volume adjustment or gain modification, such as the R128_TRACK_GAIN or a user-facing volume knob, the adjustment MUST be applied ''in addition'' to this output gain in order to achieve playback at the desired volume.

An encoder SHOULD set the output gain to zero, and instead apply any gain prior to encoding, when this is possible and does not conflict with the user's wishes. The output gain should only be nonzero when the gain is adjusted after encoding, or when the user wishes to adjust the gain for playback while preserving the ability to recover the original signal amplitude.

Although the output gain has enormous range (+/- 128 dB, enough to amplify inaudible sounds to the threshold of physical pain), most applications can only reasonably use a small portion of this range around zero. The large range serves in part to ensure that gain can always be losslessly transferred between OpusHead and R128_TRACK_GAIN (see below) without saturating.

The gain is the 20 log<sub>10</sub> ratio of output to input sample values to be applied to the decoder output. E.g. <code>sample *= pow(10, header.gain/(20.*256))</code> where header.gain is the raw 16 bit Q7.8 value from the header.

* '''Channel mapping family'''
This byte indicates the order and semantic meaning of the various channels encoded in each Opus packet.

Each possible value of this byte indicates a ''mapping family'', which defines a set of allowed numbers of channels, and the ordered set of channel names for each allowed number of channels. Currently there are three defined mapping families, although more may be added:

* Family 0 (RTP mapping)
** Allowed numbers of channels: 1 or 2
** 1 channel: monophonic (mono)
** 2 channels: stereo (left, right)
** '''Special mapping''': this channel mapping value also indicates that the contents consists of a single Opus stream that is stereo if and only if c==2, with stream index 0 mapped to channel 0, and (if stereo) stream index 1 mapped to channel 1. When the channel mapping byte has this value, no further fields are present in OpusHead.
* Family 1 ([http://www.xiph.org/vorbis/doc/Vorbis_I_spec.html#x1-800004.3.9 Vorbis channel order])
** Allowed numbers of channels: 1 ... 8
** Channel meanings depend on the number of channels, see the Vorbis mapping for details.
* Family 255 (no defined channel meaning)
** Allowed numbers of channels: 1...255
** Channels are unidentified. General-purpose players SHOULD NOT attempt to play these streams, and offline decoders MAY deinterleave the output into separate PCM files, one per channel. Decoders SHOULD NOT produce output for channels mapped to stream index 255 (pure silence) unless they have no other way to indicate the index of non-silent channels.

The remaining channel mapping families (2...254) are reserved. A decoder encountering a reserved mapping byte should act as though the mapping byte is 255.

An Ogg Opus player MUST play any Ogg Opus stream with a channel mapping family of 0 or 1, even if the number of channels does not match the physically connected audio hardware. Players SHOULD perform channel mixing to increase or reduce the number of channels as needed.

* '''Stream count''' 'N'
This field indicates the total number of streams so the decoder can correctly parse the packed Opus packets inside the Ogg packet.

For channel mapping family 0, this value defaults to 1, and is not coded.

A multi-channel Opus file is composed of one or more individual Opus streams, each of which produce one or two channels of decoded data. Each Ogg packet contains one Opus packet from each stream. The first N-1 Opus packets are packed using the self-delimiting framing from Appendix B of the Opus Specification. The remaining Opus packet is packed using the regular, undelimited framing from Section 3 of the Opus Specification. All the Opus packets in a single Ogg packet MUST be constrained to produce the same number of decoded samples. A decoder SHOULD treat any Opus packet whose duration is different from that of the first Opus packet in an Ogg packet as if it were an Opus packet with an illegal TOC sequence.

* '''Two-channel stream count''' 'M'
Describes the number of streams whose decoders should be configured to produce two channels. This must be no larger than the number of total streams.

For channel mapping family 0, this value defaults to c-1 (i.e., 0 for mono and 1 for stereo), and is not coded.

Each packet in an Opus stream has an internal channel count of 1 or 2, which can change from packet to packet. This is selected by the encoder depending on the bitrate and the contents being encoded. The original channel count of the encoder input is not preserved by the lossy compression.

Regardless of the internal channel count, any Opus stream may be decoded as mono (a single channel) or stereo (two channels) by appropriate initialization of the decoder. The "two-channel stream count" field indicates that the first M Opus decoders should be initialized in stereo mode, and the remaining N-M decoders should be initialized in mono mode. The total number of decoded channels (M+N) MUST be no larger than 255, as there is no way to index more channels than that in the channel mapping.

* '''Channel mapping'''
Contains one index per output channel indicating which decoded channel should be used. If the index is less than 2*M, the output MUST be taken from decoding stream (index/2) as stereo and selecting the left channel if index is even, and the right channel if index is odd. If the index is 2*M or larger, the output MUST be taken from decoding stream (index-M) as mono. As a special case, an index of 255 means that the corresponding output channel MUST contain pure silence.

For channel mapping family 0, the first index defaults to 0, and if c==2, the second index defaults to 1. Neither index is coded.

The number of output channels (c) is not constrained to match the number of decoded channels (M+N). A single index MAY appear multiple times, i.e., the same decoded channel may be mapped to multiple output channels. Some decoded channels might not be assigned to any output channel, as well.

==== Comment Header ====

- 8 byte 'OpusTags' magic signature (64 bits)
- The remaining data follows the vorbis-comment header design used in OggVorbis (without the "framing-bit"), OggTheora, and Speex:
* Vendor string (always present).
** 4-byte little-endian length field, followed by length bytes of UTF-8 vendor string.
* TAG=value metadata strings (zero or more).
** 4-byte little-endian string count.
** Count strings consisting of 4-byte little-endian length and length bytes of UTF-8 string in "tag=value" form.

One new comment field is introduced for Ogg Opus:
R128_TRACK_GAIN=-573
representing the volume shift needed to normalize the track's volume. The gain is a Q7.8 fixed point number in dB, as in the OpusHead "output gain" field. This field is similar to the [[VorbisComment#Replay_Gain|REPLAYGAIN_TRACK_GAIN field in Vorbis]], although the normal volume reference is the [http://tech.ebu.ch/loudness EBU-R128] standard.

An Ogg Opus file MUST NOT have more than one such field, and if present its value MUST be an integer from -32768 to +32767 inclusive, represented in ASCII with no whitespace. If present, it MUST correctly represent the R128 normalization gain (relative to the OpusHead output gain). If a player chooses to make use of the TRACK_GAIN, it MUST be applied ''in addition'' to the OpusHead output gain. If an encoder populates the TRACK_GAIN field, and the output gain is not otherwise constrained or specified, the encoder SHOULD write the R128 gain into the OpusHead output gain and write "R128_TRACK_GAIN=0". If a tool modifies the OpusHead "output gain" field, it MUST also update or remove the R128_TRACK_GAIN comment field.

There is no comment field corresponding to Replaygain's ALBUM_GAIN; that information should instead be stored in the OpusHead 'output gain' field.

To avoid confusion with multiple normalization schemes, an OpusTags packet SHOULD NOT contain any of the REPLAYGAIN_TRACK_GAIN, REPLAYGAIN_TRACK_PEAK, REPLAYGAIN_ALBUM_GAIN, or REPLAYGAIN_ALBUM_PEAK fields.

== Other Implementation Notes ==

When seeking within an Ogg Opus stream, the decoder should start decoding (and discarding the output) at least 3840 samples (80 ms) prior to the seek point in order to ensure that the output audio is correct at the seek point.

Technically valid Opus packets can be arbitrarily large due to the padding format, although the amount of non-padding data they can contain is bounded. These packets may be spread over a similarly enormous number of Ogg pages. Encoders SHOULD use no more padding than required to make a variable bitrate (VBR) stream constant bitrate (CBR). Decoders SHOULD avoid attempting to allocate excessive amounts of memory when presented with a very large packet. The presence of an extremely large packet in the stream could indicate a potential memory exhaustion attack or stream corruption. Decoders should reject a packet that is too large to process, and print a warning message.

In an Ogg Opus stream, the largest possible valid packet that does not use padding has a size of (61,298*N - 2) bytes, or about 60 kB per Opus stream. With 255 streams, this is 15,630,988 bytes (14.9 MB) and can span up to 61,298 Ogg pages, all but one of which will have a granulepos of -1. This is of course a very extreme packet, consisting of 255 streams, each containing 120 ms of audio encoded as 2.5 ms frames, each frame using the maximum possible number of bytes (1275) and stored in the least efficient manner allowed (a VBR code 3 Opus packet). Even in such a packet, most of the data will be zeros, as 2.5 ms frames, which are required to run in the MDCT mode, cannot actually use all 1275 bytes. The largest packet consisting entirely of useful data is (15,326*N - 2) bytes, or about 15 kB per stream. This corresponds to 120 ms of audio encoded as 10 ms frames in either LP or Hybrid mode, but at a data rate of over 1 Mbps, which makes little sense for the quality achieved. A more reasonable limit is (7,664*N - 2) bytes, or about 7.5 kB per stream. This corresponds to 120 ms of audio encoded as 20 ms stereo MDCT-mode frames, with a total bitrate just under 511 kbps (not counting the Ogg encapsulation overhead). With N=8, the maximum useful number of streams for the channel meanings currently defined by mapping family 1, this gives a maximum packet size of 61,310 bytes, or just under 60 kB. This is still quite conservative, as it assumes each output channel is taken from one decoded channel of a stereo packet. An implementation could reasonably choose any of these numbers for its internal limits.

== Test Vectors ==

* [[OggOpus/testvectors|Planned test vectors for OggOpus]]
* Opus test vectors

Daala

2012-06-29T23:32:21Z

Rillian: link meeting minutes and mumble recording

Daala is the current working name of a next generation video codec— to be renamed once someone insists on something better. So far the best proposed alternative is PatentCake.

For now the purposes of this page is to collect notes about things which have been discussed in informal public IRC discussion about the next generation initiative. Participants in these discussions have included Timothy Terriberry, Jason Garrett-Glaser, Loren Merritt, Ben Schwartz, Greg Maxwell, and others.

See also: [https://xiph.org/daala/ https://xiph.org/daala/]

== Weekly meetings ==

We've been having weekly progress meetings on mumble.

* 2012 June 4 [https://people.xiph.org/~giles/2012/daala_20120604.txt minutes] (actually a work week)
* 2012 June 22 [https://people.xiph.org/~giles/2012/daala_20120622.txt minutes]
* 2012 June 29 [https://people.xiph.org/~giles/2012/daala_20120629.txt minutes] [https://people.xiph.org/~giles/2012/daala_20120629.opus recording]

= Techniques =

The discussed overall structure so far has been a variable size lapped-DCT block based codec with lapping done via pre/post filtering with a specially structured (lifting) linear phase transform along the edges along with overlapped block motion compensation and the expected trimmings. The lapping can be optimized for energy compaction and other useful properties, including invert-ability, and yields excellent results with efficient finite precision math.

Other components which have been discussed include:

==Techniques applicable to all frame types==
* Multisymbol arithmetic coding
** Timothy has some trial code showing speed-up proportional to the number of bits coded at once. (ec_test.c)
* Mode prediction using the previously decoded data, e.g. coding the mode using a probability function derived from trained predictors on the surrounding blocks.
** This will be terrible for robustness but may significantly reduce signalling overhead, allowing many more modes, and provide continuous adaptation between signalling free and fully signalled modes.
* Explore legendre polynomial basis transforms instead of DCT
** May have better perceptual properties and/or result in 'less compromised' efficient implementations.
* Coefficient domain prediction to allow efficient energy preserving quantization.
* Variable partition size/shape and the use of good predictors appears to remove most of the benefit of directional transforms.
** Perhaps 45deg is still useful?
** How does this change with partition sizes? Directional transforms are clearly not that useful with 4x4.
* Transform-post filtering to allow merging smaller transform blocks (like TF merging in CELT) may allow more flexible partitioning then outright using mixed block sizes.
* Perturbed quantization mode-signalling has been discussed but mostly laughed at. ;)
* Special block modes well suited to solid color/cartoon like content— avoiding ringing.
** Are pixel prediction modes too slow?
* In general— what markov random field techniques can be applied with acceptable performance. Any?
* Designed for parallel encode and decode within each frame
** Important because
*** the proposed techniques need a lot more CPU than H.264 and VP8 for both encode and decode
*** Moore's law for single-threaded throughput is dead. Future hardware is all multicore/GPU.
** Implies
*** Getting the order of application right for the lapping filters.
*** Mandatory slicing? Maybe some kind of multilevel entropy coding to reduce redundancy between slices while minimizing the single-threaded portion of decode.

* Using PVQ and energy conservation: see http://jmvalin.ca/video/video_pvq_v3.pdf

==Techniques applicable to inter frames==
* Using x264 as a test-bed Jason and Loren demonstrated 15% rate/distortion improvements from using 10-bit intermediaries and references, estimated as being 1/3rd from quality calculation in the 10-bit space, 1/3rd from the higher precision references, and 1/3rd from higher intermediate precision in calculations (e.g. MC filter processing).
** Increased reference precision competes for memory with increased number of references. The improvements demonstrated appear to be a greater win than increasing the reference count once there are four references or so.
* Super-resolution techniques for motion-compensation references have been discussed— in particular it appears that the half-pel location is where intelligent filtering matters the most so staged computation could be effectively used to allow more expensive filtering at that level.
** Edge-directed interpolation techniques might be effectively applied to increase motion compensation accuracy, but most of the techniques known to be very effective are too slow.
** Speculation has been offered that a significant part of MC inaccuracy may be due to blending in a physically incorrect (gamma-corrected) space, though no real conclusions were made. Academic papers on motion compensation accuracy seem to have ignored this issue.
* Timothy has an example code base for a variable partition size blocking-free motion compensation scheme which merges OBMC (overlapped block motion compensation) and CGI (control-grid interpolation) with an interesting prediction/sub-division scheme and whole-frame trellis optimization of motion vectors. (daala-exp)

==Basic features==
* YUV 4:4:4, 4:2:2 , 4:2:0 subsamplings, 8-bit, 10bit.
* Alpha channel — need testing material!
* 8-bit RGB compatible mode? (e.g. YCoCg, internally or at least flagging for it)
* Efficient 3D? — need testing material!
* Lossless?
** The value of this is disputable. If nothing else it's arguable that stuffing lossless into a lossy format may be the only way to get lossless into many people's hands. Also, see below
* Good support for decode side droppable frames?
** Hopefully the referencing structure will be flexible enough to enable this even if it's not an intentional feature.

==Frills==
* Optionally storing a checksum of the expected decoded frame for decoder/encoder mismatch detection.
* Expose the number of referential descendants of a given frame (or even the whole reference DAG) for most efficient allocation of FEC.

==Wingdings==
Crazy crap that might be interesting or at least fun to make fun of...
* >10bit?
** Use cases don't seem well enough defined yet. Significant complexity. Any prospective hardware developer may hire assassins.
** Possible compromise: the video reference structure contains a backbone that can be decoded at only N bits of depth (e.g. 10), and higher precisions are only supported outside of this reference chain.
**# Precision by truncation: decode is performed twice on each frame, identically, at low and high precision. The only difference between them is the bit-depth of the transform, or possibly of the transform and MC filters. Only low-precision outputs can be referenced by subsequent frames. Useful if high-precision content is still worth watching at low precision.
**# Precision by gamma: decode is performed once at low precision as normal. Then the output frame is converted to linear-light at high precision, after which another layer of residuals is added. The second layer can be permitted to reference previous high-precision frames... tricky to use both sets of references though. Useful if high precision is used for storing linear data, but people still want to watch it on "low-end" hardware.
* Some high end digital cameras are operating jpeg-derivatives in a special mode that keeps the image in the native linear RGB bayer format in order to avoid lossy/slow demosaicing on the camera. In particular this allows white balancing in post without excessive loss. Probably out of scope for Daala itself.
** Bayer, 4:2:0, 4:2:2, and Interlacing are all special cases of a more general pattern in which the output frames are decimated/subsampled in a regular fashion. All such subsamplings could be supported by a unified framework in which the video is always stored with all planes fully sampled, with a header indicating the recommended subsampling for display. In such cases, the encoder can regard the transform as highly overcomplete, and simply ignore unneeded coefficients (presumably by leaving high frequency residuals coded as zero). This structure would in effect turn the codec into a motion-compensated interpolating/deinterlacing filter. Whether this approach is sensible presumably depends in part on how the transform is structured. It would be especially easy if the transform's highest-frequencies were coded by a wavelet-like layer.
* Lossless intra-ability: The ability to losslessly rewrite any frame as an intra frame (perhaps with significant bitrate overhead) in order to make frame accurate cuts possible.
** Or best handled by making sure that containers have working pre-roll, but presumably common GOP sizes will be greater than the number of references so even if losslessly reencoding the references is expensive it may be cheaper than pre-roll. Do both?
** Can be had for 'free' if lossless is supported, plus the right header flags to restuff the references from lossless copies in a packed hidden frame.
** Use of explicitly (rather than staged) super-resolution and/or deeper references may make this functionality unattractive due to increased overhead.
* Internal overlays which could be swapped without re-encoding? (e.g. advertising, station ID). Could also be automatically generated by a Sufficiently Advanced™ encoder to improve efficiencies for static sprites over moving backgrounds.
** Complicates making the complexity bounded. No Sufficiently Advanced™ encoder likely to ever exist. But perhaps the station id/advertising uses fully justify this.
** Could be done externally to the video codec, but if so it's no likely to be useful for anyone ever.
* A secondary reference implementation in OpenCL, maintained throughout development, to make sure that the codec is GPU-friendly and can be done efficiently using OpenCL primitives.
* SWAR-friendly arithmetic. For example, choosing transform coefficients so that no intermediate product overflows 16 bits (tricky for signed values) can sometimes enable (e.g.) 4 parallel operations in one uint64_t. This can allow a pure C reference implementation to run faster, which is valuable for initial adoption and ports to new platforms.
* Parametric decode-side blur.
** Symmetrical blur in regions that are smooth on scales longer than the block size. Could be signaled or derived from observed DC values.
** Motion blur so that moving objects are blurred along the motion vector. May require coding a shutter speed parameter (0..1 as a fraction of the inter-frame interval).
* Fancy block property prediction. (Not clear how these prediction interact with intra pred)
** Predict block properties (quantizer, energy, etc.) from MV. (0,0) probably means small delta. Larger MV's may correspond to larger deltas ... although at low shutter speeds large MVs may correlate with reduced overall HF energy.
** Predict delta spectral shape from source block spectral shape. HF/LF ratio of the delta may be correlated with the same ratio in its source blocks. Works well with decode-side fDCT.

==Negative results==

* Using Kurtosis for detecting text in a frame
** The idea was to detect a Bernouilli distribution but it's not robust and too noisy

OpusRelease

2012-06-29T21:41:41Z

Rillian: /* Release management / marketing work */ We have opus-tools binaries now

== Opus release planning ==

This page is for organizing todo items for the initial opus release. Some items would best be dribbled out after the initial release while others are more release critical (tools binaries, test vectors, examples)

=== AUTH48 ===

* Update version in Makefile.draft
* Update rfcXXXX strings
* Update sha1 in the build script

=== Release management / marketing work ===

* <strike>Windows binaries</strike>
* <strike>Mac binaries</strike>
* Get binary packages of opus and opus-tools working for major distros

* Ogg Opus test vectors

* More 'brochure' level text about Opus and its advantages

* <strike>Logo</strike>

* More images/graphs/art for the site (PEAQ sweeps?)

* Overhaul of the comparative page

* demo pages
** static demo with music, voice; like [https://people.xiph.org/~giles/2012/opus this one]
** albums
** audiobooks
** remote control encoder (oneman)
** emscripten decoder
** A recording of a remote jamming event (bemasc)
** update Greg's CELT try-tool[http://people.xiph.org/~greg/trial_tool.png] for libopus
** Son-of-[http://people.xiph.org/~xiphmont/demo/celt/demo.html Monty-CELT-demo] (An in-the-weeds technology demo)
*** This may actually need to be a series of two, one covering LP mode and one covering hybrid and the rest of the system.

* Improved documentation (e.g. an API overview doc that isn't a doxygen dump), "idiots guide to using libopus"

* Bugtracker

* Client applications beyond Firefox, esp VLC and foobar2000. See [[OPUS TODO]] for a list.

* Icecast release with opus support [https://github.com/krad-radio/icecast-oneman/commit/35927ca52f8e538eb20d8a185b8c10f1f2e9118a patch here]

* '''Expanding this list'''

=== Development work ===

* fix remaining fixed point overflows (gmaxwell/jmspeex)
* Make opus_custom_demo read/write little endian on BE hosts.
* Boring standard makefiles for many systems (VMS too? haha)
* Oggdropish GUI for opusenc
* Additional tools (validator, gain, udp streaming example)
* libao for opus-tools
* LP-mode CBR (boundary conditions, iterations, harmonic average)
* Fix -fstack-protector-all (e.g. mingw32) or remove it
* <strike>MIN32/MAX32 used stupidly in celt/ where the arguments evaluated twice.</strike>
* <strike>why does --speech + 24kbit make opus tools code stereo where its mono without speech?</strike>
* make test_opus_encode test more stupidly low rates.
* random duration for the first/last frame in opus-tools
* Activity CTL

== Later work ==

* short block high rate waste in mdct mode
* (mdct mode) silence detection
* Prod the second implementation (Tim's) into something (quasi-)releasable

Work In Progress

2012-06-29T21:39:32Z

Rillian: s/CELT/Opus/

* '''General Usage:'''
** [[Ogg_Index]]: Introducting index headers into Ogg
** [[Metadata]]: Various types of Ogg metadata including the [[M3F]] (Multimedia Metadata Format) and [[XMLEmbedding]]
** [[MIME_Types_and_File_Extensions]]: MIME Types and file extensions for Ogg multimedia files
** [[Subtle]]: Subtitling tool for professional use that intends to support most subtitle formats including CMML and OggKate
** [[OggText]]: A generic media mapping for (discontinuous) text codecs into Ogg
** [[ROE]]: A description format for describing the tracks and languages etc. of an Ogg multitrack composition

* '''Compressed Codecs:'''
** [[OggOpus]]: A low-latency lossy audio codec
** [[Daala]]: A next-generation video codec research project
** [[Theora]]: A lossy video codec [[TheoraTodo]]
** [[OggDirac]]: The "next-generation" wavelet based video codec, lossy or lossless
** [[OggMNG]]: A mapping for encapsulating the MNG animation format in Ogg

* '''Uncompressed Codecs:'''
** [[OggKate]]: A codec for karaoke and text encapsulation in Ogg
** [[OggPCM]]: Uncompressed PCM audio, currently being implemented
** [[OggSpots]]: A mapping for encapsulating timed images in Ogg
** [[OggUVS]]: Uncompressed RGB and YUV video

* '''Abandonware''' (nobody working on those as far as we know)
** [[Ghost]]: A "next-generation" audio codec (vapourware so far -- don't hold your breath)
** [[Oggless]]: Embedding Xiph codecs like Vorbis in containers other than Ogg
** [[IceShare]]: P2P content distribution
** [[OggPCM_Draft1]]: Original uncompressed PCM audio proposal
** [[OggRGB]]: Original uncompressed RGB video proposal
** [[OggWrit]]: Text phrase codec (e.g. subtitles)
** [[OggYUV]]: Original uncompressed YUV video proposal

[[Category:Developers stuff]]

OggOpus/testvectors

2012-06-26T00:24:56Z

Rillian: link to greg's file collection

This page lists test vectors needed for OggOpus which are specific to the Ogg mapping (separate from the opus bitstream test vectors, though they do some bitstream testing as a side efffect)

Greg is collecting a draft file set at https://people.xiph.org/~greg/opus_testvectors/

* All test vectors should be chained files with at least two parts
** Chained file where the second link has no pregap and starts with inter frames (to ensure that decoder state is reset)
* Pre-skip (set large pre-skip with a chime "if you just heard a chime, your player is broken")
* Multichannel
** Multichannel stereo (e.g. mono+mono)
** Multichannel w/pre-skip and random channel maps
** Multichannel with silent channels
*** Totally silent multichannel (Should this one be invalid?)
** Multichannel with repeated channels (i.e. one stream used for multiple channels)
** Multichannel with 256 channels
** Mapping tests for the Vorbis mappings (e.g. name of the speaker spoken by each speaker)
* Files with crazy input rate.
* Header-gain set very high with a very quiet input (silent if you don't implement header gain).
* Header-gain set very low with an input that will clip a decoder if the header gain is not done internally.
* Header-gain set very low, and R128_TRACK_GAIN to normalize it
** matching WAV outputs ... but matching to what?
* Single packet per page
* Utterly stuffed pages with constant continued pages
* Pages whose contents are entirely and partially dropped frames (len=0) (maybe redundant with bitstream tests)
* Files with chimes after the end (testing end length chopping)
* File with all opus modes and frame sizes
* Stereo files using many mono frames at the beginning/end
* OpusTags comment values containing very large nonsense comments, duplicate comment values etc.
* Files with non-zero initial granulepos, pre-skip, trimmed last page to check duration calculation

=== Illegal test vectors that MUST fail ===
* Zero streams (N=0)
* Too many two-output streams
** M>N
** M<=N but M+N>255
* Channels mapped to nonexistent stream indices (255 > index >= M+N)
* Illegal OpusTags comments
** Total length larger or shorter than the packet
** Illegal field names
** Illegal field contents
** Illegal field (no "=")
** Multiple R128_TRACK_GAIN comments (should this be required to fail?)
** R128_TRACK_GAIN comments containing illegal values (should this be required to fail?)
*** Non-ASCII encodings of correct-looking values
* All GP==0
* first data granulepos too small
* preskip > final granulepos

OPUS TODO

2012-06-13T17:05:10Z

Rillian: /* 1.0 Launch */ Some progress on these goals

== IETF draft ==
* <s>New comparison tool</s> done in draft-11
* <s>Update test vectors</s> done in draft-11

== Spec ==

* <s>Finish codec draft</s>
* Get draft through the RFC process
* <s>Ogg mapping (including multi-channel)</s>. See: [[OggOpus]]
* Matroska mapping. See: [[MatroskaOpus]]
* RTP payload format

== 1.0 Launch ==
* De-uglify webpage
* <s>Add logo</s>
* FAQ
* Promotional material
* <s>Opus tools releases</s>

== Other ==

* <s>Logo See: [https://bugzilla.mozilla.org/show_bug.cgi?id=689261 Mozilla bug 689261] for some discussion</s>
* Test vectors
* Listening tests
* Documentation (at a minimum every exported symbol should have complete and accurate documentation)
* Add content to opus-codec.org
** The above documentation
** Presentations
** Examples and test results (hyperlink to Monty's demo, gmaxwell's HA results page, etc)
* Oggz-validate (should also validate opus toc)

== Opus-tools ==
* Build infrastructure (e.g. autotools)
* A simple real time streaming example tool
* <s>Multichannel support</s> doneish.
* Replaygain (half done— needs a gain tool)
* <s>Testing (incl. jenkins automation)</s> doneish

== Third party software ==
* Support in ekiga
* Support in mumble
* Support in asterisk
* Support in icecast
* Support in firefox (rtcweb and in ogg)
* Support in VLC
* Support in ogg123
* Support in ffmpeg
* Support in rockbox
* Support in foobar2000
* Support in gstreamer
* Support in mplayer
* Support in xmms
* Support in oggdsf
* Support in xiphqt
* Support in RoarAudio (specs + roard + libroardsp)

== Future work ==
* Smart automatic mode decision
* psymodel based VBR
* Remove copy in inverse MDCT
* Save some float<->int conversions
* Improvements to LP mode CBR (greg has some code)

OPUS TODO

2012-06-13T17:03:38Z

Rillian: /* Spec */ We have a draft for the ogg mapping

== IETF draft ==
* <s>New comparison tool</s> done in draft-11
* <s>Update test vectors</s> done in draft-11

== Spec ==

* <s>Finish codec draft</s>
* Get draft through the RFC process
* <s>Ogg mapping (including multi-channel)</s>. See: [[OggOpus]]
* Matroska mapping. See: [[MatroskaOpus]]
* RTP payload format

== 1.0 Launch ==
* De-uglify webpage
* Add logo
* FAQ
* Promotional material
* Opus tools releases

== Other ==

* <s>Logo See: [https://bugzilla.mozilla.org/show_bug.cgi?id=689261 Mozilla bug 689261] for some discussion</s>
* Test vectors
* Listening tests
* Documentation (at a minimum every exported symbol should have complete and accurate documentation)
* Add content to opus-codec.org
** The above documentation
** Presentations
** Examples and test results (hyperlink to Monty's demo, gmaxwell's HA results page, etc)
* Oggz-validate (should also validate opus toc)

== Opus-tools ==
* Build infrastructure (e.g. autotools)
* A simple real time streaming example tool
* <s>Multichannel support</s> doneish.
* Replaygain (half done— needs a gain tool)
* <s>Testing (incl. jenkins automation)</s> doneish

== Third party software ==
* Support in ekiga
* Support in mumble
* Support in asterisk
* Support in icecast
* Support in firefox (rtcweb and in ogg)
* Support in VLC
* Support in ogg123
* Support in ffmpeg
* Support in rockbox
* Support in foobar2000
* Support in gstreamer
* Support in mplayer
* Support in xmms
* Support in oggdsf
* Support in xiphqt
* Support in RoarAudio (specs + roard + libroardsp)

== Future work ==
* Smart automatic mode decision
* psymodel based VBR
* Remove copy in inverse MDCT
* Save some float<->int conversions
* Improvements to LP mode CBR (greg has some code)

OpusVersions

2012-05-28T19:33:05Z

Rillian: /* Scheme 6: A.B */ scheme 6 is the same as scheme 3

Possible version schemes:

== Scheme 1: A.B.C.D ==

* A=1
* B is the RFC version
* C is for feature improvements
* D is for bugfixes

== Scheme 2: A.B.C ==

* A is the RFC version
* B is for feature improvements
* C is for bugfixes

== Scheme 3: A.B.C ==

* A=1
* B is the RFC version and for feature improvements
* D is for bugfixes

== Scheme 4: A.B.C ==

* A is the RFC versions that aren't both backward- and forward-comparible (e.g. extensions)
* B is for feature improvements and RFC versions that are both backward- and forward-comparible (e.g. clarifications, minor fixes)
* C is for bugfixes

== Scheme 5: A.B.C ==

* A is for any new RFC
* B is for feature improvments compliant with A
* C is for bug fixes

== Scheme 6: A.B ==

* A is for features and RFC changes
* B is for bugfixes

This is Scheme 3 with the initial '1' dropped.

OpusVersions

2012-05-28T19:31:52Z

Rillian: two more possibilities

OggOpus

2012-05-23T23:40:12Z

Rillian: /* Ogg Mapping for Opus */ mime-type and filename extension

== Ogg Mapping for Opus ==

The IETF Opus codec is a low-latency audio codec optimized for both voice and general-purpose audio. See the [http://tools.ietf.org/html/draft-ietf-codec-opus Opus Specification] for technical details.

Almost everything about Opus is either fixed or dynamically switchable, so most of the usual ID and setup header parameters in the header packets of an Ogg encapsulation aren't needed. In particular, bitrate, packet duration, mono/stereo flags, and coding modes are all dynamically switchable from packet to packet. The first one or two bytes in each data packet, the start of the 'TOC sequence' that defines the layout of the packet, specifies all of these parameters for that particular packet. See Section 3 of the Opus Specification for the exact format of the TOC sequence.

The remaining parameters that must be signaled are

* The magic number for stream identification,
* The stream count and coupling for multichannel audio, and
* Any metadata or tags.

=== Content Type ===

The recommended mime-type for Ogg Opus files is '''audio/ogg''', defined in [http://www.ietf.org/rfc/rfc5334.txt RFC 5334].

If more specificity is desired, one can distinguish Opus files as 'audio/ogg; codecs=opus'.

The recommended filename extension for Ogg Opus files is '''.opus'''.

=== Packet Organization ===

Opus is framed in a continuous logical [http://www.xiph.org/ogg/doc/framing.html Ogg stream].

There are two mandatory headers. The granule position of the pages containing these headers MUST be zero.

The first packet in the logical Ogg stream MUST contain the identification header, which uniquely identifies a stream as Opus audio. It MUST begin with the 8 bytes "OpusHead". It MUST be placed alone in the first page of the logical Ogg stream. This page MUST have the ’beginning of stream’ flag set.

The second Opus packet MUST contain the comment header. It must begin with the 8 bytes "OpusTags". It MAY span one or more pages, beginning on the second page of the logical stream. However many pages it spans, the comment header packet MUST finish the page on which it ends.

All subsequent pages are audio data pages and the packets they contain are audio data packets. The first audio page SHOULD NOT have the 'continued packet' flag set (which would indicate the first audio packet is continued from a previous page). Packets MUST be placed into Ogg pages in order until the end of stream. Audio packets MAY span page boundaries. A decoder MUST treat a zero-byte audio packet as if it were an Opus packet with an illegal TOC sequence. The last page SHOULD have the 'end of stream' flag set, but implementations should be prepared to deal with truncated streams which do not have a page marked 'end of stream'. The final packet SHOULD complete on the last page, i.e., the final lacing value should be less than 255. There MUST NOT be any more pages in an Opus logical stream after a page marked 'end of stream'.

=== Granule Position ===

The granule position of an audio page encodes the total number of PCM samples in the stream up to and including the last fully-decodable sample from the last packet ''completed'' on that page. A page that is entirely spanned by a single packet (that completes on a subsequent page) has no granule position, and the granule position field MUST be set to the special value ’-1’ in two's complement.

The granule position of an audio page is in units of PCM audio samples at a fixed rate of 48 kHz (per channel; a stereo stream’s granule position does not increment at twice the speed of a mono stream). It is possible to run a decoder at other sampling rates, but the format and this specification always count samples assuming a 48 kHz decoding rate.

The duration of an Opus packet may be any multiple of 2.5 ms, up to a maximum of 120 ms. This duration is encoded in the TOC sequence at the beginning of each packet. The number of samples returned by a decoder corresponds to this duration exactly, even for the first few packets. For example, a 20 ms packet fed to a decoder running at 48 kHz will always return 960 samples. A demuxer can parse these TOC sequences to work backwards or forwards from a packet with a known granule position (i.e., the last packet completed on some page) in order to assign granule positions to every packet, or even every individual sample. The one exception is the last page in the stream, as described below.

All other pages with completed packets after the first MUST have a granule position equal to the number of samples contained in packets that complete on that page plus the granule position of the most recent page with completed packets. This guarantees that a demuxer can assign individual packets the same granule position when working forwards as when working backwards. There must not be any gaps. In order to support capturing a stream that uses discontinuous transmission (DTX), an encoder SHOULD emit packets that explicitly request the use of Packet Loss Concealment (PLC) (i.e., with a frame length of 0, as defined in Section 3.2.1 of the Opus Specification) in place of the packets that were not transmitted.

There is some amount of latency introduced during the decoding process, to allow for overlap in the MDCT modes, stereo mixing in the LP modes, and resampling, and the encoder will introduce even more latency (though the exact amount is not specified). Therefore the first few samples produced by the decoder do not correspond to any real, input audio, but are instead composed of padding inserted by the encoder to compensate for this latency. These samples must be stored and decoded, as Opus is an asymptotically convergent predictive codec, meaning the decoded contents of each frame depend on the recent history of decoder inputs. A 'pre-skip' field in the ID header signals the number of samples which should be skipped at the beginning of the stream. This provides sufficient history to the decoder so that it has already converged before the stream's output begins. It may also be used to perform sample-accurate cropping of existing encoded streams. This amount need not be a multiple of 2.5 ms, may be smaller than a single packet, or may span the contents of several packets.

The PCM sample position is determined from the granule position using the formula

'PCM sample position' = 'granule position' - 'pre-skip' .

For example, if the granule position of the first page is 59971, and the pre-skip is 11971, then the PCM sample position of the last decoded sample from the first page is 48000. This may be converted into a playback time using the formula

'PCM sample position'
'playback time' = --------------------- .
48000.0

The initial PCM sample position before any samples are played is normally '0'. In this case, the PCM sample position of the first audio sample to be played starts at '1', because it marks the time on the clock ''after'' that sample has been played, and a stream that is exactly one second long has a final PCM sample position of '48000', as in the example here.

Vorbis streams use a granule position smaller than the number of audio samples contained in the first page to indicate that some of those samples must be trimmed from the output. However, to do so it requires that the first page contains exactly two packets, in order to allow the decoder to perform PCM position adjustments before needing to return any PCM data. Opus uses the pre-skip mechanism for this purpose instead, since the encoder may introduce more than a single packet's worth of latency, and since very large packets in streams with a very large number of channels may not fit on a single page.

The page with the 'end of stream' flag set MAY have a granule position that indicates the page contains less audio data than would normally be returned by decoding up through the final packet. This is used to end the stream somewhere other than an even frame boundary. The granule position of the most recent audio page with completed packets is used to make this determination, or '0' is used if there were no previous audio pages with a completed packet. The difference between these granule positions indicates how many samples to keep after decoding the packets that completed on the final page. The remaining samples are discarded. The number of discarded samples SHOULD be smaller than the number decoded from the last packet.

The granule position of the first audio page with a completed packet MAY be larger than the number of samples contained in packets that complete on that page, however it MUST NOT be smaller, unless that page has the 'end of stream' flag set. Allowing a granule position larger than the number of samples allows the beginning of a stream to be cropped without rewriting the granule position of all the remaining pages. This means that the PCM sample position just before the first sample to be played may be larger than '0', but the PCM sample position relative to '0' should still be used for the purposes of synchronization when multiplexing with other logical streams. This does not affect the behavior of pre-skip: exactly 'pre-skip' samples should be skipped from the beginning of the decoded output, even if the initial PCM sample position is greater than zero.

On the other hand, a granule position that is smaller than the number of decoded samples prevents a demuxer from working backwards to assign each packet or each individual sample a valid granule position, since granule positions must be non-negative. A decoder MUST reject as invalid any stream where the granule position is smaller than the number of samples contained in packets that complete on the first page with a completed packet, unless that page has the 'end of stream' flag set. It MAY defer this action until it decodes the last packet completed on that page. If that page has the 'end of stream' flag set, a demuxer can work forwards from the granule position '0', but MUST reject as invalid any stream where the granule position is smaller than the 'pre-skip' amount. This would indicate that more samples should be skipped from the initial decoded output than exist in the stream.

==== ID Header ====

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 'O' | 'p' | 'u' | 's' |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 'H' | 'e' | 'a' | 'd' |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| version = 1 | channel count | pre-skip |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| original input sample rate in Hz |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| output gain Q7.8 in dB | channel map | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ :
| |
: optional channel mapping table... :
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Brief description of each field:

- Magic signature: "OpusHead" (64 bits)
- Version number (8 bits unsigned): 0x01 for this spec
- Channel count 'c' (8 bits unsigned): MUST be > 0
- Pre-skip (16 bits unsigned, little endian)
- Input sample rate (32 bits unsigned, little endian): informational only
- Output gain (16 bits, little endian, signed Q7.8 in dB) to apply when
decoding
- Channel mapping family (8 bits unsigned)
-- 0 = one stream: mono or L,R stereo
-- 1 = channels in vorbis spec order: mono or L,R stereo or ... or FL,C,FR,RL,RR,LFE, ...
-- 2..254 = reserved (treat as 255)
-- 255 = no defined channel meaning
If channel mapping family > 0
- Stream count 'N' (8 bits unsigned): MUST be > 0
- Two-channel stream count 'M' (8 bits unsigned): MUST satisfy M <= N, M+N <= 255
- Channel mapping (8*c bits)
-- one stream index (8 bits unsigned) per channel (255 means silent throughout the file)

Detailed definition of each field:

* '''Magic signature'''
The magic signature "OpusHead" allows codec identification and is human readable. Starting with 'Op' helps distinguish it from data packets, as this is an invalid TOC sequence.

* '''Version'''
The version number MUST always be '1' for this version of the encapsulation specification.

Implementations SHOULD treat streams where the upper four bits of the version number match a recognized specification as backwards-compatible with that specification. That is, the version number can be considered split into "major" and "minor" version sub-fields, with changes to the "minor" sub-field in the lower four bits signaling compatible changes. For example, a decoder implementing this specification SHOULD accept any stream with a version number 15 or less, and SHOULD assume any stream with a version number 16 or greater is incompatible. The initial version '1' was chosen to keep implementations from relying on this byte as a null terminator for the OpusHead string.

* '''Channel count''' 'c'
The number of channels byte specifies the number of output channels (1...255) for this Ogg Opus stream.

* '''Pre-skip'''
This is the number of samples (at 48 kHz) to discard from the decoder output when starting playback, and also the number to subtract from a page's granule position to calculate its PCM sample position.

When constructing cropped Ogg Opus streams, a pre-skip of at least 3840 samples (80 ms) is RECOMMENDED to ensure complete convergence.

* '''Input sample rate'''
This is ''not'' the sample rate to use for playback of the encoded data.

Opus has a handful of coding modes, with internal audio bandwidths of 4, 6, 8, 12, and 20 kHz. Each packet in the stream may have a different audio bandwidth. Regardless of the audio bandwidth, the reference decoder supports decoding any stream at a sample rate of 8, 12, 16, 24, or 48 kHz. The original sample rate of the encoder input is not preserved by the lossy compression.

An Ogg Opus player SHOULD select the playback sample rate according to the following procedure:
* If the hardware supports 48 kHz playback, decode at 48 kHz,
* else if the hardware's highest available sample rate is a supported rate, decode at this sample rate,
* else if the hardware's highest available sample rate is less than 48 kHz, decode at the next higher supported rate and resample,
* else decode at 48 kHz and resample.

However, the 'input sample rate' field allows the encoder to pass the sample rate of the original input stream as metadata. This may be useful when the user requires the output sample rate to match the input sample rate. For example, a non-player decoder writing PCM format to disk might choose to resample the output audio back to the original input rate to reduce surprise to the user, who might reasonably expect to get back a file with the same sample rate as the one they fed to the encoder.

A value of zero indicates 'unspecified'. Encoders SHOULD write the actual input rate or zero, but decoder implementations which do something with this field SHOULD take care to behave sanely if given crazy values (e.g. don't
actually upsample the output to 10 MHz if requested).

* '''Output gain'''
This is a gain to be applied by the decoder. Virtually all players and media frameworks should apply it by default. If a player chooses to apply any volume adjustment or gain modification, such as the R128_TRACK_GAIN or a user-facing volume knob, the adjustment MUST be applied ''in addition'' to this output gain in order to achieve playback at the desired volume.

An encoder SHOULD set the output gain to zero, and instead apply any gain prior to encoding, when this is possible and does not conflict with the user's wishes. The output gain should only be nonzero when the gain is adjusted after encoding, or when the user wishes to adjust the gain for playback while preserving the ability to recover the original signal amplitude.

Although the output gain has enormous range (+/- 128 dB, enough to amplify inaudible sounds to the threshold of physical pain), most applications can only reasonably use a small portion of this range around zero. The large range serves in part to ensure that gain can always be losslessly transferred between OpusHead and R128_TRACK_GAIN (see below) without saturating.

The gain is the 20 log<sub>10</sub> ratio of output to input sample values to be applied to the decoder output. E.g. <code>sample *= pow(10, header.gain/(20.*256))</code> where header.gain is the raw 16 bit Q7.8 value from the header.

* '''Channel mapping family'''
This byte indicates the order and semantic meaning of the various channels encoded in each Opus packet.

Each possible value of this byte indicates a ''mapping family'', which defines a set of allowed numbers of channels, and the ordered set of channel names for each allowed number of channels. Currently there are three defined mapping families, although more may be added:

* Family 0 (RTP mapping)
** Allowed numbers of channels: 1 or 2
** 1 channel: monophonic (mono)
** 2 channels: stereo (left, right)
** '''Special mapping''': this channel mapping value also indicates that the contents consists of a single Opus stream that is stereo if and only if c==2, with stream index 0 mapped to channel 0, and (if stereo) stream index 1 mapped to channel 1. When the channel mapping byte has this value, no further fields are present in OpusHead.
* Family 1 ([http://www.xiph.org/vorbis/doc/Vorbis_I_spec.html#x1-800004.3.9 Vorbis channel order])
** Allowed numbers of channels: 1 ... 8
** Channel meanings depend on the number of channels, see the Vorbis mapping for details.
* Family 255 (no defined channel meaning)
** Allowed numbers of channels: 1...255
** Channels are unidentified. General-purpose players SHOULD NOT attempt to play these streams, and offline decoders MAY deinterleave the output into separate PCM files, one per channel. Decoders SHOULD NOT produce output for channels mapped to stream index 255 (pure silence) unless they have no other way to indicate the index of non-silent channels.

The remaining channel mapping families (2...254) are reserved. A decoder encountering a reserved mapping byte should act as though the mapping byte is 255.

An Ogg Opus player MUST play any Ogg Opus stream with a channel mapping family of 0 or 1, even if the number of channels does not match the physically connected audio hardware. Players SHOULD perform channel mixing to increase or reduce the number of channels as needed.

* '''Stream count''' 'N'
This field indicates the total number of streams so the decoder can correctly parse the packed Opus packets inside the Ogg packet.

For channel mapping family 0, this value defaults to 1, and is not coded.

A multi-channel Opus file is composed of one or more individual Opus streams, each of which produce one or two channels of decoded data. Each Ogg packet contains one Opus packet from each stream. The first N-1 Opus packets are packed using the self-delimiting framing from Appendix B of the Opus Specification. The remaining Opus packet is packed using the regular, undelimited framing from Section 3 of the Opus Specification. All the Opus packets in a single Ogg packet MUST be constrained to produce the same number of decoded samples. A decoder SHOULD treat any Opus packet whose duration is different from that of the first Opus packet in an Ogg packet as if it were an Opus packet with an illegal TOC sequence.

* '''Two-channel stream count''' 'M'
Describes the number of streams whose decoders should be configured to produce two channels. This must be no larger than the number of total streams.

For channel mapping family 0, this value defaults to c-1 (i.e., 0 for mono and 1 for stereo), and is not coded.

Each packet in an Opus stream has an internal channel count of 1 or 2, which can change from packet to packet. This is selected by the encoder depending on the bitrate and the contents being encoded. The original channel count of the encoder input is not preserved by the lossy compression.

Regardless of the internal channel count, any Opus stream may be decoded as mono (a single channel) or stereo (two channels) by appropriate initialization of the decoder. The "two-channel stream count" field indicates that the first M Opus decoders should be initialized in stereo mode, and the remaining N-M decoders should be initialized in mono mode. The total number of decoded channels (M+N) MUST be no larger than 255, as there is no way to index more channels than that in the channel mapping.

* '''Channel mapping'''
Contains one index per output channel indicating which decoded channel should be used. If the index is less than 2*M, the output MUST be taken from decoding stream (index/2) as stereo and selecting the left channel if index is even, and the right channel if index is odd. If the index is 2*M or larger, the output MUST be taken from decoding stream (index-M) as mono. As a special case, an index of 255 means that the corresponding output channel MUST contain pure silence.

For channel mapping family 0, the first index defaults to 0, and if c==2, the second index defaults to 1. Neither index is coded.

The number of output channels (c) is not constrained to match the number of decoded channels (M+N). A single index MAY appear multiple times, i.e., the same decoded channel may be mapped to multiple output channels. Some decoded channels might not be assigned to any output channel, as well.

==== Comment Header ====

- 8 byte 'OpusTags' magic signature (64 bits)
- The remaining data follows the vorbis-comment header design used in OggVorbis (without the "framing-bit"), OggTheora, and Speex:
* Vendor string (always present).
** 4-byte little-endian length field, followed by length bytes of UTF-8 vendor string.
* TAG=value metadata strings (zero or more).
** 4-byte little-endian string count.
** Count strings consisting of 4-byte little-endian length and length bytes of UTF-8 string in "tag=value" form.

One new comment field is introduced for Ogg Opus:
R128_TRACK_GAIN=-573
representing the volume shift needed to normalize the track's volume. The gain is a Q7.8 fixed point number in dB, as in the OpusHead "output gain" field. This field is similar to the [[VorbisComment#Replay_Gain|REPLAYGAIN_TRACK_GAIN field in Vorbis]], although the normal volume reference is the [http://tech.ebu.ch/loudness EBU-R128] standard.

An Ogg Opus file MUST NOT have more than one such field, and if present its value MUST be an integer from -32768 to +32767 inclusive, represented in ASCII with no whitespace. If present, it MUST correctly represent the R128 normalization gain (relative to the OpusHead output gain). If a player chooses to make use of the TRACK_GAIN, it MUST be applied ''in addition'' to the OpusHead output gain. If an encoder populates the TRACK_GAIN field, and the output gain is not otherwise constrained or specified, the encoder SHOULD write the R128 gain into the OpusHead output gain and write "R128_TRACK_GAIN=0". If a tool modifies the OpusHead "output gain" field, it MUST also update or remove the R128_TRACK_GAIN comment field.

There is no comment field corresponding to Replaygain's ALBUM_GAIN; that information should instead be stored in the OpusHead 'output gain' field.

To avoid confusion with multiple normalization schemes, an OpusTags packet SHOULD NOT contain any of the REPLAYGAIN_TRACK_GAIN, REPLAYGAIN_TRACK_PEAK, REPLAYGAIN_ALBUM_GAIN, or REPLAYGAIN_ALBUM_PEAK fields.

== Other Implementation Notes ==

When seeking within an Ogg Opus stream, the decoder should start decoding (and discarding the output) at least 3840 samples (80 ms) prior to the seek point in order to ensure that the output audio is correct at the seek point.

Technically valid Opus packets can be arbitrarily large due to the padding format, although the amount of non-padding data they can contain is bounded. These packets may be spread over a similarly enormous number of Ogg pages. Encoders SHOULD use no more padding than required to make a variable bitrate (VBR) stream constant bitrate (CBR). Decoders SHOULD avoid attempting to allocate excessive amounts of memory when presented with a very large packet. The presence of an extremely large packet in the stream could indicate a potential memory exhaustion attack or stream corruption. Decoders should reject a packet that is too large to process, and print a warning message.

In an Ogg Opus stream, the largest possible valid packet that does not use padding has a size of (61,298*N - 2) bytes, or about 60 kB per Opus stream. With 255 streams, this is 15,630,988 bytes (14.9 MB) and can span up to 61,298 Ogg pages, all but one of which will have a granulepos of -1. This is of course a very extreme packet, consisting of 255 streams, each containing 120 ms of audio encoded as 2.5 ms frames, each frame using the maximum possible number of bytes (1275) and stored in the least efficient manner allowed (a VBR code 3 Opus packet). Even in such a packet, most of the data will be zeros, as 2.5 ms frames, which are required to run in the MDCT mode, cannot actually use all 1275 bytes. The largest packet consisting entirely of useful data is (15,326*N - 2) bytes, or about 15 kB per stream. This corresponds to 120 ms of audio encoded as 10 ms frames in either LP or Hybrid mode, but at a data rate of over 1 Mbps, which makes little sense for the quality achieved. A more reasonable limit is (7,664*N - 2) bytes, or about 7.5 kB per stream. This corresponds to 120 ms of audio encoded as 20 ms stereo MDCT-mode frames, with a total bitrate just under 511 kbps (not counting the Ogg encapsulation overhead). With N=8, the maximum useful number of streams for the channel meanings currently defined by mapping family 1, this gives a maximum packet size of 61,310 bytes, or just under 60 kB. This is still quite conservative, as it assumes each output channel is taken from one decoded channel of a stereo packet. An implementation could reasonably choose any of these numbers for its internal limits.

== Test Vectors ==

* [[OggOpus/testvectors|Planned test vectors for OggOpus]]
* Opus test vectors

OpusRelease

2012-05-21T18:46:50Z

Rillian: /* Release management / marketing work */ link oneman's patch

== Opus release planning ==

This page is for organizing todo items for the initial opus release. Some items would best be dribbled out after the initial release while others are more release critical (tools binaries, test vectors, examples)

=== Release management / marketing work ===

* Windows binaries
* Mac binaries
* Get binary packages of opus and opus-tools working for major distros

* Ogg Opus test vectors

* More 'brochure' level text about Opus and its advantages

* Logo

* More images/graphs/art for the site (PEAQ sweeps?)

* Overhaul of the comparative page

* demo pages
** static demo with music, voice; like [https://people.xiph.org/~giles/2012/opus this one]
** albums
** audiobooks
** remote control encoder (oneman)
** emscripten decoder
** A recording of a remote jamming event (bemasc)
** update Greg's CELT try-tool[http://people.xiph.org/~greg/trial_tool.png] for libopus
** Son-of-[http://people.xiph.org/~xiphmont/demo/celt/demo.html Monty-CELT-demo] (An in-the-weeds technology demo)
*** This may actually need to be a series of two, one covering LP mode and one covering hybrid and the rest of the system.

* Improved documentation (e.g. an API overview doc that isn't a doxygen dump), "idiots guide to using libopus"

* Bugtracker

* Client applications beyond Firefox, esp VLC and foobar2000. See [[OPUS TODO]] for a list.

* Icecast release with opus support [https://github.com/krad-radio/icecast-oneman/commit/35927ca52f8e538eb20d8a185b8c10f1f2e9118a patch here]

=== Development work ===

* fix remaining fixed point overflows (gmaxwell/jmspeex)
* Make opus_custom_demo read/write little endian on BE hosts.
* Boring standard makefiles for many systems (VMS too? haha)
* Oggdropish GUI for opusenc
* Additional tools (validator, gain, udp streaming example)
* libao for opus-tools
* LP-mode CBR (boundary conditions, iterations, harmonic average)
* Fix -fstack-protector-all (e.g. mingw32) or remove it
* MIN32/MAX32 used stupidly in celt/ where the arguments evaluated twice.
* why does --speech + 24kbit make opus tools code stereo where its mono without speech?
* make test_opus_encode test more stupidly low rates.
* random duration for the first/last frame in opus-tools
* short block high rate waste in mdct mode
* (mdct mode) silence detection
* Prod the second implementation (Tim's) into something (quasi-)releasable

OpusRelease

2012-05-17T17:22:50Z

Rillian: /* Release management / marketing work */ I assume 'sweaps' was a typo.

== Opus release planning ==

This page is for organizing todo items for the initial opus release. Some items would best be dribbled out after the initial release while others are more release critical (tools binaries, test vectors, examples)

=== Release management / marketing work ===

* Windows binaries
* Mac binaries
* Get binary packages of opus and opus-tools working for major distros

* Ogg Opus test vectors

* More 'brochure' level text about Opus and its advantages

* Logo

* More images/graphs/art for the site (PEAQ sweeps?)

* Overhaul of the comparative page

* demo pages
** static demo with music, voice; like [https://people.xiph.org/~giles/2012/opus this one]
** albums
** audiobooks
** remote control encoder (oneman)
** emscripten decoder
** A recording of a remote jamming event (bemasc)
** update Greg's CELT try-tool[http://people.xiph.org/~greg/trial_tool.png] for libopus
** Son-of-[http://people.xiph.org/~xiphmont/demo/celt/demo.html Monty-CELT-demo] (An in-the-weeds technology demo)
*** This may actually need to be a series of two, one covering LP mode and one covering hybrid and the rest of the system.

* Improved documentation (e.g. an API overview doc that isn't a doxygen dump), "idiots guide to using libopus"

* Bugtracker

* Client applications beyond Firefox, esp VLC and foobar2000. See [[OPUS TODO]] for a list.

=== Development work ===

* fix remaining fixed point overflows (gmaxwell/jmspeex)
* Make opus_custom_demo read/write little endian on BE hosts.
* Boring standard makefiles for many systems (VMS too? haha)
* Oggdropish GUI for opusenc
* Additional tools (validator, gain, udp streaming example)
* libao for opus-tools
* LP-mode CBR (boundary conditions, iterations, harmonic average)
* Fix -fstack-protector-all (e.g. mingw32) or remove it
* MIN32/MAX32 used stupidly in celt/ where the arguments evaluated twice.
* why does --speech + 24kbit make opus tools code stereo where its mono without speech?
* make test_opus_encode test more stupidly low rates.
* random duration for the first/last frame in opus-tools
* short block high rate waste in mdct mode
* (mdct mode) silence detection
* Prod the second implementation (Tim's) into something (quasi-)releasable

OpusRelease

2012-05-17T00:11:16Z

Rillian: /* Opus release planning */ test vectors

== Opus release planning ==

This page is for organizing todo items for the initial opus release.

* Windows binaries
* Mac binaries
* Get binary packages of opus and opus-tools working for major distros

* Ogg Opus test vectors

* demo pages
** static demo with music, voice; like [https://people.xiph.org/~giles/2012/opus this one]
** albums
** audiobooks
** remote control encoder (oneman)
** emscripten decoder

OpusRelease

2012-05-17T00:09:54Z

Rillian: /* Opus release planning */ would be nice to get a js build working for broader demonstrations

== Opus release planning ==

This page is for organizing todo items for the initial opus release.

* Windows binaries
* Mac binaries
* Get binary packages of opus and opus-tools working for major distros

* demo pages
** static demo with music, voice; like [https://people.xiph.org/~giles/2012/opus this one]
** albums
** audiobooks
** remote control encoder (oneman)
** emscripten decoder

OpusRelease

2012-05-17T00:09:19Z

Rillian: /* Opus release planning */ Link to my demo page

== Opus release planning ==

This page is for organizing todo items for the initial opus release.

* Windows binaries
* Mac binaries
* Get binary packages of opus and opus-tools working for major distros

* demo pages
** static demo with music, voice; like [https://people.xiph.org/~giles/2012/opus this one]
** albums
** audiobooks
** remote control encoder (oneman)

OpusRelease

2012-05-17T00:07:49Z

Rillian: /* Opus release planning */ fix formatting

== Opus release planning ==

This page is for organizing todo items for the initial opus release.

* Windows binaries
* Mac binaries
* Get binary packages of opus and opus-tools working for major distros

* demo pages
** static demo with music, voice
** albums
** audiobooks
** remote control encoder (oneman)