XMLEmbedding: Difference between revisions

From XiphWiki
Jump to navigation Jump to search
m (hyperlink Silvia's comment.)
(brain dump my XML segmentation ideas.)
Line 1: Line 1:
{{draft}}
{{draft}}
Schemes such as [[MDMF]] require an embedding in physical Ogg streams. This page is for development of a specification for embedding metadata streams in Ogg. The final version will probably look like the CMML mapping.
Schemes such as [[MDMF]] require an embedding in physical Ogg streams. This page is for development of a specification for embedding XML streams in Ogg. The final version will probably look like the CMML mapping.


==Current plans==
==Current plans==
Line 6: Line 6:


===Magic number?===
===Magic number?===
Should we have a magic number for this? By convention the beginning of stream packet for codecs in Ogg identifies the packet, Strictly speaking we will have a magic number regardless, but should it be:
Should we have a magic number for this? By convention the beginning of stream packet for codecs in Ogg identifies the packet, Strictly speaking we will have a magic number regardless, but should it be:


* '<?xml' the opening of the XML declaration. In this instance the demuxer would pass this upwards to an XML parser which would derive from the rest of the bos packet what to do with it.
* '<?xml' the opening of the XML declaration. In this instance the demuxer would pass this upwards to an XML parser which would derive from the rest of the bos packet what to do with it.


* Some other sequence before the XML starts, identifying the particular stream type, probably with a version number which will imply the contents to be some form of XML. CMML does this. This may avoid the almost inevitable problem of some implementor assuming '<?xml' is always metadata, it may also reduce the difficulty of writing a demuxer. Demuxers could still peak ahead and try to look of an xml namespace it recognizes.
* Some other sequence before the XML starts, identifying the particular stream type, probably with a version number which will imply the contents to be some form of XML. CMML does this. This may avoid the almost inevitable problem of some implementor assuming '<?xml' is always metadata, it may also reduce the difficulty of writing a demuxer. Demuxers could still peek ahead and try to look of an xml namespace it recognizes.
 
===Division into packets===
 
The raw XML should ideally be broken into packets in a way that the loss of some packets, while destroying information, does not result in an invalid stream. Generally this means:
 
* The bos packet should consist of any initial processing directives, namespace declarations, and the root tag if there is one.
* Subsequent packets should be valid xml stanzas, similar to the [http://xmpp.org/ XMPP] definition, which concatenated are also valid xml.
* The eos packet can be empty. If there was a root tag in the bos packet, it should be closed here.
 
Parsers should close all open tags on encountering eos to handle truncated stream conditions. Encountering eos here means either a after processing a packet marked with the eos flag, having finished on an Ogg page with the eos flag set, or a virtual eos, implied by encountering bos flags for a new chain segment.


===Granulepos mapping===
===Granulepos mapping===
c.f. Silvia Pfeiffer on [http://lists.xiph.org/pipermail/ogg-dev/2007-September/000570.html ogg-dev]:
c.f. Silvia Pfeiffer on [http://lists.xiph.org/pipermail/ogg-dev/2007-September/000570.html ogg-dev]:
<blockquote>I suggest using the solution that CMML has come to use.
<blockquote>I suggest using the solution that CMML has come to use.
Line 25: Line 37:
Also, use the granulepos scheme that we defined for CMML pages- you're
Also, use the granulepos scheme that we defined for CMML pages- you're
going to make your lives easier.</blockquote>
going to make your lives easier.</blockquote>
That is, use a split granulepos scheme like keyframe codecs to indicate the offset to the previous packet.
Whether this is appropriate for a given XML steam will depend on its application. Metadata that applies
to the whole stream should just be included at the beginning, like OggSkeleton, and the granulepos can just
all be zero. Time-based data, like a slideshow, should be muxed throughout the stream.


==Test Files==
==Test Files==

Revision as of 10:29, 14 September 2007

Schemes such as MDMF require an embedding in physical Ogg streams. This page is for development of a specification for embedding XML streams in Ogg. The final version will probably look like the CMML mapping.

Current plans

(taken from mailing list discussions)

Magic number?

Should we have a magic number for this? By convention the beginning of stream packet for codecs in Ogg identifies the packet, Strictly speaking we will have a magic number regardless, but should it be:

  • '<?xml' the opening of the XML declaration. In this instance the demuxer would pass this upwards to an XML parser which would derive from the rest of the bos packet what to do with it.
  • Some other sequence before the XML starts, identifying the particular stream type, probably with a version number which will imply the contents to be some form of XML. CMML does this. This may avoid the almost inevitable problem of some implementor assuming '<?xml' is always metadata, it may also reduce the difficulty of writing a demuxer. Demuxers could still peek ahead and try to look of an xml namespace it recognizes.

Division into packets

The raw XML should ideally be broken into packets in a way that the loss of some packets, while destroying information, does not result in an invalid stream. Generally this means:

  • The bos packet should consist of any initial processing directives, namespace declarations, and the root tag if there is one.
  • Subsequent packets should be valid xml stanzas, similar to the XMPP definition, which concatenated are also valid xml.
  • The eos packet can be empty. If there was a root tag in the bos packet, it should be closed here.

Parsers should close all open tags on encountering eos to handle truncated stream conditions. Encountering eos here means either a after processing a packet marked with the eos flag, having finished on an Ogg page with the eos flag set, or a virtual eos, implied by encountering bos flags for a new chain segment.

Granulepos mapping

c.f. Silvia Pfeiffer on ogg-dev:

I suggest using the solution that CMML has come to use.

The XML file is essentially the same as an unencapsulated physical bitstream.

Then there is a mapping into a logical bitstream, where some of the default information - in particular the XML header - are split off and put into the bos packet - nothing really needs to go into the eos packet. There's also a magic number and a version number.

Also, use the granulepos scheme that we defined for CMML pages- you're

going to make your lives easier.

That is, use a split granulepos scheme like keyframe codecs to indicate the offset to the previous packet. Whether this is appropriate for a given XML steam will depend on its application. Metadata that applies to the whole stream should just be included at the beginning, like OggSkeleton, and the granulepos can just all be zero. Time-based data, like a slideshow, should be muxed throughout the stream.

Test Files

It is a barrier to the widespread introduction of any metadata format that the Vorbis I spec only requires players to support an unaccompanied Vorbis stream; many Ogg Vorbis players will refuse to play augmented streams, especially if the content is not recognised (although many recent players do succeed). As a prelude to development of an Ogg metadata format it will be necessary to encourage developers to introduce more flexible Ogg filters.

To help with testing the following files are available, based on a speculative (and very basic) metadata format. In each case the derivative files are under the same license as the original. Two sets are provided to allow chained stream testing. On some players the seek tests produce an annoying clicking—if you like the music get the originals. Please notice that filenames are mixed case and add a note in discussion if you find a broken link.

Original (Vorbis I): Bach - Nun freut euch lieben Christen performed by Debbie Hu, from Pandora Records and available under the EFF OAL.

Original (Vorbis I): On The Moon (Trip Hop mix) by Disharmonic, from ccMixter and available under the Creative Commons Attribution 2.5 license.