This is a draft document. A work in progress. A scratchpad for ideas. It should not be widely circulated in this form.

Context

We received an e-mail from a consultant studying the suitability of Theora for use in "eGovernment", on behalf of the IDABC, an EU governmental agency responsible for "Interoperability" with an emphasis on open source. The investigation is in the context of European Interoperability Framework, about which there has been some real controversy.

The method of assessment is the Common Assessment Method for Standards and Specifications, including the questions below.

CAMSS Questions

Part 4: Market Criteria

This group of Market criteria analyses the formal specification in the scope of its market environment, and more precisely it examines the implementations of the formal specification and the market players. This implies identifying to which extent the formal specification benefits from market support and wide adoption, what are its level of maturity and its capacity of reusability.

Market support is evaluated through an analysis of how many products implementing the formal specification exist, what their market share is and who their end-users are. The quality and the completeness (in case of partitioning) of the implementations of the formal specification can also be analysed. Availability of existing or planned mechanisms to assess conformity of implementations to the standard or to the specification could also be identified. The existence of at least one reference implementation (i.e.: mentioning a recognized certification process) - and of which one is an open source implementation - can also be relevant to the assessment. Wide adoption can also be assessed across domains (i.e.: public and private sectors), in an open environment, and/or in a similar field (i.e.: best practices).

A formal specification is mature if it has been in use and development for long enough that most of its initial problems have been overcome and its underlying technology is well understood and well defined. Maturity is also assessed by identifying if all aspects of the formal specification are considered as validated by usage, (i.e.: if the formal specification is partitioned), and if the reported issues have been solved and documented.

Reusability of a formal specification is enabled if it includes guidelines for its implementation in a given context. The identification of successful implementations of the standard or specification should focus on good practices in a similar field. Its incompatibility with related standards or specifications should also be taken into account.

The ideas behind the Market Criteria can also be expressed in the form of the following questions:

Market support

Does the standard have strong support in the marketplace?

Yes. For example, among web browsers, support for Xiph's Ogg, Theora, and Vorbis standards is now included by default in Mozilla Firefox, Google Chrome, and the latest versions of Opera, representing hundreds of millions of installed users just in this market alone. Further, a QuickTime component exists which enables use of Xiph's Ogg, Theora, and Vorbis standards in all Mac OS X applications that make use of the QuickTime framework - which includes Safari/Webkit, iMovie, QuickTime, and many others. On Windows, DirectShow filters exist which also enable all Windows applications that use the DirectShow framework to use Xiph's Ogg, Theora, and Vorbis standards.

What products exist for this formal specification ?

Theora is a video codec, and as such the required products are encoders, decoders, and transmission systems. All three types of products are widely available for Theora.

How many implementations of the formal specification are there?

Xiph does not require implementors to acquire any license before implementing the specification. Therefore, we do not have a definitive count of the number of implementations. In addition to the reference implementation, which has been ported to most modern platforms and highly optimized for x86 and ARM CPUs and TI C64x+ DSPs, we are aware of a number of independent, conformant or mostly-conformant implementations. These include two C decoders (FFmpeg and QTheora), a Java decoder (Jheora), a C# decoder, an FPGA decoder, and an FPGA encoder.

Are there products from different suppliers in the market that implement this formal specification ?

Yes. Corporations such as Atari, Canonical, DailyMotion, Elphel, Fluendo, Google, Mozilla, Novell, Opera, Red Hat, Sun Microsystems, Ubisoft, and countless others have supplied products with an implementation of the Theora standard.

Are there many products readily available from a variety of suppliers?

Yes. Theora has been deployed in embedded devices, security cameras, video games, video conferencing systems, web browsers, home theater systems, and many other products. A complete, legal, open-source reference implementation can also be downloaded free of charge, including components for all major media frameworks (DirectShow, gstreamer, and Quicktime), giving the plethora of applications which use these frameworks the ability to use the codec.

What is the market share of the products implementing the formal specification, versus other implementations of competing formal specifications ?

Theora playback is extremely widely available, covering virtually the entire market of personal computers. Theora is also increasingly available in mobile and embedded devices. Since we do not require licensing for products that implement the specification, we do not have market share numbers that can be compared with competing formal specifications. Because implementations are readily available and free, Theora is included in many products that support multiple codecs, and is sometimes the only video codec included in free software products.

Who are the end-users of these products implementing the formal specification?

The end users are television viewers, video gamers, web surfers, movie makers, business people, video distribution services, and anyone else who interacts with moving pictures.

Maturity

Are there any existing or planned mechanisms to assess conformity of the implementations of the formal specification?

Yes. In addition to a continuous peer review process, we maintain a suite of test vectors that allow implementors to assess decoder conformity. We also provide free online developer support and testing for those attempting to make a conforming implementation. An online validation service is available.

Is there a reference implementation (i.e.: mentioning a recognized certification process)?

Yes. Xiph maintains a reference implementation called libtheora. In addition to serving as a reference, libtheora is also highly optimized to achieve the maximum possible speed, accuracy, reliability, efficiency, and video quality. As a result, many implementors of Theora adopt the reference implementation.

Is there an open source implementation?

Yes. libtheora is made available under a completely permissive BSD-like license. Its open-source nature also contributes to its quality as a reference implementation, as implementors are welcome to contribute their improvements to the reference. There are also several other open source implementations.

Does the formal specification show wide adoption?
- across different domains? (I.e.: public and private)

Yes. In addition to the private companies mentioned in the previous section, Theora has also been specified as the sole format supported by non-profit institutions such as Wikipedia, currently the 6th largest website in the world, or as one of a small number of preferred formats supported by other public organizations, such as the Norwegian government.

- in an open environment?

Yes. On open/free operating systems such as those distributed by Novell/SuSE, Canonical, and Red Hat, Theora is the primary default video codec.

- in a similar field? (i.e.: can best practices be identified?)

Yes. Most prominently, Theora has been used for eGovernment video distribution in the United States at Metavid. Metavid is the most comprehensive, interactive archive of video footage from the United States legislature. Metavid does not only distribute video; they also enable citizens to engage by annotating videos and correcting transcripts. Metavid distributes its entire archive in Theora format. Metavid's source code is entirely open and reusable for any purpose, providing instant access to best practices for eGovernment with Theora. Metavid's video display component is also available separately as mv_embed, which provides reusable best practices for easy Theora display on the web.

Another important user of Theora is Wikipedia, which distributes video exclusively in Theora format. Wikipedia's best practices for Theora distribution are encapsulated in OggHandler, which can be freely reused by anyone using the open-source MediaWiki software.

Has the formal specification been in use and development long enough that most of its initial problems have been overcome?

Yes. Theora was derived from VP3, which was originally released in May 2000. The Theora specification was completed in 2004. Theora has now been used in a wide variety of applications, on the full spectrum of computing devices.

Is the underlying technology of the standard well-understood? (e.g., a reference model is well defined, appropriate concepts of the technology are in widespread use, the technology may have been in use for many years, a formal mathematical model is defined, etc.)

Yes. The underlying technology has been in use for nearly a decade, and most of the concepts have been in widespread use for even longer.

Is the formal specification based upon technology that has not been well-defined and may be relatively new?

No. The formal specification is based on technology from the On2 VP3 codec, which is substantially similar to simple block-transform codecs like H.261. This class of codecs is extremely well understood, and has been actively in use for over 20 years.

Has the formal specification been revised? (Yes/No, Nof)

The formal specification of the Theora decoder has been stable for years. However, the text of the specification is continuously revised, based on user feedback, to improve the clarity and accuracy of the description of the technology.

Is the formal specification under the auspices of an architectural board? (Yes/No)

No. Although officially maintained by the Xiph.Org Foundation, anyone is free to join this organization, and one need not even be a member to make contributions. However, the core developers will review contributions and make sure they do not contradict the general architecture and they work well with the existing code and the test cases.

Is the formal specification partitioned in its functionality? (Yes/No)

No. Theora is very deliberately not partitioned, to avoid the confusion created by a "standard" composed of many incompatible "profiles". The Theora standard does not have any optional components. A compliant Theora decoder can correctly process any Theora stream.

- To what extent does each partition participate to its overall functionality? (NN%)

N/A.

- To what extent is each partition implemented? (NN%) (cf market adoption)

N/A.

Re-usability

Does the formal specification provide guidelines for its implementation in a given organisation?

Yes. For example, the Theora specification provides "non-normative" advice and explanation for implementors of Theora decoders and encoders, including example algorithms for implementing required mathematical transforms. Xiph also maintains a documentation base for implementors who desire more guidelines beyond the specification itself.

Can other cases where similar systems implement the formal specification be considered as successful implementations and good practices?

Xiph's standards have successfully been implemented by many organisations in a wide variety of environments. We maintain (non-exhaustive) lists of products which implement Theora support, many of them open source, so that others may use them as a reference when preparing their own products.

Is its compatibility with related formal specification documented?

Yes. For example, the Theora specification also documents the use of Theora within the standard Ogg encapsulation format, and the TheoraRTP draft specification explains how to transmit Theora using the RTP standard. In addition, the specification documents Theora's compatibility with ITU-R B.470, ITU-R B.601, ITU-R B.709, SMPTE-170M, UTF-8, ISO 10646, and Ogg Vorbis.

Part 5: Standardisation Criteria

From Idabc-camss

Note: Throughout this section, “Organisation” refers to the standardisation/fora/consortia body in charge of the formal specification.

Significant characteristics of the way the organisation operates are for example the way it gives the possibility to stakeholders to influence the evolution of the formal specification, or which conditions it attaches to the use of the formal specification or its implementation. Moreover, it is important to know how the formal specification is defined, supported, and made available, as well as how interaction with stakeholders is managed by the organisation during these steps. Governance of interoperability testing with other formal specifications is also indicative.

The standardisation criteria analyses therefore the following elements:

Availability of Documentation

The availability of documentation criteria is linked to cost and online availability. Access to all preliminary results documentation can be online, online for members only, offline, offline, for members only or not available. Access can be free or for a fee (which fee?).

Every Xiph standard is permanently available online to everyone at no cost. For example, we invite everyone to download the most up-to-date copy of the Theora specification, and the latest revision of Vorbis. All previous revisions are available from Xiph's revision control system.

Intellectual Property Right

The Intellectual Property Rights evaluation criteria relates to the ability for implementers to use the formal specification in products without legal or financial implications. The IPR policy of the organisation is therefore evaluated according to:

the availability of the IPR or copyright policies of the organisation (available on-line or off-line, or not available);

The reference implementations of each codec include all necessary IPR and copyright licenses for that codec, including all documentation, and are freely available to everyone.

the organisation’s governance to disclose any IPR from any contributor (ex-ante, online, offline, for free for all, for a fee for all, for members only, not available);

Xiph does not require the identification of specific patents that may be required to implement a standard, however it does require an open-source compatible, royalty free license from a contributor for any such patents they may own before the corresponding technology can be included in a standard. These license are made available online, for free, to all parties.

the level of IPR set "mandatory" by the organisation (no patent, royalty free patent, patent and RAND with limited liability , patent and classic RAND, patent with explicit licensing, patent with defensive licensing, or none);

All standards, specifications, and software published by the Xiph.Org Foundation are required to have "open-source compatible" IPR. This means that a contribution must either be entirely clear of any known patents, or any patents that read upon the contribution must be available under a transferable, irrevocable public nonassertion agreement to all people everywhere. For example, see our On2 patent nonassertion warrant. Other common "royalty free" patent licenses are either not transferable, revocable under certain conditions (such as patent infringement litigation against the originating party), or otherwise impose restrictions that would prevent distribution under common OSI-approved licenses. These would not be acceptable.

the level of IPR "recommended" by the organisation (no patent, royalty free patent, patent and RAND with limited liability, patent and classic RAND, patent with explicit licensing, patent with defensive licensing, or none). [Note: RAND (Reasonable and Non Discriminatory License) is based on a "fairness" concept. Companies agree that if they receive any patents on technologies that become essential to the standard then they agree to allow other groups attempting to implement the standard to use these patents and they agree that the charges for the patents shall be reasonable. "RAND with limited availability" is a version of RAND where the "reasonable charges" have an upper limit.]

Xiph's recommended IPR requirements are the same as our mandatory requirements.

Accessibility

The accessibility evaluation criteria describe the importance of equal and safe accessibility by the users of implementations of formal specifications. This aspect can be related to safety (physical safety and conformance safety) and accessibility of physical impaired people (design for all).

Focus is made particularly on accessibility and conformance safety. Conformance testing is testing to determine whether a system meets some specified formal specification. The result can be results from a test suite. Conformance validation is when the conformance test uniquely qualifies a given implementation as conformant or not. Conformance certification is a process that provides a public and easily visible "stamp of approval" that an implementation of a standard validates as conformant.

The following questions allow an assessment of accessibility and conformance safety:

Does a mechanism that ensures disability support by a formal specification exist? (Y/N)

Yes. Xiph ensures support for users with disabilities by providing specifications for accessible technologies independent of the codec itself. Notably, the Xiph OggKate codec for time-aligned text and image content provides support for subtitles for internationalisation, captions for the hearing-impaired, and textual audio descriptions for the visually impaired. Further, Ogg supports multiple tracks of audio and video content in one container, such that sign language tracks and audio descriptions can be included into one file. For this to work, Xiph has defined Skeleton which holds metadata about each track encapsulated within a single Ogg file. When Theora is transmitted or stored in an Ogg container, it is automatically compatible with these accessibility measures.

Is conformance governance always part of a standard? (Y/N)

No. Xiph does not normally provide a formal conformance testing process as part of a standard.

Is a conformance test offered to implementers? (Y/N)

Yes. Xiph maintains a freely available suite of test vectors and an online validation service that can be used by anyone to check confirm basic conformance, in addition to tools such as the oggz-validate program included with liboggz, which has been widely used for conformance testing.

Is conformance validation available to implementers? (Y/N)

Yes. Informal conformance testing is available to implementors upon request, and Xiph has provided such testing for a number of implementations in the past.

Is conformance certification available? (Y/N)

Yes. Xiph does not require certification, but maintains the right to withhold the use of our trademarks from implementors that act in bad faith. Implementors may, however, request explicit permission to use our trademarks with a conforming implementation.

Is localisation of a formal specification possible? (Y/N)

Yes. We welcome anyone who wishes to translate Xiph specifications into other languages. We have no policy requiring that the normative specification be written in English.

Interoperability governance

The interoperability governance evaluation criteria relates to how interoperability is identified and maintained between interoperable formal specifications. In order to do this, the organisation may provide governance for:

open identification in formal specifications,

Yes. The Xiph codecs can be precisely identified by their MIME types, as formally defined by IETF RFC 5334, an open specification.

open negotiation in formal specifications,
open selection in formal specifications.

Meeting and consultation

The meeting and consultation evaluation criteria relates to the process of defining a formal specification. As formal specifications are usually defined by committees, and these committees normally consist of members of the organisation, this criteria studies how to become a member and which are the financial barriers for this, as well as how are non-members able to have an influence on the process of defining the formal specification. It analyses:

if the organisation is open to all types of companies and organisations and to individuals;

Yes. Xiph welcomes representatives from all companies and organizations.

if the standardisation process may specifically allow participation of members with limited abilities when relevant;

Yes. Standardization occurs almost entirely in internet communications channels, allowing participants with disabilities to engage fully in the standards development process. We also encourage nonexperts and students to assist us as they can, and to learn about Xiph technologies by participating in the standards development process.

if meetings are open to all members;

Xiph meetings are open to everyone. We charge no fee for and place no restrictions on attendance or participation. For example, anyone interested in contributing to the Theora specification may join the Theora development mailing list.

if all can participate in the formal specification creation process;

Yes. All people are welcome to participate in the specification creation process. No dues or fees are required to participate

if non-members can participate in the formal specification creation process.

Yes. Xiph does not maintain an explicit list of members, and no one is excluded from contributing to specifications as they are developed.

Consensus

Consensus is decision making primarily with regard to the approval of formal specifications and review with interest groups (non-members). The consensus evaluation criterion is evaluated with the following questions:

Does the organisation have a stated objective of reaching consensus when making decisions on standards?

There is no explicitly stated objective of reaching consensus. However, when new contributions are made, the key specification developers will be able to veto on the introduction of a new feature. Generally, differences are discussed openly and new features are adapted until they fit the overall architecture of the standards, at which stage they are introduced into the specification, standards and software.

If consensus is not reached, can the standard be approved? (answers are: cannot be approved but referred back to working group/committee, approved with 75% majority, approved with 66% majority, approved with 51% majority, can be decided by a "director" or similar in the organisation).

The standard can be approved without consensus via the decision of a "director" or similar.

Is there a formal process for external review of standard proposals by interest groups (nonmembers)?

Since anyone may participate in the development process and make proposals, there is no need for a separate formal process to include proposals by nonmembers.

Due Process

The due process evaluation criteria relates to the level of respect of each member of the organisation with regard to its rights. More specifically, it must be assured that if a member believes an error has been made in the process of defining a formal specification, it must be possible to appeal this to an independent, higher instance. The question is therefore: can a member formally appeal or raise objections to a procedure or to a technical specification to an independent, higher instance?

Yes. Even if a member fails an appeal within the organization, because all of the technology Xiph standardizes is open and freely implementable, they are always free to develop their own, competing version. Such competing versions may even still be eligible for standardization under the Xiph umbrella.

Changes to the formal specification

The suggested changes made to a formal specification need to be presented, evaluated and approved in the same way as the formal specification was first defined. This criteria therefore applies the above criteria to the changes made to the formal specification(availability of documentation, Intellectual Property Right, accessibility, interoperability governance, meeting and consultation, consensus, due process).

The exact same process is used for revisions to the standard as was used for the original development of the standard, and thus the answers to all of the above questions remain the same.

Support

It is critical that the organisation takes responsibility for the formal specification throughout its life span. This can be done in several ways such as for example a regular periodic review of the formal specification. The support criteria relates to the level of commitment the organisation has taken to support the formal specification throughout its life:

does the organisation provide support until removal of the published formal specification from public domain (Including this process?

Xiph.Org standards are never removed from the public domain. Xiph endeavors to provide support for as long as the standard remains in use.

does the organisation make the formal specification still available even when in non-maintenance mode?

Yes. All Xiph.Org standards are freely licensed and will always be available.

does the organisation add new features and keep the formal specification up-to-date?

Yes. Xiph maintains its ecosystem of standards on a continuous basis.

does the organisation rectify problems identified in initial implementations?

Yes. Xiph maintains a problem reporting system that is open to the public, and invites everyone to submit suggestions for improvements. Improvements are made both to the standards documents and to the reference implementations.

does the organisation only create the formal specification?

No. Xiph also produces high-quality reusable reference implementations of its standards, released under an open license.