IDABC Questionnaire 2009

Context

We received an e-mail from a consultant studying the suitability of Theora for use in "eGovernment", on behalf of the IDABC, an EU governmental agency responsible for "Interoperability" with an emphasis on open source. The investigation is in the context of European Interoperability Framework, about which there has been some real controversy.

The method of assessment is the Common Assessment Method for Standards and Specifications, including the questions below.

CAMSS Questions

Part 4: Market Criteria

This group of Market criteria analyses the formal specification in the scope of its market environment, and more precisely it examines the implementations of the formal specification and the market players. This implies identifying to which extent the formal specification benefits from market support and wide adoption, what are its level of maturity and its capacity of reusability.

Market support is evaluated through an analysis of how many products implementing the formal specification exist, what their market share is and who their end-users are. The quality and the completeness (in case of partitioning) of the implementations of the formal specification can also be analysed. Availability of existing or planned mechanisms to assess conformity of implementations to the standard or to the specification could also be identified. The existence of at least one reference implementation (i.e.: mentioning a recognized certification process) - and of which one is an open source implementation - can also be relevant to the assessment. Wide adoption can also be assessed across domains (i.e.: public and private sectors), in an open environment, and/or in a similar field (i.e.: best practices).

A formal specification is mature if it has been in use and development for long enough that most of its initial problems have been overcome and its underlying technology is well understood and well defined. Maturity is also assessed by identifying if all aspects of the formal specification are considered as validated by usage, (i.e.: if the formal specification is partitioned), and if the reported issues have been solved and documented.

Reusability of a formal specification is enabled if it includes guidelines for its implementation in a given context. The identification of successful implementations of the standard or specification should focus on good practices in a similar field. Its incompatibility with related standards or specifications should also be taken into account.

The ideas behind the Market Criteria can also be expressed in the form of the following questions:

Market support

• Does the standard have strong support in the marketplace?

Yes. For example, among web browsers, support for Xiph's Ogg, Theora, and Vorbis standards is now included by default in Mozilla Firefox, Google Chrome, and the latest versions of Opera, representing many millions of installed users just in this market alone.

• What products exist for this formal specification ?

Theora is a video codec, and as such the required products are encoders, decoders, and transmission systems. All three types of products are widely available for Theora.

• How many implementations of the formal specification are there?

Xiph.org does not require implementors to acquire any license before implementing the specification. Therefore, we do not know how many implementations have been created. From voluntary notification given to us by implementors, we know that Theora has at least been implemented in embedded devices, security cameras, video games, video conferencing systems, web browsers, home theater systems, and many other applications.

• Are there products from different suppliers in the market that implement this formal specification ?

Yes. Corporations such as Novell, Opera, Google, Mozilla, Red Hat, Sun Microsystems, Canonical, DailyMotion, Elphel, and countless others have supplied implementations of the Theora standard.

• Are there many products readily available from a variety of suppliers?

Yes. Theora is extremely readily available: any user with a modern home computer can have a complete, legal, open source implementation, free of charge, downloaded and running in less than 30 seconds.

• What is the market share of the products implementing the formal specification, versus other implementations of competing formal specifications ?

Theora playback is extremely widely available, covering virtually the entire market of personal computers. Theora is also increasingly available in mobile and embedded devices.

• Who are the end-users of these products implementing the formal specification?

The end users are television viewers, movie makers, business people, video distribution services, and anyone else who interacts with moving pictures.

Maturity

• Are there any existing or planned mechanisms to assess conformity of the implementations of the formal specification?

Yes. In addition to a continuous peer review process, we maintain a suite of test vectors that allow implementors to assess decoder conformity.

• Is there a reference implementation (i.e.: mentioning a recognized certification process)?

Yes. Xiph.org maintains a constantly updated reference implementation called libtheora. libtheora is maintained not only as a reference, but also to achieve the maximum possible speed, accuracy, reliability, efficiency, and video quality. As a result, many implementors of Theora adopt the reference implementation.

• Is there an open source implementation?

Yes. libtheora is made available under a completely permissive BSD-like license. Its open-source nature also contributes to its quality as a reference implementation, as implementors are welcome to contribute their improvements to the reference. There are also several other open source implementations.

• Does the formal specification show wide adoption?
  • across different domains? (I.e.: public and private)
  • in an open environment?

Yes. On open/free operating systems such as those distributed by Novell/SuSE, Canonical, and Red Hat, Theora is the primary default video codec.

• • in a similar field? (i.e.: can best practices be identified?)
• Has the formal specification been in use and development long enough that most of its initial problems have been overcome?

Yes. Theora has now been used in a wide variety of applications, on the full spectrum of computing devices.

• Is the underlying technology of the standard well-understood? (e.g., a reference model is welldefined, appropriate concepts of the technology are in widespread use, the technology may have been in use for many years, a formal mathematical model is defined, etc.)
• Is the formal specification based upon technology that has not been well-defined and may be relatively new?

No. The formal specification is based on technology from the On2 VP3 codec, which is substantially similar to simple block-transform codecs like H.261. This class of codecs is extremely well understood, and has been actively in use for over 20 years.

• Has the formal specification been revised? (Yes/No, Nof)

Yes. The specification is continuously revised to improve clarity and accuracy.

• Is the formal specification under the auspices of an architectural board? (Yes/No)
• Is the formal specification partitioned in its functionality? (Yes/No)

No. Theora is very deliberately not partitioned, to avoid the confusion created by a "standard" composed of many incompatible "profiles". The Theora standard does not have any optional components. A compliant Theora decoder can correctly process any Theora stream.

• • To what extent does each partition participate to its overall functionality? (NN%)
  • To what extent is each partition implemented? (NN%) (cf market adoption)

Re-usability

• Does the formal specification provide guidelines for its implementation in a given organisation?

Yes. For example, the Theora specification document provides "non-normative" advice and explanation for implementors of Theora decoders and encoders, including example algorithms for implementing required mathematical transforms. Xiph also maintains a documentation base for implementors who desire more guidelines beyond the specification itself.

• Can other cases where similar systems implement the formal specification be considered as successful implementations and good practices?

Xiph.org standards have successfully been implemented by many organisations in a wide variety of environments. We encourage independent implementations and unexpected applications of our standards.

• Is its compatibility with related formal specification documented?

Yes. For example, the Theora specification also documents the use of Theora within the standard Ogg encapsulation format, and the TheoraRTP draft specification explains how to transmit Theora using the RTP standard.

Part 5: Standardisation Criteria

From Idabc-camss

Note: Throughout this section, “Organisation” refers to the standardisation/fora/consortia body in charge of the formal specification.

Significant characteristics of the way the organisation operates are for example the way it gives the possibility to stakeholders to influence the evolution of the formal specification, or which conditions it attaches to the use of the formal specification or its implementation. Moreover, it is important to know how the formal specification is defined, supported, and made available, as well as how interaction with stakeholders is managed by the organisation during these steps. Governance of interoperability testing with other formal specifications is also indicative.

The standardisation criteria analyses therefore the following elements:

Availability of Documentation

The availability of documentation criteria is linked to cost and online availability. Access to all preliminary results documentation can be online, online for members only, offline, offline, for members only or not available. Access can be free or for a fee (which fee?).

Every Xiph.org standard is permanently available online to everyone at no cost. For example, we invite everyone to download the most up-to-date copy of the Theora specification, and the latest revision of Vorbis.

Intellectual Property Right

The Intellectual Property Rights evaluation criteria relates to the ability for implementers to use the formal specification in products without legal or financial implications. The IPR policy of the organisation is therefore evaluated according to:

• the availability of the IPR or copyright policies of the organisation (available on-line or off-line, or not available);
• the organisation’s governance to disclose any IPR from any contributor (ex-ante, online, offline, for free for all, for a fee for all, for members only, not available);
• the level of IPR set "mandatory" by the organisation (no patent, royalty free patent, patent and RAND with limited liability , patent and classic RAND, patent with explicit licensing, patent with defensive licensing, or none);

Xiph.org standards have a mandatory requirement that our standards be absolutely free of IPR encumbrance. This means that a standard must either be entirely clear of any known patents, or any patents that read upon the standard must be available under an irrevocable public nonassertion agreement to all people everywhere. For example, see our On2 patent nonassertion warrant.

• the level of IPR "recommended" by the organisation (no patent, royalty free patent, patent and RAND with limited liability, patent and classic RAND, patent with explicit licensing, patent with defensive licensing, or none). [Note: RAND (Reasonable and Non Discriminatory License) is based on a "fairness" concept. Companies agree that if they receive any patents on technologies that become essential to the standard then they agree to allow other groups attempting to implement the standard to use these patents and they agree that the charges for the patents shall be reasonable. "RAND with limited availability" is a version of RAND where the "reasonable charges" have an upper limit.]

Xiph's mandatory IPR requirements are the most stringent possible, and so our IPR recommendation is necessarily the same as the mandatory requirements.

Accessibility

The accessibility evaluation criteria describe the importance of equal and safe accessibility by the users of implementations of formal specifications. This aspect can be related to safety (physical safety and conformance safety) and accessibility of physical impaired people (design for all).

Focus is made particularly on accessibility and conformance safety. Conformance testing is testing to determine whether a system meets some specified formal specification. The result can be results from a test suite. Conformance validation is when the conformance test uniquely qualifies a given implementation as conformant or not. Conformance certification is a process that provides a public and easily visible "stamp of approval" that an implementation of a standard validates as conformant.

The following questions allow an assessment of accessibility and conformance safety:

• Does a mechanism that ensures disability support by a formal specification exist? (Y/N)

Yes. Theora is compatible with specifications like OggKate, which provides subtitles for the hearing-impaired, as well as Skeleton, which provides metadata that can be used to specify scene description audio tracks for the visually impaired.

• Is conformance governance always part of a standard? (Y/N)
• Is a conformance test offered to implementers? (Y/N)

Yes. We maintain a library of test vectors so that implementors can ensure that their systems are conforming.

• Is conformance validation available to implementers? (Y/N)
• Is conformance certification available? (Y/N)
• Is localisation of a formal specification possible? (Y/N)

Interoperability governance

The interoperability governance evaluation criteria relates to how interoperability is identified and maintained between interoperable formal specifications. In order to do this, the organisation may provide governance for:

• open identification in formal specifications,
• open negotiation in formal specifications,
• open selection in formal specifications.

Meeting and consultation

The meeting and consultation evaluation criteria relates to the process of defining a formal specification. As formal specifications are usually defined by committees, and these committees normally consist of members of the organisation, this criteria studies how to become a member and which are the financial barriers for this, as well as how are non-members able to have an influence on the process of defining the formal specification. It analyses:

• if the organisation is open to all types of companies and organisations and to individuals;
• if the standardisation process may specifically allow participation of members with limited abilities
• when relevant;
• if meetings are open to all members;
• if all can participate in the formal specification creation process;
• if non-members can participate in the formal specification creation process.

Consensus

Consensus is decision making primarily with regard to the approval of formal specifications and review with interest groups (non-members). The consensus evaluation criterion is evaluated with the following questions:

• Does the organisation have a stated objective of reaching consensus when making decisions on standards?
• If consensus is not reached, can the standard be approved? (answers are: cannot be approved but referred back to working group/committee, approved with 75% majority, approved with 66% majority, approved with 51% majority, can be decided by a "director" or similar in the organisation).
• Is there a formal process for external review of standard proposals by interest groups (nonmembers)?

Due Process

The due process evaluation criteria relates to the level of respect of each member of the organisation with regard to its rights. More specifically, it must be assured that if a member believes an error has been made in the process of defining a formal specification, it must be possible to appeal this to an independent, higher instance. The question is therefore: can a member formally appeal or raise objections to a procedure or to a technical specification to an independent, higher instance?

Changes to the formal specification

The suggested changes made to a formal specification need to be presented, evaluated and approved in the same way as the formal specification was first defined. This criteria therefore applies the above criteria to the changes made to the formal specification(availability of documentation, Intellectual Property Right, accessibility, interoperability governance, meeting and consultation, consensus, due process).

Support

It is critical that the organisation takes responsibility for the formal specification throughout its life span. This can be done in several ways such as for example a regular periodic review of the formal specification. The support criteria relates to the level of commitment the organisation has taken to support the formal specification throughout its life:

• does the organisation provide support until removal of the published formal specification from public domain (Including this process?

Xiph.org standards are never removed from the public domain. Xiph endeavors to provide support for as long as the standard remains in use.

• does the organisation make the formal specification still available even when in non-maintenance mode?

Yes. All Xiph.org standards are freely licensed and will always be available.

• does the organisation add new features and keep the formal specification up-to-date?

Yes. Xiph.org maintains its ecosystem of standards on a continuous basis.

• does the organisation rectify problems identified in initial implementations?

Yes. Xiph.org maintains a problem reporting system that is open to the public, and invites everyone to submit suggestions for improvements. Improvements are made both to the standards documents and to the reference implementations.

• does the organisation only create the formal specification?

No. Xiph.org also produces high-quality reusable reference implementations of its standards, released under an open license.