IDABC Questionnaire 2009

Context

We received an e-mail from a consultant studying the suitability of Theora for use in "eGovernment", on behalf of the IDABC, an EU governmental agency responsible for "Interoperability" with an emphasis on open source. The investigation is in the context of European Interoperability Framework, about which there has been some real controversy.

The method of assessment is the Common Assessment Method for Standards and Specifications, including the questions below.

CAMSS Questions

Part 4: Market Criteria

This group of Market criteria analyses the formal specification in the scope of its market environment, and more precisely it examines the implementations of the formal specification and the market players. This implies identifying to which extent the formal specification benefits from market support and wide adoption, what are its level of maturity and its capacity of reusability.

Market support is evaluated through an analysis of how many products implementing the formal specification exist, what their market share is and who their end-users are. The quality and the completeness (in case of partitioning) of the implementations of the formal specification can also be analysed. Availability of existing or planned mechanisms to assess conformity of implementations to the standard or to the specification could also be identified. The existence of at least one reference implementation (i.e.: mentioning a recognized certification process) - and of which one is an open source implementation - can also be relevant to the assessment. Wide adoption can also be assessed across domains (i.e.: public and private sectors), in an open environment, and/or in a similar field (i.e.: best practices).

A formal specification is mature if it has been in use and development for long enough that most of its initial problems have been overcome and its underlying technology is well understood and well defined. Maturity is also assessed by identifying if all aspects of the formal specification are considered as validated by usage, (i.e.: if the formal specification is partitioned), and if the reported issues have been solved and documented.

Reusability of a formal specification is enabled if it includes guidelines for its implementation in a given context. The identification of successful implementations of the standard or specification should focus on good practices in a similar field. Its incompatibility with related standards or specifications should also be taken into account.

The ideas behind the Market Criteria can also be expressed in the form of the following questions:

Market support

• Does the standard have strong support in the marketplace?
• What products exist for this formal specification ?
• How many implementations of the formal specification are there?
• Are there products from different suppliers in the market that implement this formal specification ?
• Are there many products readily available from a variety of suppliers?
• What is the market share of the products implementing the formal specification, versus other implementations of competing formal specifications ?
• Who are the end-users of these products implementing the formal specification?

Maturity

• Are there any existing or planned mechanisms to assess conformity of the implementations of the formal specification?
• Is there a reference implementation (i.e.: mentioning a recognized certification process)?
• Is there an open source implementation?
• Does the formal specification show wide adoption?
  • across different domains? (I.e.: public and private)
  • in an open environment?
  • in a similar field? (i.e.: can best practices be identified?)
• Has the formal specification been in use and development long enough that most of its initial problems have been overcome?
• Is the underlying technology of the standard well-understood? (e.g., a reference model is welldefined, appropriate concepts of the technology are in widespread use, the technology may have been in use for many years, a formal mathematical model is defined, etc.)
• Is the formal specification based upon technology that has not been well-defined and may be relatively new?
• Has the formal specification been revised? (Yes/No, Nof)
• Is the formal specification under the auspices of an architectural board? (Yes/No)
• Is the formal specification partitioned in its functionality? (Yes/No)
  • To what extent does each partition participate to its overall functionality? (NN%)
  • To what extent is each partition implemented? (NN%) (cf market adoption)

Re-usability

• Does the formal specification provide guidelines for its implementation in a given organisation?
• Can other cases where similar systems implement the formal specification be considered as successful implementations and good practices?
• Is its compatibility with related formal specification documented?

Part 5: Standardisation Criteria

From Idabc-camss

Note: Throughout this section, “Organisation” refers to the standardisation/fora/consortia body in charge of the formal specification.

Significant characteristics of the way the organisation operates are for example the way it gives the possibility to stakeholders to influence the evolution of the formal specification, or which conditions it attaches to the use of the formal specification or its implementation. Moreover, it is important to know how the formal specification is defined, supported, and made available, as well as how interaction with stakeholders is managed by the organisation during these steps. Governance of interoperability testing with other formal specifications is also indicative.

The standardisation criteria analyses therefore the following elements:

Availability of Documentation

The availability of documentation criteria is linked to cost and online availability. Access to all preliminary results documentation can be online, online for members only, offline, offline, for members only or not available. Access can be free or for a fee (which fee?).

Every Xiph.org standard is permanently available online to everyone at no cost. For example, we invite everyone to download the most up-to-date copy of the specification Theora, and the most recent version specification for Vorbis.

Intellectual Property Right

The Intellectual Property Rights evaluation criteria relates to the ability for implementers to use the formal specification in products without legal or financial implications. The IPR policy of the organisation is therefore evaluated according to:

• the availability of the IPR or copyright policies of the organisation (available on-line or off-line, or not available);
• the organisation’s governance to disclose any IPR from any contributor (ex-ante, online, offline, for free for all, for a fee for all, for members only, not available);
• the level of IPR set "mandatory" by the organisation (no patent, royalty free patent, patent and RAND with limited liability , patent and classic RAND, patent with explicit licensing, patent with defensive licensing, or none);

Xiph.org standards have a mandatory requirement that our standards be absolutely free of IPR encumbrance. This means that a standard must either be entirely clear of any known patents, or any patents that read upon the standard must be available under an irrevocable public nonassertion agreement to all people everywhere. For example, see our On2 patent nonassertion warrant.

• the level of IPR "recommended" by the organisation (no patent, royalty free patent, patent and RAND with limited liability, patent and classic RAND, patent with explicit licensing, patent with defensive licensing, or none). [Note: RAND (Reasonable and Non Discriminatory License) is based on a "fairness" concept. Companies agree that if they receive any patents on technologies that become essential to the standard then they agree to allow other groups attempting to implement the standard to use these patents and they agree that the charges for the patents shall be reasonable. "RAND with limited availability" is a version of RAND where the "reasonable charges" have an upper limit.]

Xiph's mandatory IPR requirements are the most stringent possible, and so our IPR recommendation is necessarily the same as the mandatory requirements.

Accessibility

The accessibility evaluation criteria describe the importance of equal and safe accessibility by the users of implementations of formal specifications. This aspect can be related to safety (physical safety and conformance safety) and accessibility of physical impaired people (design for all).

Focus is made particularly on accessibility and conformance safety. Conformance testing is testing to determine whether a system meets some specified formal specification. The result can be results from a test suite. Conformance validation is when the conformance test uniquely qualifies a given implementation as conformant or not. Conformance certification is a process that provides a public and easily visible "stamp of approval" that an implementation of a standard validates as conformant.

The following questions allow an assessment of accessibility and conformance safety:

• Does a mechanism that ensures disability support by a formal specification exist? (Y/N)
• Is conformance governance always part of a standard? (Y/N)
• Is a conformance test offered to implementers? (Y/N)

Yes. We maintain a library of test vectors so that implementors can ensure that their systems are conforming.

• Is conformance validation available to implementers? (Y/N)
• Is conformance certification available? (Y/N)
• Is localisation of a formal specification possible? (Y/N)

Interoperability governance

The interoperability governance evaluation criteria relates to how interoperability is identified and maintained between interoperable formal specifications. In order to do this, the organisation may provide governance for:

• open identification in formal specifications,
• open negotiation in formal specifications,
• open selection in formal specifications.

Meeting and consultation

The meeting and consultation evaluation criteria relates to the process of defining a formal specification. As formal specifications are usually defined by committees, and these committees normally consist of members of the organisation, this criteria studies how to become a member and which are the financial barriers for this, as well as how are non-members able to have an influence on the process of defining the formal specification. It analyses:

• if the organisation is open to all types of companies and organisations and to individuals;
• if the standardisation process may specifically allow participation of members with limited abilities
• when relevant;
• if meetings are open to all members;
• if all can participate in the formal specification creation process;
• if non-members can participate in the formal specification creation process.

Consensus

Consensus is decision making primarily with regard to the approval of formal specifications and review with interest groups (non-members). The consensus evaluation criterion is evaluated with the following questions:

• Does the organisation have a stated objective of reaching consensus when making decisions on standards?
• If consensus is not reached, can the standard be approved? (answers are: cannot be approved but referred back to working group/committee, approved with 75% majority, approved with 66% majority, approved with 51% majority, can be decided by a "director" or similar in the organisation).
• Is there a formal process for external review of standard proposals by interest groups (nonmembers)?

Due Process

The due process evaluation criteria relates to the level of respect of each member of the organisation with regard to its rights. More specifically, it must be assured that if a member believes an error has been made in the process of defining a formal specification, it must be possible to appeal this to an independent, higher instance. The question is therefore: can a member formally appeal or raise objections to a procedure or to a technical specification to an independent, higher instance?

Changes to the formal specification

The suggested changes made to a formal specification need to be presented, evaluated and approved in the same way as the formal specification was first defined. This criteria therefore applies the above criteria to the changes made to the formal specification(availability of documentation, Intellectual Property Right, accessibility, interoperability governance, meeting and consultation, consensus, due process).

Support

It is critical that the organisation takes responsibility for the formal specification throughout its life span. This can be done in several ways such as for example a regular periodic review of the formal specification. The support criteria relates to the level of commitment the organisation has taken to support the formal specification throughout its life:

• does the organisation provide support until removal of the published formal specification from public domain (Including this process?

Xiph.org standards are never removed from the public domain. Xiph endeavors to provide support for as long as the standard remains in use.

• does the organisation make the formal specification still available even when in non-maintenance mode?

Yes. All Xiph.org standards are freely licensed and will always be available.

• does the organisation add new features and keep the formal specification up-to-date?

Yes. Xiph.org maintains its ecosystem of standards on a continuous basis.

• does the organisation rectify problems identified in initial implementations?

Yes. Xiph.org maintains a problem reporting system that is open to the public, and invites everyone to submit suggestions for improvements. Improvements are made both to the standards documents and to the reference implementations.

• does the organisation only create the formal specification?

No. Xiph.org also produces high-quality reusable reference implementations of its standards, released under an open license.