Volume 1, Issue 10 - November 2001
   
 

Building the VoiceXML Forum Certification Program

By Gerald M. Karam and Kenneth G. Rehor

Introduction

The VoiceXML Forum" is developing a certification program for VoiceXML products for the benefit of its members, vendors, and the buying public.  This program would allow vendors to submit their products (or services) for evaluation against a suite of tests that would demonstrate conformance to recommended specifications or standards needed to produce a conforming VoiceXML product.  Initially, the Certification Program will concentrate on VoiceXML platforms; testing programs for tools and applications may follow at a future time. A product passing the certification program would then be permitted to be labeled as conformant.  The test suite and certification process would be managed and developed by the Forum members and an independent, arm’s length certification authority.

The certification program operates to give vendors independent evaluation and protection of their intellectual property, and when desired, the opportunity to publicly or privately communicate their certification to customers and partners.  The VoiceXML Forum Certification Program will be open to all vendors who wish to have their products conformance tested. VoiceXML Forum Promoter and Sponsor members can participate in the test suite development and maintenance process, and the management of the certification process, as appropriate.

As of late October 2001, a request for proposals (RFP) has been provided to organizations that expressed interest in providing the services of the certification authority.  The VoiceXML Forum expects to be able to complete the RFP process and make a selection of acceptable candidates before the end of the December 2001.

What’s Included in the Certification Program?

The Certification Program includes conformance testing of all recommended specifications and standards that would be necessary to build a conforming VoiceXML language processor, under the testing architecture shown in Figure 1.  This diagram shows the product to be tested, the system under test or SUT, and the testing system that is comprised of the test driver (which may be just a telephone handset or some other automated test driver) and the reference web server that contains the suite of tests and records the test outcomes.  The test cases are built around the features in a specification that must be built in order to satisfy define a conforming implementation; furthermore, the specifications include the definitions of conformance. As such, conformance process begins with the specifications themselves.

Figure1: The Conformance Testing Architecture

A recommended specification is a document produced by an organization that, through some development process, creates recommended practice or behavior for some technology.  The World Wide Web Consortium (W3C http://www.w3.org) and the Internet Engineering Task Force (IETF http://www.ietf.org) are two such widely recognized bodies; the W3C typically produces recommended specifications in the areas of Internet content (such as HTML) and the IETF typically produces recommended specifications for Internet protocols (such as HTTP).

A standard is a specification produced through the internationally recognized standards bodies that typically organize around country representation rather than company representation; examples of these include the International Organization for Standardization (ISO http://www.iso.org) and in the United States, the American National Standards Institute (ANSI http://www.ansi.org).

For VoiceXML the primary body producing specifications is now the W3C.  On October 23, 2001 the VoiceXML Forum and the W3C announced a Memorandum of Understanding concerning VoiceXML activities, in which the VoiceXML Forum recognized the W3C as the sole body responsible for the development of the VoiceXML Language specification, and a coordination of the conformance efforts between the VoiceXML Forum and the W3C.  We’ll return to how this coordination will work a little later.

Building a VoiceXML Processor requires the implementation of features from a variety of specifications; however, the conformance testing of these specifications would be confined to their execution as part of a VoiceXML processor as there may be features that are outside of the VoiceXML context.  In other words, some specifications will be subject to full coverage testing, that is, conformance testing of all mandatory features; those where a VoiceXML processor needs only a few features will be subject to sufficient coverage testing.  Sufficient coverage testing does not do conformance testing of mandatory features, but tests to show a VoiceXML processor's adequate use of these specifications, not its complete use of them.

For full conformance coverage testing the following specifications are expected to be included:

  • W3C VoiceXML 2.0: Currently this is in the Working Draft (WD) step in the W3C recommendation process; see http://www.w3.org/TR/2001/WD-voicexml20-20011023/
  • W3C Speech Recognition Grammar Format (SRGF) 1.0: Currently this is in the Working Draft (WD) step; see http://www.w3.org/TR/speech-grammar/
  • W3C Speech Synthesis Markup Language (SSML) 1.0: Currently this is in the Working Draft (WD) step; see http://www.w3.org/TR/speech-synthesis
  • W3C Semantic Interpretation for Speech Recognition 1.0: Currently this is still under development for its first Working Draft (WD) thus it is not publicly disclosed as yet.  It will be used to provide semantic interpretation features to support the speech recognition grammar.

Other specifications such as the W3C Natural Language Semantics Markup Language (NLSML) (http://www.w3.org/TR/nl-spec/) may be included in the requirements and testing in the future.

For sufficient coverage testing the following specifications are expected to be included:

For the full coverage tests, the test requirements fall into three categories: required, optional, and platform-optional. Required features are denoted by "must" or “shall” in the specification, or otherwise are indicated as a mandatory feature of a conforming implementation. For a required feature, an implementation is required to support the syntax and semantics as described in the specification. Optional features are denoted by "should" in the specification. If an implementation wishes to be certified that it supports the optional feature, the conformance test would show that the feature is implemented according to the syntax and semantics as described in the specification. If an implementation does not support the optional feature, it must throw the appropriate "error.unsupported" code as described in the specification. Platform-Optional features are denoted by "may" in the specification. An implementation may or may not support a platform-optional feature. Examples include various optional or suggested audio formats, ASR formats, etc. These are not tested under conformance testing since their behavior may be platform specific.

Who’s who in the Certification Program.

The Certification Program involves several groups:  the VoiceXML Forum itself, an Independent Certification Authority (CA), Vendors, and possibly, Independent Testing Labs (TL). The Independent Certification Authority is an organization that, under contract to the VoiceXML Forum, will operate the certification program at arms-length, to ensure confidentiality of certification tests.  Vendors will have products that are candidates for certification. Vendors may include but are not limited to VoiceXML Forum members. Independent Testing Labs are companies that, perhaps recommended by the VoiceXML Forum, or perhaps on their own, would offer to provide independent testing of VoiceXML products against the conformance test suite. The key activities of these parties in the  certification program are summarized below.

VoiceXML Forum:

  • Define the policies and procedures of the certification program; the key task will be to create a process that is open, fair and gives confidence to both vendors and the buying public.
  • Manages the contract with the CA; this will require ongoing work both in monitoring the technical work of the CA, as well as the contractual obligations and any financial maters.
  • Co-develops and co-maintains the test suite, with the CA; this will be a task largely borne by the VoiceXML members who participate in the conformance activities.
  • Controls the problem reporting process, and provides resolutions of test suite deficiencies (with CA); this will be in part a function of administration, but again will involve VoiceXML members who participate in the conformance activities.
  • Work with the W3C to resolve ambiguities in applicable W3C specifications; this will be done through our W3C Liaison channel, as well as VoiceXML members who participate on the W3C and additional relations that may be established between the VoiceXML Forum and the W3C for this purpose.
  • If desired, recommends TL; the VoiceXML Forum may in the future choose to qualify, and then recommend companies that wish to provide VoiceXML testing services.  However, at this time, we are focused only on selecting the certification authority.
  • Controls and manages the certification brand.  The VoiceXML Forum and its administrative and legal staff will control the use of the VoiceXML Forum certification marks, to ensure that only those companies who have successfully met the obligations of the certification program can advertise their product using our designated marks.

CA:

  • Co-develops and co-maintains the test suite, with the VoiceXML Forum; the CA would bring its technical expertise to the table to create the main elements of the test suite, and all the application programs needed for the reference web server.
  • Distributes the Test Suite to Vendors and other authorized entities; the CA would act as the channel and collect all appropriate revenues.
  • Manages the certification application, testing, and notification process; as described in the next section, the Vendor’s initial relationship during certification is with the CA, not with the VoiceXML Forum, so that the Vendor can maintain its confidentiality.  Consequently, the CA manages much of the execution of the certification process on a day-to-day basis without the involvement of the VoiceXML Forum.
  • Operates the reference web server; this web server would be used for all conformance testing when a product is going through formal certification; thus all vendors would be subject to the same reference test suite. Since the CA is the only body allowed to collect the conformance test results, it must have responsibility for this critical component of the testing activity.
  • Operates the problem reporting process, and provides resolutions of test suite deficiencies (with VoiceXML Forum); again as the main interface for vendors, problem reporting must come through the CA, and the identities of Vendors protected.  However, part of this process also belongs to the VoiceXML Forum as the technical authority for the test suite.  Part of the problem reporting process will also be public so that vendors can understand the status of unresolved issues.

Vendors:

  • Test their platforms against the test suite managed by the CA, or have their platforms tested by the TL against the CA test suite.

TL:

  • Provide independent testing services for Vendors; a vendor may wish to have the added marketing value of having their formal conformance testing done by an independent third party, and therefore could make a marketing claim about how their testing was done, as separate from the certification activity itself.
  • Provide other services outside of the certification program, such as performance testing.

Next we’ll see how this all fits together in the daily operation of the certification program.

Continued...

back to the top

 

Copyright © 2001 VoiceXML Forum. All rights reserved.
The VoiceXML Forum is a program of the
IEEE Industry Standards and Technology Organization (IEEE-ISTO).