VoiceXML Review - Columns - Speak & Listen

Volume 1, Issue 3 - Mar. 2001

Answers to Your Questions About VoiceXML

By Jeff Kunins

In this monthly column, an industry expert will answer common questions about VoiceXML and related technologies. Readers are encouraged to submit questions about VoiceXML, including development, voice-user interface design, and speech technology in general, or how VoiceXML is being used commercially in the marketplace. If you have a question about VoiceXML, e-mail it to and be sure to read future issues of VoiceXML Review for the answer.

This month, we're going to tackle the first few reader-submitted questions. These are a great start, and we look forward to hearing from the readership about what questions most need answers in this forum..

Q: To evaluate the performance of an VoiceXML interpreter, is there any benchmark specified by the W3C or the VoiceXML Forum?

A: Currently, there are no officially sanctioned metrics or benchmarks for evaluating VoiceXML interpreter performance. However, it is definitely the case that large companies who are actively building VoiceXML-based solutions are considering performance when evaluating potential vendors.

The distributed nature of VoiceXML, which allows application logic to be cleanly separated from the underlying speech and telephony infrastructure, is one of its greatest strengths. It revolutionizes the IVR market by making outsourced IVR practical and cost-effective. However, this same characteristic presents significant technology hurdles that VoiceXML platform vendors must solve in order to guarantee reliable performance and scalability. .

As food for thought, I will present a few key metrics that I believe make sense to consider when evaluating VoiceXML platforms--either for on-premises or outsourced deployments. These are not in any way meant to be exhaustive; rather they are indicative of the kinds of tests customers should be thinking about.

It's important to note that many of these critical performance metrics are not entirely specific to the VoiceXML interpreter, but also involve the relevant subsystems required for actually "rendering" a VoiceXML application, such as speech recognition, audio processing, etc.

Performance Metrics for all VoiceXML Platforms (a subset)

All metrics should be considered under normal (~50%) and stressed (~80+%) capacity levels.

The number of simultaneous inbound/outbound ports per unit of rack space (port density).
The time to begin playing a prompt of several standardized sizes, both first time and subsequent times in a session (ms).
Recognition time for standardized static grammars of various sizes/complexities (ms).
Recognition time for standardized dynamic grammars of various sizes/complexities (ms).
Recognition time for utterances matching application-scope grammars of standardized size (ms).
Inter-state execution time for forms/event handlers within the same document (ms).
Inter-state execution time for forms/event handlers in new documents, discounting Web server response time (ms).
Execution time for standardized blocks of JavaScript (ms).
Processing time for standardized blocks of <record>ed audio (ms).
Performance variations for calls in progress when a single given call enters a JavaScript loop.

Additional Performance Metrics for Outsourced VoiceXML Platforms (subset)

Percentage packet loss metrics from customer Web servers to vendor network.
Latency from customer Web servers to vendor network (ms).
Consistently available IP bandwidth from customer Web servers to vendor network (Mbit/sec).
Number of simultaneous audio streams without audio breakup.
Bandwidth consumed for standardized use of standardized applications of various complexities (measuring cache performance).
Latency in reflecting changes to application content (measuring cache performance).
Number of simultaneous <record> posts without performance degradation of other calls.

Continued...

back to the top