Results of the VoiceXML Forum Survey

By Jonathan Engelsma and Joni Brennan,

In August 2002 the VoiceXML Forum's Education Committee conducted an on-line survey on VoiceXML. Our objective was to try to gauge the interest level within the VoiceXML community on the various ongoing and proposed activities within the Forum so that we can better calibrate our efforts with the interests of the Forum's membership and beyond.

While this exercise could hardly be characterized as a proper scientific survey, we do believe it gives a reasonably accurate picture of the level of interest within the community for the projects we currently have underway, or hope to begin in the near future. In what follows we will present an overview of the results and draw our conclusions from it.

On behalf of the VoiceXML Forum, we wish to thank all of you who participated and trust you will find the results as interesting as we did. While the survey was anonymous, a number of you included your optional email address for future follow up. Those of you who did not have the opportunity to participate, but are interested in getting involved in the Forum's activities, feel free to send us email and tell us your interests to .We would be happy to point you to the appropriate people within the Forum.

The Survey

The survey itself consisted of three sections: tools, conformance, and developer certification. Conformance is an ongoing activity while the developer certification activity has just begun recently. The Forum has very recently started to take a closer look at the area of VoiceXML-related tools and how it can play a role in helping proliferate open and standard tooling for VoiceXML developers, and perhaps providing a directory resource that helps people find the VoiceXML tools that suit their particular needs.

The actual survey questions have been appended to this report as an appendix.

The W3C has now moved from encouraging the use of XML Schema to the stronger position of explicitly discouraging the use of DTDs. While the creation of a schema when you already have a DTD is fairly straightforward, the fact that SSML is expected to be embedded in other markup languages (of which VoiceXML is the first example) brought additional requirements to the table:

Demographics

Respondents were able to indicate the primary role of their company within the VoiceXML industry as well as their own individual role within their particular organization. While we had a fairly even spread of responses from various types of companies as shown in the pie chart in Figure 1, more than half of the respondents indicated they were developers, as shown in Figure 2. This is hardly surprising, given the fact that the three areas the survey focused on (tools, conformance, and developer certification) are usually more interesting to developers than they are to the people they work for or the people who sell the things they build.

We posted invitations for the survey in a number of methods, including a link to the online survey, announcements to various VoiceXML mailing lists, all of which are owned by people who at one point or another expressed interest in VoiceXML, as well as posted announcement on 15+ online VoiceXML-related message boards and/or newsgroups maintained by various companies. We of course have no idea as to how many people received the invitation in the external message boards. We do know that the respondents include individuals from companies who are not formally members of the VoiceXML Forum.

Overall the number of responses was in line with our expectations. We had 89 complete responses, many of them containing very detailed and thoughtful responses to the more subjective questions. In what follows we will provide an overview of the results in each of the three areas.

Part I - VoiceXML-Related Tools

While the first question in this area was subjective, our inclusion of a list of different types of tools no doubt impacted how people answered. Nevertheless, we received some very detailed responses, and some definite trends can be identified. Of the 89 responses, 11 simply indicated that "all" of the tools listed were important to them. For those that provided more detailed answers, we have attempted to group responses into general categories of tools:

• Grammar Tools: tooling for writing and testing grammars.
• Code Generators: tools that take some type of high level input (typically graphical) from which VoiceXML markup can be generated.
• Syntax Checkers/Editors: Editors that accept VoiceXML and related markup and inform the user of errors in the markup.
• Debug/Test Tools: Tools that allow a developer run/debug a VoiceXML application in a standalone desktop environment.
• UI Tools: Tools that support development/refinement of the voice user interface (i.e. wizard of oz tools), but also tuning user interfaces by examining data collected from real system.
• Other: Suggestions that did not fit in any of the above categories.

Figure 3 illustrates that interest in tools in the Debug/Test category dominated with a total of 42 responses. The Syntax Checkers/Editors and UI Tools categories closely followed this, with 34 and 21 responses, respectively. Perhaps what surprised us most in this section was the relatively low interest in grammar-related tools. We would have expected more interest in this area, given that this is a non-trivial part of developing a good voice application. One theory is that the grammar tools shipped by the major speech resource vendors with their recognizers are adequate and this particular need has already been addressed.

Some of the ideas suggested in the "other" category include:

• A centralized VoiceXML Test Portal that has a VoiceXML gateway from each of the major platform vendors. The idea is for developers to have a one stop shop for quickly testing whether or not their application runs on all the major platforms.
• A flow chart generator - given a VoiceXML program, produce a graphical representation of it. Presumably the graphical representation could be used to visualize and/or debug a running system?
• Trouble Shooting Assistants - A VoiceXML editor feature that sort of resembles the infamous paperclip in Office. It would help the developer by suggesting better ways to accomplish a particular task in VoiceXML, and automatically identifying potential issues.

Question 2 was designed to shed some light on what VoiceXML tools are currently out there. While we recognize, given the fact that VoiceXML is a relatively recent technology, that many organizations at this point will be depending on internal home-grown tools to assist them, we were interested in determining how many companies felt they might make these tools available, either for free or commercially. The vast majority of the respondents (61) responded negatively to this question or didn't respond at all, while 26 respondents indicated they had tools that might be of use externally. See Figure 4. Among those responses that indicated they had developed tools, the responses varied. A few indicated that they had complete IDE's, others had smaller tools, like syntax checkers, while quite a few indicated they had tools, but they weren't mature enough to be used by other companies. In some cases, the tools already are commercially available, but bundled with the company's platforms.

The last question was designed to help us gauge whether or not there was enough interest in tooling to attract people to participate in a Forum tools initiative. The response (see Figure 5) indicates there is, and we have recently kicked off a tools activity within the Forum. Participants who expressed interest in this, and provided their email address can expect to be contacted soon!

Conclusion: In general the responses to the questions in Section I lead us to conclude that there is a lot of interest in VoiceXML related tools (debug/test tools in particular) that are not widely available today. In addition, a lot of the tooling that is being used today seems to have been developed internally, and there is perhaps an opportunity for stand-alone VoiceXML tools that can target a variety of deployment platforms.

Part II - VoiceXML Conformance

An overwhelming majority of the respondents indicated that an open VoiceXML conformance test suite and standard test reporting system is critical to your business, as shown in Figure 6. The typical response from application developers was that it would ensure their applications were portable to all platforms. The typical service provider response was that it would help them in vendor selection when it came to choosing platforms. Platform vendors tend to see it as a marketing opportunity, as it gives them an opportunity to demonstrate the quality and completeness of their implementation, as well as an aid in making sure they are keeping up with a still evolving specification. A number of response expressed the notion that conformance must be formally defined, tested and certified by an independent "authority".

The ten or so responses that indicated conformance was not critical did not in general provide much explanation as to why they felt that way.

The second question in this section was directed towards platform vendors and was intended to determine how many of them take the VoiceXML Forum's conformance program seriously. (i.e. how many of them intend to subject their platforms to conformance testing, once defined.) The results here are quite interesting as 39 responded they would; yet only 21 identified themselves as platform vendors in the demographics section. We suspect that either the question was confusing, or perhaps people answered affirmatively to emphasize that conforming platforms are indeed important to them. See Figure 7.

The third question was directed towards application developers, and to a certain extent service providers. The results at first are rather surprising, one would think the application developers would be more keen on conforming platforms at this point then the platform vendors. See Figure 8. However, given the fact there are currently no formally certified conforming platforms at the moment, and the fact that application developers typically target platforms that exist, maybe the response is not so surprising, but more indicative of a poorly conceived survey question? In any case, the idea was to determine if application developers were intent on targeting conformant platforms once they exist.

It should be noted that the 5 no responses almost uniformly explained that they would target platforms based on what their customers want, rather then conformance.

In the fourth question, a little over half of the respondents indicated a willingness to contribute to the Forum's conformance program. See Figure 9. Those participants who provided email addresses will be contacted, those who didn't, but would like to get involved, please contact us.

In terms of whether or not self-reporting of conformance test results was sufficient or if an independent party should do the testing opinions seem to be somewhat split. See Figure 10. While more people leaned in the direction of having an independent party do the testing, a fair share of people recognized the importance of having the independent party, but for practical purposes believe self-reporting was more realistic for now. A total of 10 responses took the middle ground suggesting that there was room for two different levels of certification: self reporting and independent. Smaller companies, with fewer resources, might opt for the former, while companies with more resources (or customers who insist on it) would take the latter approach.

Conclusion: While most (platform/service providers and application developers alike) feel conformance to be very important, there is a difference in opinion on whether or not its best to have self reporting of test results, or have an independent party do the testing.

Part III - VoiceXML Developer Certification

Just over half of the responses (52) indicated some level of support (i.e. answered yes or maybe) for the idea that formal certification of VoiceXML application developers would help assess the skill set of a developer. See Figure 11. While the majority of the negative responses didn't explain their position, several did indicate they felt VoiceXML was so simple that that certification is in effect overkill. (i.e. anybody can do it!) A couple of others indicated they had been burned in the past by hiring developers who were certified in other popular developer certification programs, only to find out later they didn't have the skills their certification implied they had.

In terms of specific interest in a Forum managed developer certification program, over 80% of both developers and managers expressed some level of interest. See charts for Questions 2 and 3 in this section in Figure 12 and Figure 13 below.

Interest level in Forum certified training courses was slightly lower, but nevertheless a substantial portion of both developers and managers expressed some level of interest. See Figure 14 and Figure 15 below.

Conclusion: Although support for certification programs in general as a tool in measuring developer's abilities was expressed, the response was mixed. On the other hand, support for a Forum administered VoiceXML Developer Certification program seems fairly strong.

Acknowledgements

This effort was made possible by the hard work and dedication of our colleagues on the VoiceXML Forum's Education committee: Kurt Kolok, Jim Larson, Rob Marchand, Brett McDowell, Ken Rehor, and Eric Tober.

Appendix - The Survey

Section I

1. Please specify which types of software tools would make you a more productive VoiceXML developer if they were made available?

Some example tools might include:

• Wizard of Oz assistants--Capture user input for dialog design.
• Code Generators: tools that take some type of high level input (typically graphical) from which VoiceXML markup can be generated.
• VoiceXML editors--Select and insert tags
• Syntax Checkers/Editors: Editors that accept VoiceXML and related markup and inform the user of errors in the markup.
• Rehearsal tools--Debug dialogs by reading output and typing input
• Testing harnesses--Drive iterative testing and refinement
• Debuggers--Catch logic errors
• Log report generators--Understand what users do in an application

2. Has your company developed VoiceXML tools that might be of use to other companies? If so, please characterize your company's tool(s) in a general way (i.e. test tool, editor, debuggers, etc.), along with their availability and approximate cost.

3. Are you interesting in helping define and develop a set of common VoiceXML tools?

Section II - Conformance

1. Is an open VoiceXML conformance test suite and standard test reporting system critical to your business? Please explain.

2. If your firm develops VoiceXML platforms, do you intend to ensure your platform conforms to the test suite being developed by the VoiceXML Forum? Please explain.

3. If your firm develops VoiceXML applications, do you intend to primarily target platforms that conform to the test suite being developed by the VoiceXML Forum? Please explain.

4. Are you willing to contribute to the Forum's conformance test suite?

5. Do you feel self-reporting of conformance test results to be sufficient, or should an independent party control test execution and reporting? Please explain.

Section III - Developer Certification

1. Do you feel that formal certification in VoiceXML Application Development will be beneficial to assess the skill sets of prospective developers? Please explain.

2. If you are a developer, would you be interested in obtaining a Forum sponsored "VoiceXML Developer" certification for yourself?

3. If you are a manager, would you be interested in obtaining Forum sponsored certification for your employees?

4. If you are a manager, approximately how many of your employees would you encourage to obtain certification?

5. If you are a developer, would you be interested in receiving Forum accredited training in VoiceXML Application Development for yourself?

6. If you are a manager, would you be interested in receiving Forum accredited training in VoiceXML Application Development for your employees?

7. If you are a manager, how many of your employees would you like to receive Forum accredited training in VoiceXML Application Development?

back to the top

 

Copyright © 2001-2002 VoiceXML Forum. All rights reserved.
The VoiceXML Forum is a program of the
IEEE Industry Standards and Technology Organization (IEEE-ISTO).