Technical Benchmarking
Technical benchmarking, or comparing statistics generated by a call center system against published industry norms, is another common technique for evaluating a system. It has the advantage of using data which is already being generated by existing systems, such as average hold times, and call abandon rates, so the only additional expense is buying benchmark statistics from a third party.
Unfortunately, these statistics are often promoted as a measure of caller satisfaction, when they are really proxies at best. For example, it is clearly good if the average hold time is shorter than industry norms, but that doesn't mean customers are being well served.
In the worst case, relying too heavily on technical benchmarking can lead to a customer service operation managing to the numbers, rather than managing to customer service. For example, call center agents under pressure to reduce their average call time have been known to abruptly hang up on callers with difficult problems. That certainly will reduce average call length, but at the expense of customer satisfaction.
In addition, while technical benchmarking can help decide if a system is performing poorly, it can't tell you if a replacement system will be any better, since it is only meaningful once a system is rolled out.
Focus Groups
Focus groups, intensive interviews with small numbers of customers, are similar in many ways to Wizard of Oz testing. This method can generate a lot of ideas for improvement, and qualitative feedback about a new or existing customer service operation, but it can't generate statistically valid or comparative data.
Employee Test Calls
Employee test calls are a very common method for testing new automated systems. This involves having employees call into an application, and provide feedback for improvement (often through a survey). It has the sole advantage of being fast and cheap.
Unless the system is intended to be used by employees (for example, an HR hotline), this method can actually be worse than doing no testing at all.
The problem lies in the fact that employees are a very different group of people than customers. Employees are familiar with the jargon and processes of the company and industry, where customers generally are not. We have experience with several companies which successfully tested new applications using employee calls, yet found that the expensive new system completely failed to serve the needs of real customers.
As a result, we strongly recommend that companies not rely on employee calls to test a new system.
