Metrics: Measuring Your Success
Two metrics are key when determining if a bot your team has created is successful:
- Transaction completion rate
- The user’s perspective
Transaction completion rate is relatively easy to obtain, while obtaining the user’s perspective is very difficult.
Transaction completion rate is a measure of the number of interactions the intelligent bot is able to automate as a percentage of the attempts it tries. This number is critical for determining the return on investment (ROI) an intelligent bot may provide. One way to think of bot ROI is to compare the burdened cost of doing customer support only with live agents in the contact center versus the cost for achieving the same support level using a mix of virtual and live agents. This difference in cost is the ROI when examined over a period of months or years.
Other measurable transaction-related statistics include the following:
- Recognition accuracy – how well the bot translates spoken text into written words
- Intent recognition accuracy – how often the bot correctly recognizes what the customer wants
- Disposition – comparing measurements like customer retention rate or churn rate before and after the bot is put into service
- Upsell rate – did deploying the bot provide more time for the live agents to upsell and are you seeing a significant improvement in upselling revenue?
- Agent efficiency – are live agents more effective because the bot is able to collect introductory data, or perhaps the bot is able to suggest helpful information during the call?
The exact measures a bot-enabled contact center chooses to track will depend on the business case. Good practice suggests setting measurable success goals in advance of starting an intelligent bot project.
The customer perspective side of using intelligent bots is much more difficult to measure. The idea here is trying to determine how the customer feels when interacting with an automated virtual agent as opposed to working with a live agent. The backlash of a poor interaction with a live agent can sort of be measured by the sentiment analysis in the discussion with the agent. In a poor intelligent bot interaction, the person will simply disengage. This makes it harder to get a sense as to how people really feel about using the bot.
There are some indirect measures of satisfaction, however. One is watching the Net Promoter Score to see if it moves up or down after bot introduction.
Another indirect measure is watching to see which customers return to use the bot. If they come back, this implies (but does not necessarily prove) that you’ve developed a useful bot.
Dialogflow in Action
Vonage started working with Google about 18 months ago, and it was also one of the CCAI launch partners. Vonage is highlighting a proof of concept it has in process with its customer, American Financing (see related
No Jitter article). American Financing is implementing Virtual Agent (essentially Dialogflow) and Agent Assist within the framework of Vonage’s customer contact center.
In this application, both the intelligent bot and Agent Assist are working off the same knowledge base containing unstructured data that has been fed into CCAI. During working hours, a customer can call in and connect to a highly trained agent who is supported by Agent Assist. However, after hours, American Financing has fewer agents on call. Customers calling in during this time can receive general information from the bot and are then helped by live agents, as needed. These agents rely heavily on the information Agent Assist provides them to offer reliable information to these after-hours callers.
What’s Next
The next article in this series will appear in late June, addressing the questions:
- When would you consider building your intelligent bots using just Dialogflow without a contact center partner?
- What do CCAI contact center partners or CCaaS partners offer that you don’t get using Dialogflow alone?
