Web focused metrics do not calculate misunderstood requests or message delays. Chatbots require new metrics. If you are already familiar with web metrics, jump to chatbot specific metrics:
We have previously written an in-depth guide on conversational bots/chatbots. The basics of chatbots are provided in another article. We argued that the chatbots will change the world and it is important to catch-up with the technology. It is also important to be aware of the capabilities of chatbots and constantly measuring the performance of the chatbots. Therefore in this article, we will share some of the key metrics.
What is a metric?
a metric is a quantifiable measure that is used to track and assess the status of a specific business process
Defining the metrics is important. Since many of the capabilities of the chatbot will be measured through those metrics. For a newly created chatbot, those metrics can fluctuate dramatically. After implementing the chatbot, companies need to monitor it closely. Since the expectations are efficiency enhancement, faster response, and greater conversion, companies need to define the right metrics. This way, the performance of the bot can be monitored and improved efficiency. For data scientist, this dimension of the issue provides many use cases.
- 1- Total Users: This is the most basic metric. It captures the number of people using your chatbot. This matter because its trend shows the change in the number of users and therefore the amount of data your chatbot has been exposed to. Also, this would provide critical information regarding the market size calculations and potentially the effect of your chatbot.
- 2- Active Users: Active users can be defined as the people who read a message in the chatbot in a defined time frame. This is your potential targets. Measuring the potential effects of a promotional activity can be estimated from that number. The number of people who read your message is critical. This is something like the promotional content on social media. Engagement is not guaranteed but, the content is seen by the people
- 3- Engaged Users: Those users are the ones who communicate with the chatbot. They receive and send messages. This is important since your chatbot will be able to provide the conversation statistics based on this sub-sample. They are likely to shape your decision regarding the effectiveness of the chatbot. It doesn’t make any sense if the chatbot is not able to start the conversation with the users.
- 4- New Users: This metric captures the overall success of your chatbot promotion campaign. New users will be necessary to keep an active user number. Customers preferences changes over time and the amount of interaction with the chatbot shows an exponential decay. For that reason, new users will keep your customer base strong.
First four metrics capture the overall trend in your user base, but you will be needing a greater detail regarding how an individual interacts with your chatbot. One such metric group is the message metrics.
- 5- Conversation Starter Messages: This is the number of messages where you start the interaction by the bot. This critical for measuring the organicness of your platform. It is possible to elicit a response by sending messages to users, but as the time moves forward, companies would need a lower number for that metric. Since we will implement chatbot for customer relations management and digital marketing, after the initial greeting, we need continuing users to send messages to chatbot directly.
- 6- Bot Messages: Bot messages are the total number of messages sent by the chatbot in each interaction. This measures the length of a conversation between a customer and the chatbot. We normally want the number of messages to be high, but there is one critical condition, our chatbot needs to respond correctly. In case of misunderstanding or failure to comprehend the input by the user, the chatbot will say similar words repeatedly.
- 7- In Messages: This category shows the messages sent by the user. We need to see whether the user engages in with the chatbot or not. If this category is significantly low, we don’t need to use a chatbot. Using regular social media such as a Facebook Page or a Twitter account may make more sense, rather than using a Facebook Messenger chatbot or a voice-assisted technology.
- 8- Miss Messages: Miss messages are the ones our chatbot can’t process. This metric may be hard to calculate. Requires the times the chatbot miss interprets the input. This would be a key metric if the firm starts to engage in countries where the language is used more idiomatically.
- 9- Total Conversations: Number of conversation started and successfully completed on a given day. This is the concept engaged users
- 10- New Conversations: Number of new conversations started. This captures both the inexperienced users and the conversations that are initiated by the returning users on a different matter, problem or an order.
Startups such as Pulse Chat focuses on measuring the number of messages and provides the details in their platform. Their innovative solution makes it possible to maintain and improve the quality of the conversation instantly.
We will now introduce some other metrics that are critical for measuring the performance of a chatbot.
- 11- Retention Rate: This is the percentage of users that return to using the chatbot on given time frame. This important since we need to keep the customer engage to extract valuable insight regarding the customers’ preferences by making them spend time on the chatbot. Higher retention rate can be achieved by promotional campaigns such as chat-to-receive-discount or a lottery like a word guessing games. The important is keeping that high through an organic process. This mostly can be achieved by providing a high-quality chatbot that meets the customers’ expectations and needs.
- 12- Goal Completion Rate (GCR): This captures the percentage of successful engagement through chatbot. Users will probably try to reach different information or service. For an e-commerce company’s bot relevant goals can be informing user about a product’s details or purchasing a product. This shows that the number of times our bot successfully processed the input and provided the asked information. There are other ways to utilize this concept. One way is to mine data through the questions asked by the users. This would show an overall trend in consumer preferences; hence a greater emphasis can be placed on that issue while training the chatbot.
In the above graph, we can see an example of such use-case. Users mostly used the chatbot for scheduling their rides, hence greater emphasis should be placed on that issue, for keeping the consumer engaged and active.
- 12/13/14 – Goal Completion time/messages/taps: Chatbots need to provide a seamless and efficient experience and they have plenty of substitutes such as company’s web pages or apps. Minimizing the effort to complete a goal can improve user experience.
- 15- Fallback Rate (FBR): No robot is perfect; therefore chatbots are expected to fail sometimes, but what really matters is those failures happen regularly or in some extreme cases. Fallback Rate captures that information. This is the percentage of times our chatbot failed or experienced a near-failure situation. Our aim is to minimize this since we need a chatbot that replaces humans such as customer service representatives, salespeople, and call center people. When experience a high fallback rate, it might be right to try finding new data sources or training sets for improving the performance.
- 16- User Satisfaction: A new metric can be defined through the exit surveys. Customers or the people engaging with the chatbot can rate their experience to achieve further product excellence. This can be implemented as a binary variable such as “did the bot perform well? – Yes, or No” or we can create more complex evaluation forms to rank and provide points for each different category. This metric would capture the overall effectiveness of the bot from the user experience point. directly provided by the user.
- 17- Virality: Not all bots are viral but bots that motivate user to include other’s in the conversation can achieve viral growth. If your solution can involve others, that will surely aid growth. You can read an analysis of how viral features helped growth of the simple bot, Roll.
Those are some of the key performance metrics regularly utilized. It is possible to change the metric and create metrics. Content-based filtering is one way to achieve the desired details of the metric. Ad-hoc metrics can be defined, but for a typical use case, those 14 metrics will capture the most critical Chatbots are the major field to test and see the capabilities of artificial intelligence.
One major company focusing on the analytical side of the chatbots is Botanalytics. Botanalytics is based in San Francisco and is the leading company in conversational analytics tools. They were established in 2016, their main goal is to improve human-bot interactions and conversational UI through data. By offering analytics for bots, they empower bot developers, corporate brands and agencies to improve their chatbots with historical and real-time conversation data.
Are you looking for an AI vendor?