(Courtesy of Faber Novel)

We have previously written a comprehensive guide on chatbots. Then we started to develop objective metrics for measuring the performance of a chatbot. We also shared some of the success stories since success stories are rare and ambitious designers of conversational interfaces need to study them because, for every hundred of failures, there are only a few success stories. Now, we will be discussing how to test whether you are achieving an improvement in those metrics or not. A/B testing, which you probably used before for website or app, is also applicable to chatbot testing. It is a way to conduct beta testing and user testing.

What is A/B testing?

A/B testing is defined as comparing two versions of a product to see which one performs better. You compare two products by showing the two variants to similar visitors at the same time

Even though it has been used for ages in other fields of marketing. Currently, there are not many A/B testing alternatives available for chatbots. Basically, it is a way to experiment distinct characteristics of the chatbots, through randomized trials, companies can collect data and decide on which alternative to use. Testing a chatbot is conducted through an automated testing process. This testing automation makes it possible to hasten development processes of the chatbot and ensures further quality assurance.

We can dissect the process into two separate test steps of the chatbot design. One is deciding on the visual factors of the chatbot such as the design, color or the location of the chatbot on the web page.  The other factor to decide is the conversational factors. Such as the quality and the performance of the algorithm. Those two factors need to be tested for a better user experience.

We will try to cover the conversational factors first;

Conversational Factors

We have previously written an article regarding the key metrics for chatbots. Through A/B testing, the key metrics to follow will likely to remain the same. Retention rates and drop-offs will still be the significant factor in deciding the success of the chatbot. User engagement rates for different chat bot alternatives will still be significant.

One such test is about deciding on how to start the conversation, should the chatbot start with a standard salutation or use distinct messages such as emoji included messages. This would be the key factor since the customer funnel flows through the first engagement. If the chatbot is successful in eliciting the desired action which is making the user interact with the chatbot, we are more likely to reach a higher audience and higher conversation flow.

After the initial contact, different alternatives for messaging can be created. This might be achieved through using customer data. Data sources such as user’s search history or location can be a fantastic way to achieve that customization. Different conversational messages can be created, but the order of this message can dramatically affect the hazard rate of the customer. Since after the first few messages, most chatbots can detect the characteristics and keep the conversation flow. But artificial intelligence can experience problems if the initial contact never occurred.

One such chatbot company is Botanalytics. They provide chatbot analytics tools. Their platform can provide insights and reports such as users’ activity (as a graph and number), conversation activity (as a graph and number), average conversation steps per user, average conversation per user, most common keywords, most active hours, and average session length. Botanalytics provides correlation analysis to bot owners. It is possible to elaborate the details of the test groups and cohorts.

Second is deciding on the formality of the language, the question is to decide on whether using a formal language increases the engagement or not. Depending on the customer or the user profile, this is of crucial importance, multiple responses need to be tested. Therefore, the level of formality will shape the input provided by the user hence making it a rather difficult conversation to process by the artificial intelligence algorithm.

(Courtesy of http://spadeworx.com)

The tradeoff is whether the language of the bot increases and brings a greater return on engagement or not. Since we have the instinct to buy from or engage with the people who share the same characteristics, bot responses will affect this dramatically. Hence the only channel the user experiences the chatbot is through its language. Therefore, return on engagement will be the key metric for that type of A/B testing.

Visual Factors

Design of the chatbot is also important, but this is rather a not-so-technical side of the chatbots. Still, it is a crucial factor for the success of user experience. This can be done by changing the frame color or button color. Basically, this is the part where the firms would utilize the traditional A/B testing procedures.

The effects of visual and conversational factors are needed to be studied further. Currently, there is no data regarding which factor influences more. Separating and deciding on which factor to focus on is still important. But right now, as a whole, A well-structured and engaging UX in messenger chatbots may boost retention rates up to 70%.

For the chatbots, the companies can still utilize the methods mentioned in the survey analytics article. The right experimental design will make it possible the construct the right counterfactual and provide objective results. But the most basic steps for deciding on how to implement a chatbot A/B test are as follows;

7 Steps

  1. Choose the platform to conduct the A/B testing
  2. Analyze the chatbot funnel. Create a list of visual factors to conduct the test.
  3. Do the same for conversational factors, different algorithms, different structures
  4. Decide on the test method to use, control for interactions between the factors to be tested, gather as much data as possible
  5. Compare and analyze the alternatives, if necessary, test for additional factors
  6. Keep testing, keep it as a dynamic process and achieve higher performance
  7. Improve your Chatbot and Enjoy!
(Courtesy of softwareadvice.com)

Chatfuel is one such platform that helps enterprises for A/B testing and other enterprise solutions. They started in 2016 with the goal to make bot-building easy for anyone. They started as a Telegram platform, now they specialize on Facebook Messenger, where some of their key customers are NFL and NBA teams, publishers like TechCrunch and Forbes, and millions of others.

As aimultiple.com, we provide AI vendor profiles include a detailed description of their services and 500+ customer references in 100+ use cases. To find out more about autonomous checkout systems or other use cases, visit us.

Are you looking for an AI vendor?

Let us find the right vendor for your business


How useful was this post?

Click on a star to rate it!

Average rating / 5. Vote count:

What did you like about this post?

Appreciate it if you leave your name and surname so we can publish your review as a testimonial

1 comment

Leave a Reply

Your email address will not be published. Required fields are marked *