Creating a chatbot analytics tool
The problem
Soon after I joined, it became clear that the manual process of the design analytics wouldn't scale, and, as we roll the virtual assistant to a bigger audience, the manual approach might leave blind spots on critical design flaws. After some market research, we decided to build an analytics tool ourselves - there was nothing good enough on the market back then. Creating the tool ourselves, among other things, helped us ensure data privacy.
My role was to ideate and design the requirements and then polish the tool, and my colleague built the scripts to match these requirements.
Tool's objective
Find, quantify, and analyse the most significant problem areas on an intent level.
Target analysis on the main problem areas instead of blindly and manually trying to find and quantify patterns in thousands of rows of data.
Defining user stories
As a conversation designer,
I want to be able to detect "bad" signs, not on a conversation level, but on a decision points level, so that I can target my analysis for the main problem areas.
I want to be able to see at each decision point if customers deviate from the designed flow so that I can find the problem areas for further analysis.
I want these deviations from the designed flow to be grouped by type and quantified so that I can prioritise the most important ones.
I want to be able to see and download filtered data so that I can analyse that segment in more detail.
I want deviations from the designed flow to be grouped, highlighted, and quantified so that I know where customers break the flow and what's the volume.
Tool's performance
As a result, we had a tool that reduced approximately one-fifth of the designers' time spent on performance analytics, which improved efficiency and team morale (sticking to manual processes that everyone knows the algorithm can do is never fun).
Key takeaways
Before I start, I must say there're many great articles and guidelines around the process and core principles of conversational design, so I won't cover these. Instead, I want to talk about some insights that I had while working on the Virtual Assistant that isn't that mainstream and can be niche but worked perfectly well for us.
Setting expectations early and having the right personality is everything. Don't overlook the right welcome message - it is vital. Customers should know they're talking to a bot, not a human, so introduce yourself and your tone of voice.
The whole flow should feel conversational, users should be able to type at any decision point, and the bot should understand an entry. But sometimes, force-click buttons are a must. e.g. if a transaction is about to happen. A typed response can trigger a wrong transaction.
Small talks feel human, but don't go overboard with them. These intents can confuse natural language understanding (NLU). It's better to say that we don't understand than trigger small talk inappropriately.
Understand the underlying infrastructure thoroughly first. When designing new intents, you should spend most of your time understanding the business processes and tech infrastructure behind that particular problem you're trying to solve. The business side is essential for defining your personas and possible scenarios and understanding how we could serve these customers (what rules apply, what would live agents do).
Never take product structure and customer-facing documentation for granted; question all flows and decision points when building conversational design. Infrastructure might work entirely differently from what you expect by looking at a user-friendly product description. There might be technical nuances you have to cater for.
Improving the design is not always about making changes to the converation flow - you may find that a better customer experience requires API improvements. The designer's goal is to highlight all gaps, including technical.
If a user enquires about anything related to money, bring more trust by either providing exact amounts or directing customers to agents immediately. Approximate and indicative amounts seldom satisfy customers.
