Guest post by Simon McLoughlin, Lead Mobile Engineer, Innovation Exchange, IBM Ireland Ltd.
As discussed in our previous blog post Can we simply listen to social media to capture the publics voice?, for Project MIDAS we opted to build a chatbot to gather the data we needed. Initially it was difficult to change our way of thinking, we were used to getting data instantly from APIs. But the data we got from our tests proved to be worth the effort. It was specific to our use case, it wasn’t ambiguous, we didn’t need to infer opinion, we just had the answers we needed.
The challenge now was taking this approach and finding a way to make it work for any set of questions that policy makers might have. We worked with IBM marketing to understand what works and doesn’t on social media and designed a tool to build a chatbot powered survey, keeping in mind what we learned. Here are some of the key things we tried to incorporate.
Keeping it short:
One big take away from marketing was an understanding of how little time we can expect to have someone’s focus for on social media. A large portion of social media interactions take place on mobile phones. It is very common for phone calls, text messages or notifications to interrupt the user. If they leave the survey, it’s very unlikely they will return. We need to ensure our survey is as short and quick to access as possible.
We can get to the questions as quick as possible by not forcing the user to read large pages of introductory text, instructions and FAQ’s. Instead we automatically pre-train all our chatbots to identify questions from the public and have a catalogue of answers ready to go.
Using decision trees, we can decide which question to ask next depending on the user’s previous answer. If the user is unlikely to answer anything other than “yes” to question 2, based on their answer to question 1. Then why waste their time by asking them that follow on question. Instead let’s ask one of the other key questions to make the most of our limited time with the user.
Smarter multiple-choice questions:
We created a form for policy makers to use when creating multiple choice questions. Each option is autocomplete using Dbpedia. The metadata we obtain from this allows us to train the bot to spot the option in freeform text. This means we can ask members of the public freeform questions rather than asking them to choose “a”, “b” or “c”. This gives members of the public the chance to add additional explanations that we can analyse, or we can have the bot ask subsequent questions to get more details.
Spotting unexpected responses:
The bots can now identify the multiple-choice answers that we are expecting to see. Using generic named entity recognition combined with this data, means the bot can detect when a member of the public mentions something we weren’t expecting to see. This gives us an opportunity to ask follow on questions and get potentially valuable information that would have been lost in a traditional survey.
This can be very useful for validating the questions you are asking the public. If you ask a multiple-choice question and people are frequently replying unexpected answers, it might mean there is a problem with the question itself. Maybe whoever wrote the question made a bad assumption or didn’t take something important into account.
Detecting questions from the public:
Members of the public may not be able to understand a question or may be confused. If this happens they can ask the bot a question instead of giving an answer. If the bot is unable to provide one, the owner of the survey will be notified. They can add an explanation and tweak the question so it’s less difficult to understand. Policy makers can do this in real-time as the issues are detected. They don’t need to wait until the end of the survey to discover there was an issue with the second question. This ensures the data is as accurate as it can be with no ambiguity.
Engaging with the public:
Marketing and Analytics had both told us that the best way to get members of the public to engage with you or your bot, was to engage with them. For that reason, the bot is capable of giving members of the public a high-level overview of the results of the survey so far. When dealing with health policies that will affect the public, being open and showing the ongoing results will have an impact on their willingness to share the survey, retweet the tweet, use the hashtag etc, to spread awareness of the campaign. Which will in turn increase the reach and the richness of the data.
We’ve taken all the above and developed a dashboard that allows policy makers to train a chatbot at will, quickly and cheaply. The responses will be automatically analysed by Watson to provide a clear set of results to the policy maker instantly. It is hoped that providing better and quicker access to the publics opinion will go a long way to help policy makers make better informed decisions that we can all get behind.