Can we simply listen to social media to capture the publics voice? (It’s more complicated than that!)

Guest post by Simon McLoughlin from IBM Innovation Exchange, IBM Ireland

When it comes to making informed decisions about health policies, it’s just as important to get the publics opinion as it is to get the statistics. As we’ve seen in recent times, with elections and referendums, traditional means of surveying the public are not always effective. Organisations are moving more and more towards social media to better understand the public’s perception, but how do they do it?

Our initial idea was that we could use the Twitter API – their interface that allows programmers to access Twitter data –  to download a bunch of tweets that included people talking about a given topic and then run some machine learning techniques to understand the public’s perception on a range of health policies… but we found it was more complicated than that.

Some of the challenges we encountered were:

  • There was a huge amount of noise or false positives. We searched for “sugar tax” and found users tweeting Alan Sugar about income tax, or about a music album named sugar tax.
  • We assumed the public are tweeting the exact answers we were looking for – but we quickly found that most people talk in generalities about health on social media detailed – You can get a high-level sense of the publics sentiment towards a policy, but you won’t get a 5-point plan detailing how the public think it could be improved.
  • How do you interpret what a person is saying when they retweet or repost?
    • Did someone retweet because the share the same opinion?
    • Are they just interested in both sides of the argument?
    • Did they retweet it sarcastically to make fun of the point?

That meant that simply downloading data from social media networks was not going to provide reliable results to inform policy makers. To get the answers we needed, we decided we were going to have to engage with the public and ask them the questions we needed answers to.

Rather than having room full of researchers manning computers and chatting to people via instant messaging we needed to look at ways of automating the conversation then conversation we wanted to have with the public.  We built a dashboard that allows us to feed data into IBM’s Watson Conversation tool on IBM Bluemix cloud service, to control an automated social media chatbot that can engage with members of the public. It can ask them questions about the policy, ask them for suggestions or improvements and dig down into why they feel this way. Where there is ambiguity the bot can educate the public on what the policy is or means to them. This is very different to the standard helpdesk style chatbot were it simply tries to answer a question for a customer, instead our bot is driving the conversation in order to get something from them.

Using Watson’s Natural Language Understanding, we can analyse all of this content and aggregate it into a clear and concise breakdown of how the public feel about the policy, what they would do differently and the issues they have with it. Having a tool like this will allow policy makers to quickly, easily and cheaply get the answers they need, to provide the support the public need.

We at the Innovation Exchange are privileged to be working with the MIDAS consortium to help make this technology a reality to improve health polices across Europe and the wellbeing of the general public. The IBM Ireland Innovation Exchange (IIX) was launched in 2015 as a dedicated vehicle to unlock the potential of cross industry and sector convergence to stimulate economic, scientific and societal opportunity for IBM. Since its inception the IIX initiative has grown and expanded its formal partnerships with over 150+ multinational, small & medium sized companies and research centers enabling innovative multi-party collaborations producing outputs that are bleeding edge and unique.