Guest post by Gordon Doyle, Senior Software Developer, IBM Ireland Ltd
As discussed in our previous blog post “Improving surveys with chatbots” (http://www.midasproject.eu/2018/05/25/improving-surveys-with-chatbots/), we built a chatbot and a social media dashboard that allows policy makers to engage with the public and gain valuable insights on a promoted topic. In the background, the chatbot uses Watson Natural Language Understanding and Watson Assistant services to deliver the policy maker’s questions. These smart software services, along with the online resource Dbpedia, allow for a sophisticated, interactive question and answer service through social media giving the public a more engaging experience than a traditional survey, while delivering the same insights to the surveyor.
Social media as the user interface
The MiDAS social dashboard allows a policy maker to create a campaign that will be run through social media. Twitter was chosen as the initial social media platform to act as the user interface as it has a very large user base and it gives the public a way to voice their opinions. This is a good fit for the MiDAS social dashboard. It gives the policy maker access to their target audience and the public have a modern, familiar way to respond with their opinion. When a campaign is launched through the social dashboard, the chatbot uses the MiDAS Twitter account to send out a descriptive, initial Tweet about the topic. The public are invited to respond with their opinion through private direct messages to the MiDAS Twitter account. As the chatbot is a software service it needs a programmatic way to handle this interactivity. Twitter provides a “development toolkit” to do this. Tweets, Tweet responses and private direct messages can be automated using Twitter’s API.
The only constant is change
We held a successful initial trial of the social dashboard and the chatbot. We were planning some changes and our next trial when Twitter announced that their API model was “changing” (https://blog.twitter.com/developer/en_us/topics/tools/2017/announcing-more-functionality-to-improve-customer-engagements-on-twitter.html)! The chatbot uses Twitter’s Direct Messaging APIs to handle direct messaging between the MiDAS Twitter account and the engaged Twitter user. Twitter’s soon-to-be-deprecated Streaming APIs were used to handle the notifications necessary to enable the chatbot to work in real-time. This was to be replaced by a new Account Activity API with a different operating model. This posed a real challenge to our Twitter fronted chatbot! We would have to re-implement how the chatbot and social dashboard interacted with Twitter.
Out with the old, in with the new
Our saving grace is that we did not write much low-level code needed to interact with the Twitter APIs in the first place! We used an open source NodeJS Twitter client library called “Twitter for Node.js” (https://www.npmjs.com/package/twitter). Our code wrapped the functionality this client library provides in higher-level logical operations that the chatbot needs to perform such as reply to tweets and sending questions via direct messages. Our task now was to replace this Twitter client library with an alternative that would work with the new API operating model. We found “Twitter Lite” (https://www.npmjs.com/package/twitter-lite), an actively maintained, open source NodeJS Twitter client library that provides updated functionality to work with the new API models.
Up and running, again
The chatbot functionality is driven by the Watson Natural Language Understanding and Watson Assistant services. The Watson NLU service performs the sentiment analysis on the question responses and the Watson Assistant service provides flow control for the conversation. The results of using these Watson services determine how the chatbot responds to the Twitter user. The Twitter client library is used to post the chatbot responses to Twitter. We modified the existing calls to the old Twitter client library with the format used by Twitter Lite and removed the old client library as a dependency. The Twitter Account Activity API uses a webhook model to notify our social dashboard and chatbot that there is an event on our MiDAS Twitter account that we should know about. The webhook model requires that we provide an endpoint URL in the social dashboard to the MiDAS Twitter account’s configuration for the Account Activity API. Once Twitter has this endpoint URL it will send notifications to us. We setup this webhook endpoint and tested to make sure everything worked as before. The chatbot is now sending Tweets and direct messages again!