Webinar Recap: Question Answering and Virtual Assistants With Deep Learning

I recently hosted a webinar with my colleague Andy Liu on “Question Answering and Virtual Assistants with Deep Learning.” I’ve linked the full replay below and am sharing this post to break down some of the points that Andy and I covered in more detail.

Watch the Entire Webinar Recording “Question Answering and Virtual Assistants with Deep Learning”

First, some definitions (with apologies to the Deep Learning aficionados out there who already know some of this.) A Question Answering (QA) System is a system that automatically answers questions asked with natural language. Questions tend to be slightly longer than a standard key word search.

A chatbot is a computer program that uses natural language processing (NLP) and artificial intelligence (AI) to answer questions and elicit more questions to conduct a conversation.

Deep Learning is part of a broader family of machine learning methods based on the layers used in artificial neural networks. Learning can be supervised, semi-supervised or unsupervised.

Current chatbot solutions are usually narrowly focused on a few specific areas but do not cover a broad range of scenarios. They may be able to answer questions for common use cases like hotel booking or restaurant reservations out of the box, but they can handle only a limited number of questions.

Chatbot administrators manually provide examples, specify patterns of intent, build ontologies or use rule-based approaches. These chatbots only rarely use the company’s own knowledge bases. They aren’t really learning and having a conversation — they are just following a script.

Deep Learning to Power QA Systems

At Lucidworks AI Lab, we have taken a different approach to designing a virtual assistant system. We use Deep Learning to power QA solutions.

Call centers and internal IT teams have used these systems for self-help avoidance of calls or tickets. E-commerce teams have used rules engines for product recommendations.

Some have extracted QA pairs from Slack and email conversations to extract some knowledge. But those legacy systems can’t scale their intelligence with the pace of the modern business. They can’t keep up with changing customer expectations. Based on static rules and ontologies, they fall behind, fall out of sync, and provoke users to go back to Google for answers.

The Lucidworks Question Answering system automatically recommends answers from a dynamic FAQ pool or turns to historical data and finds relevant answers to similar questions that have been asked before.

Even if no historical question-answer pairs exist, you can begin with a cold-start solution, based on unsupervised learning. Overall, you’ll be able to  improve search for long queries by leveraging neural semantic search capabilities.

Deep Learning With Small Training Sets

The Labs team conducted a comprehensive study comparing various methods and learned that an approach based on Deep Learning provides the best results, even with small training data. This approach doesn’t require some of the heavy feature engineering work required by classical approaches to NLP.

QA systems can be trained in Docker containers, which makes it easy to use either on-prem or in the cloud.

Paired with auto pilot mode, it provides automatic parameter tuning and dynamic decisions based on data and hardware information. This means that users who are not data scientists can run QA modules and get state-of-the-art results with minimum supervision. Or, data scientists can use an advanced mode, without the standard guardrails.

Fusion supports optimized query pipelines that enable fast run-time neural search. It leverages Solr token-based search or dense vector clustering for retrieving initial candidates for further reranking by the QA model.

As we mentioned in the webinar, this functionality is not yet generally available in Lucidworks Fusion, but is available to select customers through the summer. If you’re interested in evaluating this Question Answering (QA) functionality, you can pilot the Lucidworks Question Answering solution in early preview before it becomes generally available this summer.

Sava is AI Research Engineer at Lucidworks, whose main focus is researching, developing and integrating cutting-edge NLP algorithms into Fusion AI platform. His work helps users obtain more relevant search results that are also enriched by additional information like sentiment, topics, extracted named entities etc.