Neural IR Search: Machine Learning & Deep Learning Algorithms

Neural IR search uses machine learning and deep learning algorithms to help systems understand the context and intent of whatever we put in the search bar. While simple keyword searching still has a place in our internet of today, this advanced technology is the future of precise search needed for things like legal documents or scientific papers.

Tia takes it all the way back to 1999 to explain how simple keyword searching has evolved into Neural IR search.

You can binge-watch all of season two or jump on back to watch season one.

Subscribe to the Lucid Thoughts channel and be sure to leave your questions and comments on each video.

Transcript:

Picture it – the year is 1999.

Cher helps you believe in life after love while you Ask Jeeves for smart food options to keep in your y2k bunker.

But the search results are… a little off.

Luckily things are very different today.

In the beginning of internet search, we relied on keywords and then phrases, but now we ask questions and expect the computer to know what we mean. Neural IR search is the latest way we’re helping computers truly understand even our craziest queries.

This method uses machine learning and deep learning algorithms to help systems understand the context and intent of whatever we put in the search bar.

One technique uses signal data to help determine which part of the sentence or phrase is most important. So, in this instance, it will recognize “what do I do with” as well as “Y2k bunker” to give you more accurate content.

Spotting synonyms is also important. Looking at user history and clicks helps the machine figure out which words are used in the same way.

Now let’s go a little deeper.

There are specific programs that help machines get an even more precise understanding of the meaning behind our sentences.

Algorithms like Word2Vec are trained to turn sentences and phrases into vectors, or lines on a graph.
And remember all of that trigonometry that you learned in high school? The cosine is used to determine the distance between specific sentences and phrases.

Basically, the closer the phrases are in the content, the more likely it is that that’s what you’ve been looking for.

However Neural IR search goes beyond this using signals (beat as the host points to the area where the signals video will appear). As we pick results, the system „learns“ the distance between the language we’re searching for and then clicking on.

The neural network recalculates the distance between the „vectors“ or „lines“ based on how we use the results. So the system learns from experience and results continue to improve.

And with these powers combined – this is a Neural IR Search!

(ahem)

While simple keyword search still has a place in our internet of today, this level of advanced technology is the future of precise search needed for things like legal documents or scientific papers.

Sorry, Jeeves!