5 Things Every Personalization Engine Must Have

Personalization isn’t what it used to be. When the term “personalization” was originally coined, it referred to the ability to tailor digital content and experiences, based on expected behavior of groups. At best, marketers and merchandisers chose those shopper cohorts based mostly on demographic data, with some light behavioral analysis. Things have evolved quickly with the „personalization engine“.

This “old school” personalization treated you the same way as it treated a million other people that looked like you on paper and behaved in roughly similar ways in the real world. As it turns out, this wasn’t very “personalized.” Maybe “group-ized” would be a better word.

Hyper-personalization Is Now Possible

But technologies and consumer expectations have changed dramatically, and now uniquely personal experiences are possible. With far more data, more processing power, more machine learning algorithms and millions more applications in our pockets, legitimate 1:1 hyper-personalization is within reach.

Industry analysts have noticed. Gartner, one of the leading analyst firms, now defines personalization at this hyper-personalized, one-to-one level:

“Personalization is a process that creates a relevant, individualized interaction between two parties designed to enhance the experience of the recipient. It uses insight based on the recipient’s personal data, as well as behavioral data about the actions of similar individuals to deliver an experience to meet specific needs and preferences.”

Notice the very important qualifier: “between two parties”.

Gartner also shares a warning about how hard such fine-grained personalization can be:

“Multiple ways exist to personalize a customer’s experience in the physical and digital worlds. That these two worlds are converging makes personalization even more challenging — and more desirable for customers.”

More challenging and more desirable. In the case of personalization, this means that there is both great risk and great reward to building, buying or borrowing a personalization engine.

If you’re shopping for a personalization engine, remember that there are five things that it must have:

• Ability to learn
• AI that augments human intelligence
• Ability to scale dynamically
• An acute sense of intent
• Visible, intuitive analytics

1. Ability to Learn

Early personalization engines did not possess their own intelligence. Rather, they were based on rules. They were static repeaters of a human’s expert knowledge.

Ten years ago, all but a few ecommerce sites sold from relatively modest product catalogs. There were far fewer online shoppers. Human beings could review sales reports, update static rules, and keep pace with changes in preference and competition. Not any more.

Today, a personalization engine must include artificial intelligence (variants of which include machine learning or deep learning). Why? When a site sells millions of products to hundreds of millions of shoppers, there is no way for human experts to keep up with all the behavioral signals. Humans cannot notice subtle shifts in tastes and behavior. You do not want a personalization engine that behaves like a C student that only “studies to the test” and repeats pre-canned answers without looking for new information.

Today’s personalization engines must learn on their own. They should be self-tuning. When you choose a personalization engine, make sure the provider offers artificial intelligence that is seamlessly integrated and matched with the application’s purpose.

2. AI That Augments Human Intelligence

Let me reassure you. I (and my colleagues at Lucidworks) do not believe that it’s all over for us human thinkers. Robots will not rule the world. At least for the foreseeable future, algorithms will continue to be exceedingly good at repeating a small number of focused tasks, with increasing accuracy and precision.

Machine learning has a difficult time with a high degree of variability or contextual complexity. Human intelligence can switch context thousands of times per hour. We can do “fuzzy math”.

Think of the 1988 classic movie, Rain Man. Dustin Hoffman played Raymond, who is autistic. In the movie, Raymond is the human equivalent of a machine learning algorithm. He can do very focused tasks exceedingly well (like counting toothpicks, or counting cards). But the second the task turned complex, when context matters, Raymond relies on his brother Charlie, played by Tom Cruz. Charlie possessed human intelligence.

Your personalization engine needs both Raymond and Charlie. I like the term augmented intelligence. Dr. Kjell Carlsson from Forrester Research defines it this way:

“Augmented intelligence solutions combine the best of humans and machines. They use machine learning to analyze data and detect patterns at superhuman scale and leverage automation to act at superhuman speed. They leverage human expertise to go beyond the confines of the existing data, source additional data, reason, make judgement calls, and engage with other humans.”

Find a personalization engine that augments human intelligence with the power of machine intelligence. Bring the best of Raymond and Charlie together in one solution.

3. Ability to Scale Dynamically

Speaking of scalability, static rules engines have very real scale limitations. As product catalogs grow and user behavior shifts, it becomes increasingly difficult for digital merchandisers to understand all the signals that flow from shoppers on the site and then use those to update the business rules.

Use machine learning, clustering and classification to eliminate many of the rote, repetitive tasks of maintaining rules in an engine from the likes of Endeca. This frees human beings to use their human intelligence for creativity and innovation.

With merchandisers freed up — they can now turn their attention to be the tastemakers they were educated to be. Find a solution with a drag and drop interface to allow merchandisers to boost products as needed. They should also be able to experiment and analyze results without involving IT.

Speaking of scale — scale is a good thing, not something to be feared. More users generate more data to better hone intent. Just make sure you have a workhorse like Apache Solr at the core that can scale to support many thousands of concurrent users and hundreds of thousands of queries per second.

4. Acute Sense of Intent

Another area where static approaches to personalization fall flat is intent. Each and every search might be treated as a distinct event, floating in time and space. In the real world, consumers use very different search strategies. They use different words to mean the same things. Some people can’t spell so well.

That’s where signals come in. By capturing log files, transaction data, what people click on — and what they don’t, you can better analyze user intent.

So you need to make sure you use advanced artificial intelligence in two ways. Obviously, you want to employ AI to data as it is ingested. It clusters and classifies data when it comes in, to make it more discoverable at query time. Also at ingest, make sure to employ natural language processing functions such as Named Entity Recognition (NER) to determine if a document refers to people, places, products, or some other entity that you find important.

AI is invoked again when a user makes a query. This lets you predict query intent by combining domain context, user context, and query context. Semantic knowledge graph (SKG) lets you identify domain-specific entities (people, places, things, synonyms, misspellings, topics, phrases, etc.) within queries and understands their relationships within the searched content.

Your personalization engine must constantly self-learn and tune the relevancy of its results to predict user intent and use that to further personalize the user experience.

5. Visible, Intuitive Analytics

Modern personalization engines consist of dynamic, variable, interconnected parts. Each of those components can be individually tuned or optimized, but quite often the whole does not equal the sum of its parts (sometimes for the better, sometimes for the worse).

You want a system that can show system administrators and ecommerce merchandisers how business users and customers interact with the system. The system should let you visualize data categorically, and even understand individual customer or user journeys. For best results, make sure there are connectors to analytics tools like Zeppelin, Jupyter and Tableau so that data engineers and data scientists can interact with data through their most familiar interfaces.

Personalization is a never-ending pursuit. As you and your team walk that journey, it’s very important to see what’s working and what isn’t.

Hyper-Personalize and Prosper

Search is an excellent opportunity to personalize the experience of customers through a Digital Commerce solution or to personalize the employee experience through a Digital Workplace solution. Hyper-personalization is vital, regardless of the audience. This is true whether your goals are to reduce your bounce rates, target content more precisely, or tailor insights for different types of knowledge workers.

Those will be your rewards if you create a personalization engine that has the will to learn, gives you AI guided by human hands, scales without breaking your back, senses intent, and makes it easy for you to visualize progress.