Lucidworks Managed Search: Same Solr, Fewer Chores

There’s an old proverb that I’d hear in my house all the time, “Everyone loves an elephant at the zoo, but nobody wants to take it home and clean its cage.” In the same vein, everybody wants a fast, scalable, and highly configurable search engine like Apache Solr, but nobody wants to manage the ops. If you’re reading this post, you’ve probably been to the zoo and dealt with Zookeeper. Breathe easy from here on out. Your job just became a whole lot more pleasant. Meet my dear friend Lucidworks Managed Search, managed Apache Solr for all of the major cloud vendors, the clearer of all obstacles to developer productivity. 

A Bit of My Personal Search History

Six years ago, back in 2014, when I was considering log aggregators at my startup there were three options because open source was my only requirement. Lucene, Solr, and Elasticsearch. Lucene was simply not enough and frankly required work beyond the amount of resources on our team at the time. Solr, which was impressive in many ways, did not have a viable managed offering in the cloud. Running on-premise servers in our startup was not only cost prohibitive, it would’ve been the most undifferentiated decision we could’ve made. So I settled on aggregating logs in Elasticsearch, running it in AWS because it was the only option, which was still a never ending game of whack-a-mole that worked well enough because it was just for logs. I didn’t care if I lost a document every now and then.

Managed Search GUI

We all love the command line but some architects, managers, and even busy developers need a sanity check and a break from Bash. That’s why we built the Lucidworks Managed Search GUI, an intuitive web application that makes understanding what’s going on like Algebra with a graphing calculator. Before, managing Apache Solr was more akin to Advanced Statistics with an abacus. Now you can view user activity, understand index error rates, and recover from a major failure all without leaving the LMS GUI. 

People have had trouble scaling open source search systems from the beginning and they still do:

With ease never seen before, control Apache Solr with a few button clicks. When you create a cluster, you specify the disk size, node type, and replication factor, or go with the defaults. Create, scale, edit, and delete with a mouse. The Lucidworks Managed Search (LMS) GUI makes applications of any scale simple to administer. Identify logs without digging through endless files and directories. Consider the web application atop of LMS the executive summary we’ve all been looking to email our bosses since our first production issue. If Solr gets out of hand, don’t call consultants—call the LMS API and consult the UI.

Solr Goodall

Managing full-text search is a beast so big and ferocious that Lucidworks built LMS to tame it. We poured the brain trust of many of my colleagues like Shalin Mangar, Tim Potter, and Erik Hatcher into the product. LMS handles the typical Apache Solr challenges of auto-scaling, modern security, and upgrading at speeds unmatched in the Solr userverse.

Auto-scaling

When it comes to scaling, we approach auto-scaling in two ways, offering options that suit your use case and ultimately cost less while providing improved stability. First, you can set up auto-scaling by query rate so that you only use the compute resources that you need but never have too little. When query volume exceeds a threshold, your cluster will scale to the appropriate amount of nodes based on your application(s) dynamic consumption. 

Most of those shoppers are now on your website. Black Friday in 2020 will only be higher volume than years prior. You can schedule scaling events such that your cluster will scale when you expect it will need more capacity so that predictable peaks don’t turn into wake-up calls to on-call engineers and rate-limiting on the PagerDuty API. We’ve learned over many years that it’s always better to act proactively when responding to unwieldy load. You can set up auto-scaling using the LMS UI or our secured cluster management API.

Built-In Security

Security may not be much of an issue for you today, but search clusters are under increasing attack so we have taken a number of steps to harden Solr i in addition to the usual networking steps we take to secure all managed customer deployments. All requests made to LMS are authenticated. They look more or less like this one:

curl -X POST  "https://pg08.us-west2.notonprem.lucidworksde.wpenginepowered.com/lucidwworks/library/solr/books/update?commit=true" \
   --header "Content-Type:text/json" \
   --header  "Authorization: Bearer bPg9mnJHiOiJLQk5qRlhHODhoWm1UQXZmY2xDMVU1cTlPeWpqOUxYZndmSjUwdWN0dEQ4IiwiYWxnIjoiUlMyNTYifQ.eyJ2ZXIiOjEsImp0fg654m85481o854p54yguuLotsofLinesofun-decryptablejibberishbecauseMathryuie51sjayrawg1rgrd5g1hyjksr1w81x5q4181sawsf151b5beg5wehm5t1u-5kiuol8iuouo4u8ouo4u5k17y51je5g1tge5ge84te8t3w8t4wUtw848jm4y84jy84ht84nt8n4rghsTJHTFpojGGkG5-SG15TG4RT4z54t54t54T5-15t4w5w8w87w89w7yh54h464GHHnb456jh4H4hh54g4j5j45klzjm40m4u5i45uy4y5u4-SUiQYHm-Si2QJsllmI7wwghHYA599nMSTAoZVvbG9-epoitoo1298h698h16ngnu9hnun5y9un319fuhmghmkmrejlnkmklajbcvatcqqwg" \
   -d '{"add":{ "doc":{"id":"1","title":"Lucene in Action","author":"Erik Hatcher"}}}'

We use OAuth2 and a trusted identity provider in Okta to ensure that every request made to your cluster contains the appropriate token and permissions. Lucidworks extended SolrJ and  PHP clients for communication with our API, and many more are on the way so that users can drop LMS into their existing applications with ease. Secondly, any CVEs that appear in the Solr distribution are removed immediately in Managed Search. We are often responding to them and investigating them as a part of our commitment to and investment in Apache Solr, but we also swiftly remove them from our product as soon as possible because it’s an important part of our value add. 

Upgrading was Painful Waste of Time

Upgrading to a new version can often feel like one of the least safe operations in Solr. Most longtime users remember the pain of upgrading from master-slave to SolrCloud. Many things may have broken, and some people saw performance regressions in the beginning. With our managed upgrade process, finely-grained coordination, and many of the improvements made to the Lucene and Solr layers over the years, you will notice many improvements with LMS.

There are still some limitations that mirror most users’ experiences with Solr. We occasionally have to pay off a bit of technical debt to get users to Apache Solr 7 because we cannot jump from versions 5 or 6. Some users, especially those with simple data models and schemas, will be able to make a multi-version jump. All users will notice a lower-cost upgrade experience once they’re in LMS because it is managed on our end. If there are ever breaking changes to how queries are made, you will know before there are any changes and you will receive guidance on what modifications to make on your end for things to work if any. If ever things do truly blow, “there’s an [LMS feature] for that.”

curl "https://pg01.us-west1.cloud.lucidworksde.wpenginepowered.com/managed/api/customers/lucidworks/clusters/$clusterId/backups/$backupId" \ 
-H "accept: application/json" \
-H "Authorization: Bearer $accessTokenManagedSearch"

State of the Art Disaster Recovery

One of the most valuable assets of Intellectual Property that has gone into the product is our backup and restore capabilities. Every day, or when you inform the system to do so, LMS takes a snapshot of your collections. If something ever happens to the cluster such that it cannot recover, you can click a button in the UI to simply restore a feature back to its healthy state. I’ve heard so many horror stories of unrecoverable Solr Clusters, unelectable leaders, and Zookeepers that were trampled, resulting in a corrupted index.

That’s why we built the Backup and Restore feature. We know the pain firsthand. That’s what the product and this feature are all about. Removing the painful aspects of building a state-of-the-art search engine, and at a lower cost and less hardware. We expect our users to innovate more swiftly and take greater risks knowing that they can easily restore their collections in their clusters usually in less that one minute. 

Index Your Data, then Move Very Fast

In most cases, for most operations, our customers will not need to change much. LMS just works. And if the phrase „just works“ sounds odd to you for an open source search engine, it should. We are bringing a new capability to the market, enabling ease of use, peace of mind, and most of all developer productivity. After all, developers running Apache Solr should be able to do so without all the pain and all the chores. You need to be able to focus on your value adding efforts that drive your business rather than “undifferentiated heavy-lifting.”

There will always be edge cases. If you have an ETL tool we have not encountered yet, we will work with you to get the integration done. If you are using a client library that is out of data, our engineers will smooth out the edges for you. If you are operating in a corporate environment that makes interacting with our APIs a challenge, we will make a path, working with your security teams and your network engineers to make the transition as smooth as life on the platform will be for your team.

The Lucidworks Managed Search roadmap includes an integration with Lucidworks Fusion, enhanced CDCR, a Query Replay and Solr Query Performance tool, advanced security for Solr’s new package management system, and workload partitioning for customers that want to target clusters for analytics workloads or clusters for search workloads. In the next few weeks, we will be announcing integrations to many of the most popular content management systems for businesses large and small. Many of them already exist today, though, so reach out if you want to test drive the platform. We will also feature blog posts from our Product Architect, Shalin Mangar, and our Chief Architect across all products, Tim Potter. As the start command that you will never need to run again always says, “Happy searching!“ 

Lucidworks Managed Search is now available on a limited basis. Contact us today to chat with the team. Nerds welcome! We’re all Solr contributors. If you want to request a feature, open a pull request here: https://github.com/MarcusSorealheis/Lucidworks-Managed-Search-FRs