Motley Fool Moves from Google Search Appliance to Solr

The Motley Fool’s award-winning website, Fool.com, publishes hundreds of articles each week, with over four million unique visitors and 11 million sessions per month. They chose to replace their Google Search Appliance with Solr — and hired Lucid Imagination to ensure successful implementation. “Lucene and Solr can definitely do what any commercial or proprietary search tool can do – certainly as well, or better, “ says Chad Wolfsheimer, VP of Technical Operations, Fool.com.

• Increased search relevancy and click-through-rate (CTR) by 40% compared to legacy search appliance; 48% reduction in web site exit rate (bounce)
• Big reduction in license subscription costs, and lower cost of ownership as content data grows
• Rapid migration from the Google Search Appliance; from working search platform within two weeks, to full production within 90 days

Customer Overview

Search is critical to help Fool.com users find the information they need to make investment decisions. And search results quality is vital to bringing visitors back: if they find what they need, they’ll continue to return to Fool.com – for additional subscription services and paid content. The Motley Fool chose to replace their existing implementation, based on the Google Search Appliance, with Solr — and hired Lucid Imagination to ensure successful implementation of Fool.com’s mission-critical search capability.

Key Search Improvements with Solr:

• Increased search relevancy and click-through-rate (CTR) by 40% compared to legacy search appliance
• 48% reduction in web site exit rate (bounce)
• Big reduction in license subscription costs, and lower cost of ownership as content data grows
• Rapid implementation; working search platform within two weeks, full production within 90 days
• Enhanced user search productivity by adding features such as sorting on both date and relevance, spelling correction, and “Did you mean…”

Challenges

Content is core to the Motley Fool business. At Fool.com, free content is provided to users that may not know much about the Motley Fool. This provides an introduction to Motley Fool as a brand, its investing style and ideas, and options for more paid services. The content composed of roughly 75% board posts, and 25% other content items including news, blogs, reports, and videos.

“We had something basic but useable running very, very quickly. So, that was a testament to our developers, to Lucid and to Solr, itself. We were very, very happy with that. “
– Chad Wolfsheimer, VP of Technical Operations, Fool.com

Fool.com faced a “good news, bad news” story: paid and non-paid user traffic were growing, but Fool.com needed to ensure that they could continue to effectively match growing content to that growing user population to in the face of several challenges:

• Spiraling Search Application Costs
  Fool.com’s legacy search application (Google Search Appliance) was expensive, and licensing costs were growing rapidly as the amount of content grew. In fact, software costs were outpacing hardware upgrades.
• Need For Improved Quality and Relevance of Search Results
  With substantial increases in community posts and user contributions, as well as new sources and types of content, the quality of results suffered, and users couldn’t consistently find what they were looking for.
• Operational Effectiveness
  The legacy search solution provided little visibility into technical configuration of search parameters, and tuning was difficult. Fool.com’s technical team found it hard to implement configuration changes or to understand their effects on search optimization.
• Scalability
  Content growth created issues with scalability; document indexing limits with the legacy appliance prevented addition of new content. Old content would need to be removed using scripted procedures which took up valuable development time.

Solutions

Initially, Fool.com was motivated by cost to start looking at other search solutions. The search appliance was expensive, and would become more so as the document collection grew. Freedom from per-document licensing fees was a major advantage for the Lucidworks Certified Distribution for Solr, the free, supported version of Apache Solr from Lucid Imagination.

“Lucene and Solr can definitely do what any commercial or proprietary search tool can do – certainly as well, or better. “
– Chad Wolfsheimer, VP of Technical Operations, Fool.com

More important than cost was the ability to customize applications. With many custom applications, Fool.com uses new functionality to innovate and stay competitive across their various business offerings. Rapid integration is essential; Solr’s flexibility and extensibility were a great fit for these requirements.

Fool.com also found that Solr delivered better transparency into tuning and optimization of search results. They could really understand the factors that impacted performance and quality. Unlike the closed “black box” appliance, they could have insight into the mechanics of search results, and create boosting algorithms that closely tracked their business goals up to the minute.

With the help of Lucid Imagination’s consulting team, Fool.com got rolling quickly. Fool.com defined what they wanted to achieve, and had a working pilot implementation up and running in just a few weeks.

After indexing all the content, the next step was to make adjustments to make sure that the most relevant information was delivered to users. This required a balance between relevance and recency. This boosts some specific content types that were thought to be a little more valuable than other content.

With Solr, the technical team at Fool.com was able to add Multi Selecting Faceting, so that users could stipulate subsets of the collection, and navigate to the kind of content they wanted iteratively based on their search results. Within roughly 90 days – which covered development, test, and production deployment – the implementation was complete.

“Our Lucid Imagination consultants absolutely rocked.”
– Danny Hsia, Director, Fool.com

Because here are no licensing costs associated with Solr, Fool.com enjoyed significant savings by eliminating the licensing fees associated with the previous packaged search appliance and software they had purchased. They also avoided additional costs due to increased amounts of content. Dramatic improvements were also quickly seen in click-through (CTR) and exit (bounce) rates – and these improvements have been sustained. Finally, the site expanded its range of content types from two to five.

„We’re very, very happy. Our users are telling us the results I’m getting are much better than they were before. In turn, we think that’s going to turn into success in terms of revenue to the business.“ Danny Hsia, Director, Fool.com

Software and Hardware

• Microsoft IIS 6.0., .NET web tier
• Microsoft SQL Server
• Microsoft Windows Server 2003
• Dell 2950 master server, with 8GB RAM master server
• 3 Dell 2950 query servers, each with 8 GB RAM