Tuning Apache Solr for Log Analysis

As we countdown to the annual Lucene/Solr Revolution conference in Austin this October, we’re highlighting talks and sessions from past conferences. Today, we’re highlighting Radu Gheorghe’s session on tuning Solr for analyzing logs.

Performance tuning is always nice for keeping your applications snappy and your costs down. This is especially the case for logs, social media and other stream-like data, that can easily grow into terabyte territory. While you can always use SolrCloud to scale out of performance issues, this talk is about optimizing. First, we’ll talk about Solr settings by answering the following questions:

• How often should you commit and merge?
• How can you have one collection per day/month/year/etc?
• What are the performance trade-offs for these options?

Then, we’ll turn to hardware. We know SSDs are fast, especially on cold-cache searches, but are they worth the price? We’ll give you some numbers and let you decide what’s best for your use-case.

The last part is about optimizing the infrastructure pushing logs to Solr. We’ll talk about tuning Apache Flume for handling large flows of logs and about overall design options that also apply to other shippers, like Logstash. As always, there are trade-offs, and we’ll discuss the pros and cons of each option.

Radu is a search consultant at Sematext where he works with clients on Solr and Elasticsearch-based solutions. He is also passionate about the logging ecosystem (yes, that can be a passion!), and feeds this passion by working on Logsene, a log analytics SaaS. Naturally, at conferences such as Berlin Buzzwords, Monitorama, and of course Lucene Revolution, he speaks about indexing logs. Previous presentations were about designing logging infrastructures that provide: functionality (e.g.: parsing logs), performance and scalability. This time, the objective is to take a deeper dive on performance.

Join us at Lucene/Solr Revolution 2015, the biggest open source conference dedicated to Apache Lucene/Solr on October 13-16, 2015 in Austin, Texas. Come meet and network with the thought leaders building and deploying Lucene/Solr open source search technology. Full details and registration…