Solr Powered ISFDB – Part #8: Upgrade to Solr 3.1

This is part #8 in a (never ending?) series of articles on Indexing and Searching the ISFDB.org data using Solr.

I’m a little backlogged this week because of time spent testing/tweaking the Lucene / Solr 3.1 release, So I don’t have a lot of time to add features to my ISFDB search, but this did seem like the perfect time to upgrade it to Solr 3.1.

(If you are interested in following along at home, you can checkout the code from github. I’m starting at the blog_7 tag, and as the article progresses I’ll link to specific commits where I changed things, leading up to the blog_8 tag containing the end result of this article.)

Predictions

In a perfect world, upgrading from Solr 1.4.1 to Solr 3.1 should be trivial. Most users shouldn’t need to make any explicit changes to how they are using Solr to keep using the same functionality, but changes may be desirable to take advantage of new features (or improved performance). And of course: sometimes if you do really advanced stuff with Solr (like custom plugins) you may find subtle changes are needed.

With that in mind, my prediction is that I’ll be able to upgrade Solr pretty much just by swapping in the new war file. The one thing I know I’ll break however is my UI — because I remember discovering in my last blog post that Solr 1.4.1 and Solr 3.1 use different versions of jQuery, and rather then using my own copy, I reused that version directly. But (fingers crossed) that should be the only change needed.

So my prediction (w/o peeking at my build.xml or config files) is that the total number of lines I’ll need to modify to upgrade Solr should be between 2 and 10.

Starting the Upgrade

Looking at my build.xml, I already set things up to be parameterized by version number, so hopefully just changing the apache.solr.version variable is all I need to do to download the new version.

Next I run „ant clean fetch extract“ and things look pretty good for starters.

Next I take a look at the work/apache-solr-3.1.0/CHANGES.txt file, particularly the „Upgrading from Solr 1.4“ to see if there is anything that I should be wary of. One thing that does jump out at me (that I as a Solr developer really should have remembered before I made my prediction) is this..

* The DataImportHandler jars are no longer included in the Solr
  WAR and should be added in Solr's lib directory, or referenced
  via the <lib> directive in solrconfig.xml.

I totally forget about that. DIH, which has always been considered a plugin, is now not include in the war by default. Already I’m less confident in my prediction.

Oh well, minor set back that I’ll worry about when it causes a problem. Otherwise everything looks good. So it’s on to „ant run-solr“ and … disaster …

build:
     [copy] Copying 23 files to /home/hossman/lucid/isfdb/work/isfdb-solr-home
    [mkdir] Created dir: /home/hossman/lucid/isfdb/work/isfdb-solr-home/lib
     [copy] Copying 1 file to /home/hossman/lucid/isfdb/work/isfdb-solr-home/lib

BUILD FAILED
/home/hossman/lucid/isfdb/build.xml:180: /home/hossman/lucid/isfdb/work/apache-solr-3.1.0/contrib/velocity/src/main/solr/lib not found.

Velocity Contrib is Dead, Long Live Velocity!

So apparently we (Solr Devs) dropped the ball on documenting this in the „Upgrading“ section, but it is listed lower down in the detailed changes…

* SOLR-1957: The VelocityResponseWriter contrib moved to core.

…since it’s no longer a contrib, and now part of the core solr.war, I can remove the „copy“ I had in my build.xml to get those jars. After that „ant run-solr“ gets based the ant setup work, and starts up Solr.

One Contrib Moves In, Another Moves Out

Now I have to face the music about DIH. Fortunately it’s really straight forward, essentially the reverse of what I changed for Velocity, except that all of the Solr contribs are now a little cleaner about putting the actual contrib jar in the „dist“ directory.

Now Solr starts up clean, and a full import runs just fine with no problems. All the docs are indexed fine, and the UI Still works, although the jQuery debugging links are broken (as I expected). That’s also an easy fix.

Conclusion (For Now)

And that wraps up this latest installment with the blog_8 tag.

git diff will tell you that only 3 lines of code changed between „blog_7“ and „blog_8“ tags, but that’s kind of cheating because of the velocity libs vs DIH libs change — What it counts as a single line change was really „delete 3, add 3 unrelated lines“ bringing my total up to 8 (3 – 1 + 3 + 3). Still in the range of my prediction, but I’m disappointed about those lib changes that I forgot about. In any case, I hope to get back on track with adding new features in my next post … whenever I get a chance to work on it.