Debugging Through Google is Rotting My Brain
A couple of weeks ago I built a National Provider Identifier (NPI) lookup app as a proof of concept for the Vaadin portlet stuff we’re doing. For the search portion, it used a publicly-available, for-pay REST service to search for a provider’s NPI. Our architect, however, pushed me to the freely downloadable NPI data file and told me to set up an Apache Solr instance to do the search.
I know bupkis about Apache Solr except that it embraces the Flickr spelling approach and it lets you search for things, but an online tutorial and a few Ruby scripts later and I had a working Solr instance on my local machine, complete with the entire NPI database. Spinning up Solr with the NPI database on the target Linux server proved just as easy. When I discussed Solr startup scripts with the architect on the project, however, eerie background music started playing and I knew I was in for trouble. He told me to install the Solr webapp and run it on the app server instead of as a standalone server.
Actually, I had no such foreboding. The request was reasonable and the approach superior, so I found and installed solr.war on the app server. I also configured the two app server variables that I’d read were necessary (solr.solr.home and solr.data.dir). I fired up the app server and tried to access the Solr data and got an error: “missing core name in path.” Hmm. Time to debug. And you know what I did first, don’t you? I Googled “missing core name in path” and got a page of magic answers.
I read through a ton of results, trying all the solutions I found (even those that made no sense), and nothing worked. I became more and more desperate, especially because it was Friday afternoon and I was scheduled to go on vacation all the next week. I eventually became convinced that my app server didn’t see the solr.solr.home variable I’d configured, so I began spraying variable declarations into my app server, anywhere I could find a place to define a variable, like a dog marking trees. I stopped and started the server as if I were driving in Chicago’s rush hour. Each restart brought hopeful anticipation and glum dejection. I was ready to juke out the architect, write an init script for a standalone Solr instance (which worked fine!), and blame the sysadmins if it were ever detected.
Before acquiescing, however, I decided to look at the log files, which showed me that the app server knew all about my solr.solr.home variable. I was chasing the wrong squirrels. I continued reading the log file, though, and found this:
[#|2012-03-16T21:09:18.721+0000|SEVERE|glassfish3.0.1|org.apache.solr.core.CoreContainer|_ThreadID=27;_ThreadName=Thread-1;|java.lang.RuntimeException: Can't find resource 'solrconfig.xml' in classpath or '/opt/solr/./conf/', cwd=/opt/liferay-portal-6.0.6/glassfish-3.0.1/domains/domain1/config
Aha! A real error message! I checked, and solrconfig.xml was indeed not in /opt/solr/./conf — it was in /opt/solr/npi/solr/./conf. I changed my solr.solr.home to /opt/solr/npi/solr, restarted, and my search was on!
At this point, I should be claiming victory. After all, I weaned myself from debug-by-Google-search and returned to old fashioned debugging by looking through log files and figuring out exactly what’s going on. Alas, however, I wasn’t the one who thought to look at the log files. One of the Google results I read suggested I do that.