MySolr

I got to write this down this time although this is not related to Solr/Lucene.  This has come back and bite me many times.  The error is due to incorrect SELinux context on the DocumentRoot directory.  Here is what you need to do to correct it:

chcon -R user_u:object_r:httpd_sys_content_t 

You may want to check on /var/www first to see if the context is correct by issuing this command:

ls -al --context /var/www

Okay, this is it.  This should fix this annoying issue.

I got the chance to do a barebone Lucene implementation for a client with 40 million records.  They liked to introduce faceting on the author field.  I was tempted to just go ahead with Solr.  However, it’d be counterproductive to the project because they don’t need the full package provided by Solr.  My client only wants to build the facets on top of their index with minimal changes.  Bobo became the obvious choice in this matter.  To say the least, Bobo is amazingly simple to use and yet it provides decent performance.

The biggest roadblock we faced with this implementation is the memory footprint.  When the author index was loaded using Bobo, it allocated 12G of memory.  Initially, we set our young generation size way too small, the GC algorithm we selected, CMS (Concurrent Mark Sweep), had to constantly do full sweep after every 2-3 searches.  The full sweep would halt the entire service for about a minute before returning.  It was unacceptable as it pretty much killed search altogether.  It appeared that Bobo allocates quite a bit of temporary memory to calculate the facets.  Perhaps it was the nature of our data with a lot of intersection between authors that caused the excessive memory usage.  We slowly increased the young generation size from 2G (yes I know, it’s very small) to around 8G to get a stable system with virtually zero full sweep.

Here is our current JVM config:

java -verbosegc -XX:+PrintGCDetails \
     -XX:+UseConcMarkSweepGC \
     -XX:+CMSIncrementalMode \
     -XX:+CMSIncrementalPacing \
     -XX:+UseParNewGC \
     -XX:+CMSParallelRemarkEnabled \
     -XX:+DisableExplicitGC \
     -XX:MaxGCPauseMillis=2000 \
     -XX:SoftRefLRUPolicyMSPerMB=1 \
     -XX:CMSIncrementalDutyCycleMin=10 \
     -XX:CMSIncrementalDutyCycle=50 \
     -XX:ParallelGCThreads=8 \
     -XX:GCTimeRatio=10 \
     -Xmn8g \
     -Xms22g \
     -Xmx22g

This configuration works for us.  If you run into similar JVM garbage collection issue, I hope this set of configuration will help you too.

Join the forum discussion on this post - (1) Posts