MySolr

I got to write this down this time although this is not related to Solr/Lucene.  This has come back and bite me many times.  The error is due to incorrect SELinux context on the DocumentRoot directory.  Here is what you need to do to correct it:

chcon -R user_u:object_r:httpd_sys_content_t <directory>

You may want to check on /var/www first to see if the context is correct by issuing this command:

ls -al --context /var/www

Okay, this is it.  This should fix this annoying issue.

I got the chance to do a barebone Lucene implementation for a client with 40 million records.  They liked to introduce faceting on the author field.  I was tempted to just go ahead with Solr.  However, it’d be counterproductive to the project because they don’t need the full package provided by Solr.  My client only wants to build the facets on top of their index with minimal changes.  Bobo became the obvious choice in this matter.  To say the least, Bobo is amazingly simple to use and yet it provides decent performance.

The biggest roadblock we faced with this implementation is the memory footprint.  When the author index was loaded using Bobo, it allocated 12G of memory.  Initially, we set our young generation size way too small, the GC algorithm we selected, CMS (Concurrent Mark Sweep), had to constantly do full sweep after every 2-3 searches.  The full sweep would halt the entire service for about a minute before returning.  It was unacceptable as it pretty much killed search altogether.  It appeared that Bobo allocates quite a bit of temporary memory to calculate the facets.  Perhaps it was the nature of our data with a lot of intersection between authors that caused the excessive memory usage.  We slowly increased the young generation size from 2G (yes I know, it’s very small) to around 8G to get a stable system with virtually zero full sweep.

Here is our current JVM config:

java -verbosegc -XX:+PrintGCDetails \
     -XX:+UseConcMarkSweepGC \
     -XX:+CMSIncrementalMode \
     -XX:+CMSIncrementalPacing \
     -XX:+UseParNewGC \
     -XX:+CMSParallelRemarkEnabled \
     -XX:+DisableExplicitGC \
     -XX:MaxGCPauseMillis=2000 \
     -XX:SoftRefLRUPolicyMSPerMB=1 \
     -XX:CMSIncrementalDutyCycleMin=10 \
     -XX:CMSIncrementalDutyCycle=50 \
     -XX:ParallelGCThreads=8 \
     -XX:GCTimeRatio=10 \
     -Xmn8g \
     -Xms22g \
     -Xmx22g

This configuration works for us.  If you run into similar JVM garbage collection issue, I hope this set of configuration will help you too.

Join the forum discussion on this post - (1) Posts

One DataImportHandler(DIH) configuration people may overlook is the batchSize attribute.  If you start your JVM with enough memory to store the entire table, you won’t even need to set batchSize at all.  batchSize basically tells DIH to call setFetchSize through JDBC to bring back certain number of records at once.  If you use MySQL, you may still run out of memory even when you set the batchSize attribute.  That’s due to a limitation in MySQL’s drive where the setting is ignored.  The workaround is setting batchSize to “-1″.  This will pass Integer.MIN_VALUE to MySQL as fetch size and prevent the driver from running out of memory.

<dataSource driver="org.gjt.mm.mysql.Driver" url="jdbc:mysql://localhost/db"
user="root" password="root" batchSize="-1"/>

Join the forum discussion on this post - (1) Posts