MySolr

Just installed OSX Lion on my MacBook Pro last night and immediately I found my development environment was completely messed up. After the upgrade, all PHP modules I installed previously were gone. Some libraries I installed via MacPorts were gone as well. I tried a few things, “port selfudpate” and “port upgrade outdated”, in an attempt to stabilize the environment with no success. I was getting a lot of dependency issues. That prevented me from installing any more new modules. I read somewhere that installing MacPorts from their Subversion repository should help. I gave it a try and it appeared to have solved my problem.

First of all, check out the trunk from MacPorts, compile and install.
% mkdir -p /opt/mports
% cd /opt/mports
% svn checkout http://svn.macports.org/repository/macports/trunk
% cd /opt/mports/trunk/base
% ./configure --enable-readline
% make
% sudo make install
% make distclean


Then, update MacPorts:
# sudo port selfudpate
# sudo port upgrade outdated


You may still encounter some dependency issues. But at this stage, it should be very easy to resolve by temporarily pulling old libraries from Time Machine. For example, I ran into issue with gettext. The Lion upgrade removed this library /opt/local/lib/libintl.8.dylib that was used by gettext but it did not remove gettext completely. MacPorts was confused and refused to reinstall gettext. I had to go back to Time Machine to restore this file temporarily so I can set up gettext again. If you don’t have Time Machine, it would be difficult to acquire these old libraries. If it happens that you need this file, you can download it here.

You will need to force activate the module after you placed the library.
port -f activate gettext

Now, you can uninstall or upgrade the modules.

If you are still encountering dependency issues, try adding/removing binpath in /opt/local/etc/macports/macports.conf.
binpath /bin:/sbin:/usr/bin:/usr/sbin:/opt/local/bin:/opt/local/sbin:/usr/X11R6/bin

Some modules may need this binpath in order to install but most do not require this change.

This is not related to Solr/Lucene but I have to write it down somewhere so I don’t forget it next time.  I have seen this problem a few times in the past and have always forgotten about it.  Here is the behavior, when you run a Linux VPS environment, SOMETIMES (I have to stress this) hald does not play nice with your CD/DVD virtual drive.  It may ping the drive every second unnecessarily.  For most of the time, you will hardly notice it.  However, on some occasions, it may consume enough CPU and I/O that can push your idle machine to have an extra load of 1.0 or above.  It’s quite annoying when it’s happening in a small VPS where resources is already very limited.  If this is happening in your environment, you will see a process called hald-addon-storage lingers that always consume at least 1% CPU.  To confirm that it really is eating up your resources, you can use the following command to trace the system calls it makes:

strace -t -p <pid>

You may see a bunch of open calls, such as following, on your CD/DVD drive.

open("/dev/hdc", O_RDONLY|O_NONBLOCK|O_EXCL|O_LARGEFILE) = -1
EBUSY (Device or resource busy)

So, what’s the solution?  I haven’t found a good solution for it other than stopping this pulling entirely.  Usually, you only use the CD/DVD during OS installation so stopping it shouldn’t create any adverse effect on your environment.  Here is what you should do to stop it:

• Make sure haldaemon is still running, run following command to find your CD/DVD drive’s UDI

hal-find-by-capability --capability storage.cdrom

• You should see something like this

/org/freedesktop/Hal/devices/storage_serial_QM00003

• Now create this file, /etc/hal/fdi/information/media-check-disable-cd.fdi, with following content:

<?xml version="1.0" encoding="UTF-8"?>
<deviceinfo version="0.2">
  <device>
    <match key="info.udi" string="/org/freedesktop/Hal/devices/storage_serial_QM00003">
      <merge key="storage.media_check_enabled" type="bool">false</merge>
    </match>
    </device>
</deviceinfo>

And of course, replace the “info.udi” value with your own value.  Restart haldaemon now and you should be all good.

In one of my recent project implemented with Bobo-Browse for faceting, I ran into an issue with MultiValueFacetHandler’s 1024 values per field per record limitation.  I have some odd cases in my data set where a publication can have more 2000 authors.  This limit would stop at 1024 authors and left the rest of the authors uncredited in my search results.  It wasn’t a great loss as there are perhaps only a handful of publications with this many authors.  However, it was not great for the business so I had to create a solution.  After some searches around and tingling with the source a little, I found that by removing the hard 1024 limit was a viable solution in this particular project.  My benchmark number didn’t change after the hack and I was able to get all authors in the facet.  I can understand the hard limit was to prevent overloading the facet with too many values that could kill performance or worst, cause out of memory issue.   In fact, 1024 is a pretty high limit for a single field.  In my experience, I hardly found any multi-value field can reach anywhere near that number on a single record.  However, I just encountered such oddity.  Luckily, the number of publications with more than 1024 authors is negligible compare to the million of publications in my index.  So, this little hack didn’t produce any adverse effect for me.

Couple ways to do this hack.  One is to modify BigNestedIntArray.MAX_ITEMS directly with your desire max value.  Another way is to modify the following 3 files.

• MultiValueFacetDataCache.java
• MultiValueFacetHandler.java
• BigNestedIntArray.java

Look for the following code:

_maxItems = Math.min(maxItems, BigNestedIntArray.MAX_ITEMS);

and replace it with:

_maxItems = maxItems;

And of course, call setMaxItems on your MultiValueFacetHandler to set the desired max value.

Just want to iterate that this hack has only been tested in one particular project.  It may create instability in your project if you have a lot of records with 1024+ facet values.  I was told that the array’s key is 11 bit.  So, there is a max value at 2048.  Having more than 2048 value will likely cause an exception.  Memory consumption can also become a real issue.  Consider yourself warned.