Forum

You must be logged in to post Login Register

Search 
Search Forums:


 




DataImportHandler Runs Out of Memory on Large Table
Read original blog post

UserPost

6:53 pm
May 16, 2009


admin

Admin

posts 17

1

One DataImportHandler(DIH) configuration people may overlook is the batchSize attribute.  If you start your JVM with enough memory to store the entire table, you won’t even need to set batchSize at all.  batchSize basically tells DIH to call setFetchSize through JDBC to bring back certain number of records at once.  If you use MySQL, you may still run out of memory even when you set the batchSize attribute.  That’s due to a limitation in MySQL’s drive where the setting is ignored.  The workaround is setting batchSize to “-1″.  This will pass Integer.MIN_VALUE to MySQL as fetch size and prevent the driver from running out of memory.

<dataSource driver="org.gjt.mm.mysql.Driver" url="jdbc:mysql://localhost/db"
user="root" password="root" batchSize="-1"/>

Read original blog post

Search 

About the MySolr forum

Most Users Ever Online:

9


Currently Online:

4 Guests

Forum Stats:

Groups: 2

Forums: 2

Topics: 9

Posts: 22

Membership:

There are 8 Members

There has been 1 Guest

There is 1 Admin

There is 1 Moderator

Top Posters:

mojo – 6

sturlese – 6

fudge – 1

blah – 1

egrobler – 1

phat – 0

Administrators: admin (17 Posts)

Moderators: mysolr (0 Posts)