SOLR dataimporthandler import MySQL memory overflow.

Recently, we are ready to import more than 19 million data into SOLR, with only 1 million or 2 million of the previous test data and full import without task issues. However, with 19 million data, SOLR reported a memory exception and the entire tomcat could not be used. I found that I gave Tomcat the largest heap of memory 512M. I think, maybe the memory is too small, so I changed the memory size, 1024M (set java_opts=-server-xms1024m-xmx1024m-xx:permsize=128m-xx:maxpermsize=256m). The result, or a memory overflow. I think it is not a problem with Tomcat memory (of course, if your memory is large enough, it is possible). Problem, should be out in SOLR's Dataimporthandler fetch data that block. SOLR in addition to memory overflow errors, there are other exceptions:

　　

Full Import failed:java.lang.RuntimeException:java.lang.RuntimeException: Org.apache.solr.handler.dataimport.DataImportHandlerException:Unable to execute Query:select * from POI processing Document # 1at Org.apache.solr.handler.dataimport.DocBuilder.execute (docbuilder.java:278) at Org.apache.solr.handler.dataimport.DataImporter.doFullImport (dataimporter.java:411) at ORG.APACHE.SOLR.HANDLER.DATAIMPORT.DATAIMPORTER.RUNCMD (dataimporter.java:483) at Org.apache.solr.handler.dataimport.dataimporter$1.run (dataimporter.java:464) caused by: Java.lang.RuntimeException:org.apache.solr.handler.dataimport.DataImportHandlerException:Unable to execute query: SELECT * from POI processing Document # 1at Org.apache.solr.handler.dataimport.DocBuilder.buildDocument ( docbuilder.java:418) at Org.apache.solr.handler.dataimport.DocBuilder.doFullDump (docbuilder.java:331) at Org.apache.solr.handler.dataimport.DocBuilder.execute (docbuilder.java:239) ... 3 morecaused By:org.apache.solr.handler.dataimport.DataImportHandlerexception:unable to execute Query:select * from POI processing Document # 1at Org.apache.solr.handler.dataimport. Dataimporthandlerexception.wrapandthrow (dataimporthandlerexception.java:71) at Org.apache.solr.handler.dataimport.jdbcdatasource$resultsetiterator.<init> (JdbcDataSource.java:281) at Org.apache.solr.handler.dataimport.JdbcDataSource.getData (jdbcdatasource.java:238) at Org.apache.solr.handler.dataimport.JdbcDataSource.getData (jdbcdatasource.java:42) at Org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery (sqlentityprocessor.java:59) at Org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow (sqlentityprocessor.java:73) at Org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow (entityprocessorwrapper.java:243) at Org.apache.solr.handler.dataimport.DocBuilder.buildDocument (docbuilder.java:477) at Org.apache.solr.handler.dataimport.DocBuilder.buildDocument (docbuilder.java:416) ... 5 morecaused By:com.mysql.jdbc.exceptions.jdbc4.CommunicationsExceptIon:communications Link Failurethe Last packet successfully received by the server was 395,021 milliseconds ago. The last packet sent successfully to the server was 395,021 milliseconds ago.at Sun.reflect.NativeConstructorAccessorImpl. NewInstance0 (Native Method) at Sun.reflect.NativeConstructorAccessorImpl.newInstance ( nativeconstructoraccessorimpl.java:57) at Sun.reflect.DelegatingConstructorAccessorImpl.newInstance ( delegatingconstructoraccessorimpl.java:45) at Java.lang.reflect.Constructor.newInstance (constructor.java:526) at Com.mysql.jdbc.Util.handleNewInstance (util.java:406) at Com.mysql.jdbc.SQLError.createCommunicationsException ( sqlerror.java:1074) at Com.mysql.jdbc.MysqlIO.nextRowFast (mysqlio.java:1653) at Com.mysql.jdbc.MysqlIO.nextRow ( mysqlio.java:1409) at Com.mysql.jdbc.MysqlIO.readSingleRowSet (mysqlio.java:2883) at Com.mysql.jdbc.MysqlIO.getResultSet (mysqlio.java:476) at Com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate ( mysqlio.java:2576) at Com.mysql.jdbc.MysqlIO.readAllResults (Mysqlio.java:1757) at Com.mysql.jdbc.MysqlIO.sqlQueryDirect (mysqlio.java:2167) at Com.mysql.jdbc.ConnectionImpl.execSQL (connectionimpl.java:2637) at Com.mysql.jdbc.ConnectionImpl.execSQL ( connectionimpl.java:2566) at Com.mysql.jdbc.StatementImpl.execute (statementimpl.java:782) at Com.mysql.jdbc.StatementImpl.execute (statementimpl.java:625) at Org.apache.solr.handler.dataimport.jdbcdatasource$resultsetiterator.<init> (JdbcDataSource.java:274) ... morecaused By:java.io.EOFException:Can Not read response from server. Expected to read the bytes, read bytes before connection was unexpectedly lost.at com.mysql.jdbc.MysqlIO.readFully (MySQL io.java:2497) at Com.mysql.jdbc.MysqlIO.nextRowFast (mysqlio.java:1634) ... More

I searched the Solr wiki and learned about the setbatchsize of SOLR JDBC when fetching data. By default, it puts a lot of data into memory, which is why my memory stays high and then causes memory overflow. So, I modified the Dataimporthandler configuration file: batchsize= "-1" to resolve memory overflow

The configuration file is modified as follows:

<dataSource driver="Com.mysql.jdbc.Driver" url="Jdbc:mysql://localhost:3306/world " user="root" password="root" batchsize="1"/>

Restart a SOLR core, re-import, problem solving.

　　

　　

SOLR dataimporthandler import MySQL memory overflow.