Solr Dataimporthandler imports MySQL memory overflow ., Solr memory overflow

Solr Dataimporthandler imports MySQL memory overflow ., Solr memory overflow

Recently, we are preparing to import more than 19 million of the data into Solr. In the past, only 1 millions or 2 millions of the test data was used. There is no task problem in full import. However, when the data is changed to 19 million, solr reports a Memory exception (java. lang. OutOfMemoryError: GC overhead limit exceeded), and the entire tomcat cannot be used. I found that the maximum heap memory I gave tomcat was 512 MB. In my opinion, the memory may be too small, so I increased the memory by 1024 M (set JAVA_OPTS =-server-Xms1024m-Xmx1024m-XX: PermSize = 128 M-XX: maxPermSize = 256 M ). As a result, the memory overflows. I think it is no longer a problem with tomcat memory (of course, it is feasible if your memory is large enough ). The problem is that the solr dataimporthandler obtains the data. In addition to memory overflow errors, solr has other exceptions:

　　

Full Import failed:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to execute query: select * from POI Processing Document # 1at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:278)at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:411)at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:483)at org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:464)Caused by: java.lang.RuntimeException: org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to execute query: select * from POI Processing Document # 1at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:418)at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:331)at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:239)... 3 moreCaused by: org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to execute query: select * from POI Processing Document # 1at org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:71)at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:281)at org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:238)at org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:42)at org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityProcessor.java:59)at org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)at org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:243)at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:477)at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:416)... 5 moreCaused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failureThe last packet successfully received from the server was 395,021 milliseconds ago.  The last packet sent successfully to the server was 395,021 milliseconds ago.at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)at java.lang.reflect.Constructor.newInstance(Constructor.java:526)at com.mysql.jdbc.Util.handleNewInstance(Util.java:406)at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1074)at com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:1653)at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1409)at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2883)at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:476)at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2576)at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1757)at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2167)at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2637)at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2566)at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:782)at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:625)at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:274)... 12 moreCaused by: java.io.EOFException: Can not read response from server. Expected to read 39 bytes, read 30 bytes before connection was unexpectedly lost.at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:2497)at com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:1634)... 23 more

I searched Solr wiki and learned about the setBatchSize Problem When solr jdbc retrieves data. By default, a lot of data is stored in the memory, which is also the cause of high memory usage and memory overflow. So I modified the dataimporthandler configuration file: batchSize = "-1" to solve memory overflow.

The configuration file is modified as follows:

<DataSource driver = "com. mysql. jdbc. driver "url =" jdbc: mysql: // localhost: 3306/world "user =" root "password =" root "batchSize ="-1 "/>

Restart a solr core, re-import, and solve the problem.