Kettle_ Memory Overflow Error

Original works, from the "Blue Blog" blog, Welcome to reprint, please be sure to indicate the following sources, otherwise, the legal responsibility to pursue copyright.

Deep Blue Blog:http://blog.csdn.net/huangyanlong/article/details/42453831

Kettle Memory Overflow Error Resolution

Environment:
Source-Side database: Oracle 10G R2
Target-side database: Oracle 11G R2
Kettle Version: 5.0.1-stable

Error:
When extracting large data scale, error, log information as follows:

2015/01/05 11:27:42-spoon-conversion is already open. 2015/01/05 11:27:42-spoon-opening conversion [TABLE_BWWN] ... 2015/01/05 11:27:42-spoon-Start the conversion. 2015/01/05 11:27:42-TABLE_BWWN-In order to convert the release of the patch start [table_bwwn]2015/01/05 11:27:44-max. 0-finished reading query, closing connection.2015/01/05 11:27:44-max.0-Finish processing (I=1, o=0, r=0, W=1, u=0, e=02015/01/05 12:25:08-Insert/update. 0-ERROR (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by Buildguy): unexpectederror:2015/0 1/05 12:25:08-spoon-error (Version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by Buildguy): An unexpected error occurred in Spoon:pro Bable Cause: Please close the other Spoon window before stopping spoon! 2015/01/05 12:25:08-spoon-java Heap space2015/01/05 12:25:08-insert/update. 0-ERROR (version 5.0.1-stable, build 1 from 2 013-11-15_16-08-58 by Buildguy): Java.lang.OutOfMemoryError:Java heap space2015/01/05 12:25:08-insert/update. 0-at Oracle. Jdbc.driver.OraclePreparedStatement.bindBytesAsStream (oraclepreparedstatement.java:6713) 2015/01/05 12:25:08-insertion/ Update. 0-at Oracle.jdbc.drivEr. Oraclepreparedstatement.setbytesinternal (oraclepreparedstatement.java:6700) 2015/01/05 12:25:08-insert/update. 0-at Oracle.jdbc.driver.OraclePreparedStatement.setBytes (oraclepreparedstatement.java:6660) 2015/01/05 12:25:08-insertion/ Updated. 0-at Oracle.jdbc.driver.OraclePreparedStatementWrapper.setBytes (oraclepreparedstatementwrapper.java:150) 2015 /01/05 12:25:08-Insert/update. 0-at Org.pentaho.di.core.row.value.ValueMetaBase.setPreparedStatementValue ( valuemetabase.java:4704) 2015/01/05 12:25:08-insert/update. 0-at Org.pentaho.di.core.database.Database.setValue ( database.java:928) 2015/01/05 12:25:08-insert/update. 0-at org.pentaho.di.core.database.Database.setValues (Database.java : 943) 2015/01/05 12:25:08-insert/update. 0-at Org.pentaho.di.core.database.Database.setValuesInsert (database.java:888) 2015 /01/05 12:25:08-Insert/update. 0-at org.pentaho.di.trans.steps.insertupdate.InsertUpdate.lookupValues (Insertupdate.java : 113) 2015/01/05 12:25:08-insert/update. 0-at org.pentaho.di.trans.steps.insertupdate.InsertUpDate.processrow (insertupdate.java:301) 2015/01/05 12:25:08-insert/update. 0-at Org.pentaho.di.trans.step.RunThread.run ( RUNTHREAD.JAVA:60) 2015/01/05 12:25:08-insert/update. 0-at java.lang.Thread.run (Unknown Source) 2015/01/05 12:25:08-spoon- ERROR (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by Buildguy): Java.lang.OutOfMemoryError:Java heap Space20 15/01/05 12:25:08-spoon-at java.util.Arrays.copyOf (Unknown Source) 2015/01/05 12:25:08-spoon-at Java.lang.StringC Oding.safetrim (Unknown source) 2015/01/05 12:25:08-spoon-at java.lang.stringcoding.access$100 (Unknown source) 2015/ 01/05 12:25:08-spoon-at Java.lang.stringcoding$stringdecoder.decode (Unknown Source) 2015/01/05 12:25:08-spoon-at Java.lang.StringCoding.decode (Unknown Source) 2015/01/05 12:25:08-spoon-at Java.lang.StringCoding.decode (Unknown Source) 2015/01/05 12:25:08-spoon-at java.lang.string.<init> (Unknown source) 2015/01/05 12:25:08-spoon-at ja Va.lang.string.<init> (Unknown SouRCE) 2015/01/05 12:25:08-spoon-at org.pentaho.di.core.row.value.ValueMetaBase.convertBinaryStringToString ( valuemetabase.java:1105) 2015/01/05 12:25:08-spoon-at org.pentaho.di.core.row.value.ValueMetaBase.getString ( valuemetabase.java:1452) 2015/01/05 12:25:08-spoon-at org.pentaho.di.core.row.RowMeta.getString (rowmeta.java:230) 2015/01/05 12:25:08-spoon-at Org.pentaho.di.ui.spoon.trans.TransPreviewDelegate.showPreviewGrid ( transpreviewdelegate.java:302) 2015/01/05 12:25:08-spoon-at Org.pentaho.di.ui.spoon.trans.TransPreviewDelegate.refreshView (transpreviewdelegate.java:268) 2015/01/05 12:25:08-spoon-at org.pentaho.di.ui.spoon.trans.transpreviewdelegate$1.widgetselected (TransPreviewDelegate.java : 188) 2015/01/05 12:25:08-spoon-at org.eclipse.swt.widgets.TypedListener.handleEvent (Unknown Source) 2015/01/05 12:25:08-spoon-at org.eclipse.swt.widgets.EventTable.sendEvent (Unknown Source) 2015/01/05 12:25:08-spoon-at ORG.E Clipse.swt.widgets.Widget.sendEvent (UnkNown source) 2015/01/05 12:25:08-spoon-at org.eclipse.swt.widgets.Widget.sendEvent (Unknown source) 2015/01/05 12:25:08-spoon-at org.eclipse.swt.widgets.Widget.sendEvent (Unknown Source) 2015/01/05 12:25:08-spoon-at org.eclip Se.swt.widgets.Widget.notifyListeners (Unknown Source) 2015/01/05 12:25:08-spoon-at Org.eclipse.swt.custom.CTabFolder.setSelection (Unknown Source) 2015/01/05 12:25:08-spoon-at Org.eclipse.swt.custom.CTabFolder.onMouse (Unknown Source) 2015/01/05 12:25:08-spoon-at Org.eclipse.swt.custom.ctabfolder$1.handleevent (Unknown Source) 2015/01/05 12:25:08-spoon-at Org.eclipse.swt.widgets.EventTable.sendEvent (Unknown Source) 2015/01/05 12:25:08-spoon-at Org.eclipse.swt.widgets.Widget.sendEvent (Unknown Source) 2015/01/05 12:25:08-spoon-at Org.eclipse.swt.widgets.Display.runDeferredEvents (Unknown Source) 2015/01/05 12:25:08-spoon-at Org.eclipse.swt.widgets.Display.readAndDispatch (Unknown Source) 2015/01/05 12:25:08-spoon-at Org.pentaho.di. Ui.spoon.Spoon.readAndDispatch (spoon.java:1227) 2015/01/05 12:25:08-spoon-at Org.pentaho.di.ui.spoon.Spoon.waitForDispose (spoon.java:7368) 2015/01/05 12:25:08-spoon-at Org.pentaho.di.ui.spoon.Spoon.start (spoon.java:8673) 2015/01/05 12:25:08-spoon-at Org.pentaho.di.ui.spoon.Spoon.main (spoon.java:625) 2015/01/05 12:25:08-spoon-at SUN.REFLECT.NATIVEMETHODACCESSORIMPL.INVOKE0 (Native Method) 2015/01/05 12:25:09-insert/update. 0-Finish processing (i=20575, o=20574, r=20  575, w=20574, u=0, e=12015/01/05 12:25:09-table_bwwn-table_bwwn2015/01/05 12:25:09-table_bwwn-table_bwwn2015/01/05 12:25:09-Table input. 0-error (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by Buildguy): unexpected error2015/01/0 5 12:25:09-Table input. 0-error (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by Buildguy): Org.pentaho.di.core.exce Ption. kettledatabaseexception:2015/01/05 12:25:09-table input. 0-couldn ' t get row from result set2015/01/05 12:25:09-table input. 0-ora-0 3111: Communication channel received interrupt 2015/01/05 12:25:09-table outputInto. 0-2015/01/05 12:25:09-table input. 0-at Org.pentaho.di.core.database.Database.getRow (database.java:2302) 2015/01/05 12:25 : 09-table input. 0-at Org.pentaho.di.core.database.Database.getRow (database.java:2270) 2015/01/05 12:25:09-table input. 0-at org.pe Ntaho.di.trans.steps.tableinput.TableInput.processRow (tableinput.java:153) 2015/01/05 12:25:09-table input. 0-at Org.pentaho.di.trans.step.RunThread.run (runthread.java:60) 2015/01/05 12:25:09-table input. 0-at Java.lang.Thread.run ( Unknown Source) 2015/01/05 12:25:09-table input. 0-caused by:java.sql.sqlexception:ora-03111: Communication channel received interrupt 2015/01/05 12:25:09-table output In. 0-at oracle.jdbc.driver.T4CTTIoer.processError (t4cttioer.java:439) 2015/01/05 12:25:09-table input. 0-at Oracle.jdbc.driv Er. T4cttioer.processerror (t4cttioer.java:395) 2015/01/05 12:25:09-table input. 0-at Oracle.jdbc.driver.T4C8Oall.processError ( t4c8oall.java:802) 2015/01/05 12:25:09-table input. 0-at oracle.jdbc.driver.T4CTTIfun.receive (t4cttifun.java:436) 2015/01/ 05 12:25:09-Table input. 0-at Oracle.jdbc.driver.T4CTTIfun.doRPC (t4cttifun.java:186) 2015/01/05 12:25:09-table input. 0-at Oracle.jdbc.driver.T4C8Oall.doOALL (t4c8oall.java:521) 2015/01/ 05 12:25:09-Table input. 0-at Oracle.jdbc.driver.T4CPreparedStatement.doOall8 (t4cpreparedstatement.java:205) 2015/01/05 12:25:09-table input. 0-at Oracle.jdbc.driver.T4CPreparedStatement.fetch (t4cpreparedstatement.java:1082) 2015/01/05 12:25:09-table input. 0-at Oracle.jdbc.driver.OracleResultSetImpl.close_or_fetch_from_next (oracleresultsetimpl.java:369) 2015/01/05 12:25:09-table input. 0-at oracle.jdbc.driver.OracleResultSetImpl.next (oracleresultsetimpl.java:273) 2015/01/05 12:25:09-table input. 0-at Org.pentaho.di.core.database.Database.getRow (database.java:2290) 2015/01/05 12:25:09-table input. 0-... 4 more2015/01/05 12:25:09-table input. 0-finished reading query, closing connection.2015/01/05 12:25:09-table input. 0-Finish Processing (i=20580 , O=0, R=1, w=20579, u=0, e=12015/01/05 12:25:09-table_bwwn-error (version 5.0.1-stable, build 1 from 2013-11-15_16-08 -58 by Buildguy): Error detected!2015/01/05 12:25:09-spoon-turnChange to complete!! 2015/01/05 12:25:09-table_bwwn-error (Version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by Buildguy): Error detected!201 5/01/05 12:25:09-table_bwwn-error (Version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by Buildguy): Error detected!2015/0 1/05 12:25:09-table_bwwn-table_bwwn2015/01/05 12:25:09-TABLE_BWWN-TABLE_BWWN

Workaround:
Locate the Spoon.bat file in the Kettle directory, and locate the following section:

REM ******************************************************************rem * * Set java Runtime options                                     **rem * * Change 512m to higher values in case you run out of memory   **rem * * or set the pentaho_di_java_options environment Var iable      **rem ******************************************************************if "%PENTAHO_DI_JAVA_OPTIONS%" = = "" Set pentaho_di_java_options= "-xmx512m" "-xx:maxpermsize=256m" set opt=%pentaho_di_java_options% "- djava.library.path=%libspath% ""-dkettle_home=%kettle_home% ""-dkettle_repository=%kettle_repository% ""-DKETTLE_ user=%kettle_user% ""-dkettle_password=%kettle_password% ""-dkettle_plugin_packages=%kettle_plugin_packages% ""- dkettle_log_size_limit=%kettle_log_size_limit% ""-dkettle_jndi_root=%kettle_jndi_root% "

Modify the parameters to increase the memory value (in multiples of 256), as follows:

REM ******************************************************************rem * * Set java Runtime options                                     **rem * * Change 512m to higher values in case you run out of memory   **rem * * or set the pentaho_di_java_options environment Var iable      **rem ******************************************************************if "%PENTAHO_DI_JAVA_OPTIONS%" = = "" Set pentaho_di_java_options= "-xmx1024m" "-xx:maxpermsize=2048m" set opt=%pentaho_di_java_options% "- djava.library.path=%libspath% ""-dkettle_home=%kettle_home% ""-dkettle_repository=%kettle_repository% ""-DKETTLE_ user=%kettle_user% ""-dkettle_password=%kettle_password% ""-dkettle_plugin_packages=%kettle_plugin_packages% ""- dkettle_log_size_limit=%kettle_log_size_limit% ""-dkettle_jndi_root=%kettle_jndi_root% "

Re-executing the kettle extracts the data and no longer reports a memory overflow error.

Summary:
When using kettle, it is very easy to have memory overflow when large objects are encountered, and the preset Java stack size needs to be adjusted according to the actual hardware configuration.

Original works, from the "Blue Blog" blog, Welcome to reprint, please be sure to indicate the following sources, otherwise, the legal responsibility to pursue copyright.

Deep Blue Blog:http://blog.csdn.net/huangyanlong/article/details/42453831

Kettle_ Memory Overflow Error