KETTLE _ memory overflow error and kettle overflow error

KETTLE _ memory overflow error and kettle overflow error

Original Works are from the blog of "Deep Blue blog". You are welcome to reprint them. Please note the following source when reprinting them. Otherwise, you will be held legally liable for copyright.

Deep Blue blog: http://blog.csdn.net/huangyanlong/article/details/42453831

 

Kettle memory overflow error Solution

Environment:
Source database: oracle 10G R2
Target Database: oracle 11G R2
Kettle version: 5.0.1-stable

Error:
An error is reported when a big data table is extracted. The log information is as follows:

11:27:42-Spoon-conversion enabled. 11:27:42-Spoon-converting [TABLE_BWWN]... 11:27:42-Spoon-start to execute the conversion. 11:27:42-TABLE_BWWN-start [TABLE_BWWN] 2015/01/05 11:27:44-max.0-Finished reading query, closing connection.2015/01/05 11:27:44-max.0-finish processing (I = 1, O = 0, R = 0, W = 1, U = 0, E = 02015/01/05 12:25:08-insert/update. 0-ERROR (version 5.0.1-stable, build 1 from 2013 -11-15_16-08-58 by buildguy): UnexpectedError: 2015/01/05 12:25:08-Spoon-ERROR (version 5.0.1-stable, build 1 from built by buildguy): An Unexpected ERROR occurs in Spoon: probable cause: close other Spoon windows before stopping spoon! 12:25:08-Spoon-Java heap space2015/01/05 12:25:08-insert/update. 0-ERROR (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by buildguy): java. lang. outOfMemoryError: Java heap space2015/01/05 12:25:08-insert/update. 0-at oracle. jdbc. driver. oraclePreparedStatement. bindBytesAsStream (OraclePreparedStatement. java: 6713) 12:25:08-insert/update. 0-at oracle. jdbc. driver. oraclePrepare DStatement. setBytesInternal (OraclePreparedStatement. java: 6700) 12:25:08-insert/update. 0-at oracle. jdbc. driver. oraclePreparedStatement. setBytes (OraclePreparedStatement. java: 6660) 12:25:08-insert/update. 0-at oracle. jdbc. driver. oraclePreparedStatementWrapper. setBytes (OraclePreparedStatementWrapper. java: 150) 12:25:08-insert/update. 0-at org. pentaho. di. core. row. value. valueMetaBas E. setPreparedStatementValue (ValueMetaBase. java: 4704) 12:25:08-insert/update. 0-at org. pentaho. di. core. database. database. setValue (Database. java: 928) 12:25:08-insert/update. 0-at org. pentaho. di. core. database. database. setValues (Database. java: 943) 12:25:08-insert/update. 0-at org. pentaho. di. core. database. database. setValuesInsert (Database. java: 888) 12:25:08-insert/update. 0- At org. pentaho. di. trans. steps. insertupdate. insertUpdate. lookupValues (InsertUpdate. java: 113) 12:25:08-insert/update. 0-at org. pentaho. di. trans. steps. insertupdate. insertUpdate. processRow (InsertUpdate. java: 301) 12:25:08-insert/update. 0-at org. pentaho. di. trans. step. runThread. run (RunThread. java: 60) 12:25:08-insert/update. 0-at java. lang. thread. run (Unknown Source) 2015/01/05 :25:0 8-Spoon-ERROR (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by buildguy): java. lang. outOfMemoryError: Java heap space2015/01/05 12:25:08-Spoon-at java. util. arrays. copyOf (Unknown Source) 2015/01/05 12:25:08-Spoon-at java. lang. stringCoding. safeTrim (Unknown Source) 2015/01/05 12:25:08-Spoon-at java. lang. stringCoding. access $100 (Unknown Source) 12:25:08-Spoon-at j Ava. lang. stringCoding $ StringDecoder. decode (Unknown Source) 2015/01/05 12:25:08-Spoon-at java. lang. stringCoding. decode (Unknown Source) 2015/01/05 12:25:08-Spoon-at java. lang. stringCoding. decode (Unknown Source) 2015/01/05 12:25:08-Spoon-at java. lang. string. <init> (Unknown Source) 2015/01/05 12:25:08-Spoon-at java. lang. string. <init> (Unknown Source) 2015/01/05 12:25:08-Spoon-at org. penta Ho. di. core. row. value. valueMetaBase. convertBinaryStringToString (ValueMetaBase. java: 1105) 2015/01/05 12:25:08-Spoon-at org. pentaho. di. core. row. value. valueMetaBase. getString (ValueMetaBase. java: 1452) 2015/01/05 12:25:08-Spoon-at org. pentaho. di. core. row. rowMeta. getString (RowMeta. java: 230) 2015/01/05 12:25:08-Spoon-at org. pentaho. di. ui. spoon. trans. transPreviewDelegate. showPreviewGrid (TransPrevi EwDelegate. java: 302) 2015/01/05 12:25:08-Spoon-at org. pentaho. di. ui. spoon. trans. transPreviewDelegate. refreshView (TransPreviewDelegate. java: 268) 2015/01/05 12:25:08-Spoon-at org. pentaho. di. ui. spoon. trans. transPreviewDelegate $1. widgetSelected (TransPreviewDelegate. java: 188) 2015/01/05 12:25:08-Spoon-at org. eclipse. swt. widgets. typedListener. handleEvent (Unknown Source) 2015/01/05 12:25:08-Spo On-at org. eclipse. swt. widgets. eventTable. sendEvent (Unknown Source) 2015/01/05 12:25:08-Spoon-at org. eclipse. swt. widgets. widget. sendEvent (Unknown Source) 2015/01/05 12:25:08-Spoon-at org. eclipse. swt. widgets. widget. sendEvent (Unknown Source) 2015/01/05 12:25:08-Spoon-at org. eclipse. swt. widgets. widget. sendEvent (Unknown Source) 2015/01/05 12:25:08-Spoon-at org. eclipse. swt. widgets. widget. n OtifyListeners (Unknown Source) 12:25:08-Spoon-at org. eclipse. swt. custom. CTabFolder. setSelection (Unknown Source) 2015/01/05 12:25:08-Spoon-at org. eclipse. swt. custom. CTabFolder. onMouse (Unknown Source) 2015/01/05 12:25:08-Spoon-at org. eclipse. swt. custom. CTabFolder $1. handleEvent (Unknown Source) 2015/01/05 12:25:08-Spoon-at org. eclipse. swt. widgets. eventTable. sendEvent (Unknown Sou Rce) 2015/01/05 12:25:08-Spoon-at org. eclipse. swt. widgets. widget. sendEvent (Unknown Source) 2015/01/05 12:25:08-Spoon-at org. eclipse. swt. widgets. display. runDeferredEvents (Unknown Source) 2015/01/05 12:25:08-Spoon-at org. eclipse. swt. widgets. display. readAndDispatch (Unknown Source) 2015/01/05 12:25:08-Spoon-at org. pentaho. di. ui. spoon. spoon. readAndDispatch (Spoon. java: 1227) 8-Spoon-at org. pentaho. di. ui. spoon. spoon. waitForDispose (Spoon. java: 7368) 2015/01/05 12:25:08-Spoon-at org. pentaho. di. ui. spoon. spoon. start (Spoon. java: 8673) 2015/01/05 12:25:08-Spoon-at org. pentaho. di. ui. spoon. spoon. main (Spoon. java: 625) 12:25:08-Spoon-at sun. reflect. nativeMethodAccessorImpl. invoke0 (Native Method) 12:25:09-insert/update. 0-complete processing (I = 20575, O = 20574, R = 2057 5, W = 20574, U = 0, E = 12015/01/05 12:25:09-TABLE_BWWN-TABLE_BWWN2015/01/05 12:25:09-TABLE_BWWN-TABLE_BWWN2015/01/05 12:25:09-table input. 0-ERROR (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by buildguy): Unexpected error2015/01/05 12:25:09-table input. 0-ERROR (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by buildguy): org. pentaho. di. core. exception. kettleDatabaseException: 2015/ 01/05 12:25:09-table input. 0-Couldn't get row from result set2015/01/05 12:25:09-table input. 0-ORA-03111: communication channel received interrupted 12:25:09-table input. 0-12:25:09-table input. 0-at org. pentaho. di. core. database. database. getRow (Database. java: 2302) 12:25:09-table input. 0-at org. pentaho. di. core. database. database. getRow (Database. java: 2270) 12:25:09-table input. 0-at org. pentaho. di. trans. steps. tableinput. TableInput. processRow (TableInput. java: 153) 12:25:09-table input. 0-at org. pentaho. di. trans. step. runThread. run (RunThread. java: 60) 12:25:09-table input. 0-at java. lang. thread. run (Unknown Source) 12:25:09-table input. 0-Caused by: java. SQL. SQLException: ORA-03111: communication channel received interrupted 12:25:09-table input. 0-at oracle. jdbc. driver. t4CTTIoer. processError (T4CTTIoer. java: 439) 12:25:09 -Table input. 0-at oracle. jdbc. driver. t4CTTIoer. processError (T4CTTIoer. java: 395) 12:25:09-table input. 0-at oracle. jdbc. driver. t4C8Oall. processError (T4C8Oall. java: 802) 12:25:09-table input. 0-at oracle. jdbc. driver. t4CTTIfun. receive (T4CTTIfun. java: 436) 12:25:09-table input. 0-at oracle. jdbc. driver. t4CTTIfun. doRPC (T4CTTIfun. java: 186) 12:25:09-table input. 0-at oracle. jdbc. driver. t4C8Oa Ll. doOALL (T4C8Oall. java: 521) 12:25:09-table input. 0-at oracle. jdbc. driver. t4CPreparedStatement. doOall8 (T4CPreparedStatement. java: 205) 12:25:09-table input. 0-at oracle. jdbc. driver. t4CPreparedStatement. fetch (T4CPreparedStatement. java: 1082) 12:25:09-table input. 0-at oracle. jdbc. driver. oracleResultSetImpl. close_or_fetch_from_next (OracleResultSetImpl. java: 369) 12:25:09-table input. 0-at oracle. jdbc. driver. oracleResultSetImpl. next (OracleResultSetImpl. java: 273) 12:25:09-table input. 0-at org. pentaho. di. core. database. database. getRow (Database. java: 2290) 12:25:09-table input. 0 -... 4 more2015/01/05 12:25:09-table input. 0-Finished reading query, closing connection.2015/01/05 12:25:09-table input. 0-complete processing (I = 20580, O = 0, R = 1, W = 20579, U = 0, E = 12015/01/05 12:25:09-TABLE_BWWN-ER ROR (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by buildguy): error detected! 12:25:09-Spoon-conversion completed !! 12:25:09-TABLE_BWWN-ERROR (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by buildguy): ERROR detected! 12:25:09-TABLE_BWWN-ERROR (version 5.0.1-stable, build 1 from 2013-11-15_16-08-58 by buildguy): ERROR detected! 12:25:09-TABLE_BWWN-TABLE_BWWN2015/01/05 12:25:09-TABLE_BWWN

Solution:
Find the Spoon. bat file in the kettle directory and find the following parts:

REM ******************************************************************REM ** Set java runtime options                                     **REM ** Change 512m to higher values in case you run out of memory   **REM ** or set the PENTAHO_DI_JAVA_OPTIONS environment variable      **REM ******************************************************************if "%PENTAHO_DI_JAVA_OPTIONS%"=="" set PENTAHO_DI_JAVA_OPTIONS="-Xmx512m" "-XX:MaxPermSize=256m"set OPT=%PENTAHO_DI_JAVA_OPTIONS% "-Djava.library.path=%LIBSPATH%" "-DKETTLE_HOME=%KETTLE_HOME%" "-DKETTLE_REPOSITORY=%KETTLE_REPOSITORY%" "-DKETTLE_USER=%KETTLE_USER%" "-DKETTLE_PASSWORD=%KETTLE_PASSWORD%" "-DKETTLE_PLUGIN_PACKAGES=%KETTLE_PLUGIN_PACKAGES%" "-DKETTLE_LOG_SIZE_LIMIT=%KETTLE_LOG_SIZE_LIMIT%" "-DKETTLE_JNDI_ROOT=%KETTLE_JNDI_ROOT%"

Modify the parameters to increase the memory value (multiples of 256) as follows:

REM ******************************************************************REM ** Set java runtime options                                     **REM ** Change 512m to higher values in case you run out of memory   **REM ** or set the PENTAHO_DI_JAVA_OPTIONS environment variable      **REM ******************************************************************if "%PENTAHO_DI_JAVA_OPTIONS%"=="" set PENTAHO_DI_JAVA_OPTIONS="-Xmx1024m" "-XX:MaxPermSize=2048m"set OPT=%PENTAHO_DI_JAVA_OPTIONS% "-Djava.library.path=%LIBSPATH%" "-DKETTLE_HOME=%KETTLE_HOME%" "-DKETTLE_REPOSITORY=%KETTLE_REPOSITORY%" "-DKETTLE_USER=%KETTLE_USER%" "-DKETTLE_PASSWORD=%KETTLE_PASSWORD%" "-DKETTLE_PLUGIN_PACKAGES=%KETTLE_PLUGIN_PACKAGES%" "-DKETTLE_LOG_SIZE_LIMIT=%KETTLE_LOG_SIZE_LIMIT%" "-DKETTLE_JNDI_ROOT=%KETTLE_JNDI_ROOT%"

Re-Execute kettle to extract data and no memory overflow error is reported.

Summary:
When kettle is used, memory overflow may easily occur when there is a large amount of data and large objects. In this case, you need to adjust the preset java stack size according to the actual hardware configuration.

 

Original Works are from the blog of "Deep Blue blog". You are welcome to reprint them. Please note the following source when reprinting them. Otherwise, you will be held legally liable for copyright.

Deep Blue blog: http://blog.csdn.net/huangyanlong/article/details/42453831