java.lang.OutOfMemoryError: Java heap space Hadoop

標籤：大資料   雲端運算   java   

       最近，隨著大資料的興起，Java實現的Hadoop成了這個資料領域的領跑者，不管是HDFS，還是MapReduce，還是Hive都成了很熱的詞彙。也從Hadoop這個軟體本體催生了一個依存此的大資料領域，也和時下熱的不能再熱的雲端運算拉上了關係。

       於是，作為一個程式員，不得不學習新的技術和知識，來保證自己的飯碗，這真是一個極為辛苦的差事。於是，開始接觸Hadoop。結果也就是難免不出現錯誤。

       《Hadoop Beginner‘s Guide》這本的指引，進行搭建環境，發現Hadoop已經提供deb的安裝版本，於是也就省卻了許多的多餘工作，直接在ubuntu的軟體中心雙擊之後安裝即可。

        安裝完成之後嘗試執行第一個樣本程式。

        hadoop jar hadoop-examples-1.0.4.jar pi 4 1000，不幸的是出現了如下的錯誤，

[email protected]:/usr/share/hadoop$ sudo hadoop jar hadoop-examples-1.2.1.jar pi 4 1Number of Maps  = 4Samples per Map = 115/04/17 21:54:44 INFO util.NativeCodeLoader: Loaded the native-hadoop libraryWrote input for Map #0Wrote input for Map #1Wrote input for Map #2Wrote input for Map #3Starting Job15/04/17 21:54:44 INFO mapred.FileInputFormat: Total input paths to process : 415/04/17 21:54:44 INFO mapred.JobClient: Running job: job_local1032904958_000115/04/17 21:54:44 INFO mapred.LocalJobRunner: Waiting for map tasks15/04/17 21:54:44 INFO mapred.LocalJobRunner: Starting task: attempt_local1032904958_0001_m_000000_015/04/17 21:54:44 INFO util.ProcessTree: setsid exited with exit code 015/04/17 21:54:44 INFO mapred.Task:  Using ResourceCalculatorPlugin : [email protected]15/04/17 21:54:44 INFO mapred.MapTask: Processing split: file:/usr/share/hadoop/PiEstimator_TMP_3_141592654/in/part2:0+11815/04/17 21:54:44 INFO mapred.MapTask: numReduceTasks: 115/04/17 21:54:45 INFO mapred.MapTask: io.sort.mb = 10015/04/17 21:54:45 INFO mapred.LocalJobRunner: Starting task: attempt_local1032904958_0001_m_000001_015/04/17 21:54:45 INFO mapred.Task:  Using ResourceCalculatorPlugin : [email protected]15/04/17 21:54:45 INFO mapred.MapTask: Processing split: file:/usr/share/hadoop/PiEstimator_TMP_3_141592654/in/part1:0+11815/04/17 21:54:45 INFO mapred.MapTask: numReduceTasks: 115/04/17 21:54:45 INFO mapred.MapTask: io.sort.mb = 10015/04/17 21:54:45 INFO mapred.LocalJobRunner: Starting task: attempt_local1032904958_0001_m_000002_015/04/17 21:54:45 INFO mapred.Task:  Using ResourceCalculatorPlugin : [email protected]15/04/17 21:54:45 INFO mapred.MapTask: Processing split: file:/usr/share/hadoop/PiEstimator_TMP_3_141592654/in/part0:0+11815/04/17 21:54:45 INFO mapred.MapTask: numReduceTasks: 115/04/17 21:54:45 INFO mapred.MapTask: io.sort.mb = 10015/04/17 21:54:45 INFO mapred.LocalJobRunner: Starting task: attempt_local1032904958_0001_m_000003_015/04/17 21:54:45 INFO mapred.Task:  Using ResourceCalculatorPlugin : [email protected]15/04/17 21:54:45 INFO mapred.MapTask: Processing split: file:/usr/share/hadoop/PiEstimator_TMP_3_141592654/in/part3:0+11815/04/17 21:54:45 INFO mapred.MapTask: numReduceTasks: 115/04/17 21:54:45 INFO mapred.MapTask: io.sort.mb = 10015/04/17 21:54:45 INFO mapred.LocalJobRunner: Map task executor complete.15/04/17 21:54:45 WARN mapred.LocalJobRunner: job_local1032904958_0001java.lang.Exception: java.lang.OutOfMemoryError: Java heap spaceat org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:354)Caused by: java.lang.OutOfMemoryError: Java heap spaceat org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:954)at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:422)at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366)at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:223)at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)at java.util.concurrent.FutureTask.run(FutureTask.java:266)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)at java.lang.Thread.run(Thread.java:744)15/04/17 21:54:45 INFO mapred.JobClient:  map 0% reduce 0%15/04/17 21:54:45 INFO mapred.JobClient: Job complete: job_local1032904958_000115/04/17 21:54:45 INFO mapred.JobClient: Counters: 015/04/17 21:54:45 INFO mapred.JobClient: Job Failed: NAjava.io.IOException: Job failed!at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1357)at org.apache.hadoop.examples.PiEstimator.estimate(PiEstimator.java:297)at org.apache.hadoop.examples.PiEstimator.run(PiEstimator.java:342)at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)at org.apache.hadoop.examples.PiEstimator.main(PiEstimator.java:351)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:483)at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:483)at org.apache.hadoop.util.RunJar.main(RunJar.java:160)[email protected]:/usr/share/hadoop$ sudo hadoop jar hadoop-examples-1.2.1.jar pi 1 1Number of Maps  = 1Samples per Map = 115/04/17 21:54:51 INFO util.NativeCodeLoader: Loaded the native-hadoop libraryWrote input for Map #0Starting Job15/04/17 21:54:51 INFO mapred.FileInputFormat: Total input paths to process : 115/04/17 21:54:51 INFO mapred.JobClient: Running job: job_local406287877_000115/04/17 21:54:52 INFO mapred.LocalJobRunner: Waiting for map tasks15/04/17 21:54:52 INFO mapred.LocalJobRunner: Starting task: attempt_local406287877_0001_m_000000_015/04/17 21:54:52 INFO util.ProcessTree: setsid exited with exit code 015/04/17 21:54:52 INFO mapred.Task:  Using ResourceCalculatorPlugin : [email protected]15/04/17 21:54:52 INFO mapred.MapTask: Processing split: file:/usr/share/hadoop/PiEstimator_TMP_3_141592654/in/part0:0+11815/04/17 21:54:52 INFO mapred.MapTask: numReduceTasks: 115/04/17 21:54:52 INFO mapred.MapTask: io.sort.mb = 10015/04/17 21:54:52 INFO mapred.LocalJobRunner: Map task executor complete.15/04/17 21:54:52 WARN mapred.LocalJobRunner: job_local406287877_0001java.lang.Exception: java.lang.OutOfMemoryError: Java heap spaceat org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:354)Caused by: java.lang.OutOfMemoryError: Java heap spaceat org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:954)at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:422)at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366)at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:223)at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)at java.util.concurrent.FutureTask.run(FutureTask.java:266)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)at java.lang.Thread.run(Thread.java:744)15/04/17 21:54:52 INFO mapred.JobClient:  map 0% reduce 0%15/04/17 21:54:52 INFO mapred.JobClient: Job complete: job_local406287877_000115/04/17 21:54:52 INFO mapred.JobClient: Counters: 015/04/17 21:54:52 INFO mapred.JobClient: Job Failed: NAjava.io.IOException: Job failed!at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1357)at org.apache.hadoop.examples.PiEstimator.estimate(PiEstimator.java:297)at org.apache.hadoop.examples.PiEstimator.run(PiEstimator.java:342)at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)at org.apache.hadoop.examples.PiEstimator.main(PiEstimator.java:351)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:483)at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:483)at org.apache.hadoop.util.RunJar.main(RunJar.java:160)

        查閱了忘了無數的資料，都沒有作用，通過千方百計的尋找，調試，發現關鍵在於/etc/hadoop/hadoop-env.sh檔案分配的記憶體不夠，導致記憶體不夠的錯誤。現將修改之後的檔案黏貼如下：

# Set Hadoop-specific environment variables here.# The only required environment variable is JAVA_HOME. All others are# optional. When running a distributed configuration it is best to# set JAVA_HOME in this file, so that it is correctly defined on# remote nodes.# The java implementation to use.<span style="color:#ff0000;">export JAVA_HOME=/usr/lib/jvm/jdk1.8.0</span>export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop"}# The maximum amount of heap to use, in MB. Default is 1000.<span style="color:#ff0000;">export HADOOP_HEAPSIZE=100</span>#export HADOOP_NAMENODE_INIT_HEAPSIZE=""# Extra Java runtime options. Empty by default.export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true $HADOOP_CLIENT_OPTS"# Command specific options appended to HADOOP_OPTS when specifiedexport HADOOP_NAMENODE_OPTS="-Dhadoop.security.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT $HADOOP_NAMENODE_OPTS"HADOOP_JOBTRACKER_OPTS="-Dhadoop.security.logger=INFO,DRFAS -Dmapred.audit.logger=INFO,MRAUDIT -Dhadoop.mapreduce.jobsummary.logger=INFO,JSA $HADOOP_JOBTRACKER_OPTS"HADOOP_TASKTRACKER_OPTS="-Dhadoop.security.logger=ERROR,console -Dmapred.audit.logger=ERROR,console $HADOOP_TASKTRACKER_OPTS"HADOOP_DATANODE_OPTS="-Dhadoop.security.logger=ERROR,DRFAS $HADOOP_DATANODE_OPTS"export HADOOP_SECONDARYNAMENODE_OPTS="-Dhadoop.security.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT $HADOOP_SECONDARYNAMENODE_OPTS"# The following applies to multiple commands (fs, dfs, fsck, distcp etc)<span style="color:#ff0000;">export HADOOP_CLIENT_OPTS="-Xmx200m $HADOOP_CLIENT_OPTS"</span>#HADOOP_JAVA_PLATFORM_OPTS="-XX:-UsePerfData $HADOOP_JAVA_PLATFORM_OPTS"# On secure datanodes, user to run the datanode as after dropping privilegesexport HADOOP_SECURE_DN_USER=# Where log files are stored. $HADOOP_HOME/logs by default.export HADOOP_LOG_DIR=/var/log/hadoop/$USER# Where log files are stored in the secure data environment.export HADOOP_SECURE_DN_LOG_DIR=/var/log/hadoop/# The directory where pid files are stored. /tmp by default.export HADOOP_PID_DIR=/var/run/hadoopexport HADOOP_SECURE_DN_PID_DIR=/var/run/hadoop# A string representing this instance of hadoop. $USER by default.export HADOOP_IDENT_STRING=$USER

        修改完成之後使之立即生效，執行source hadoop-env.sh即可。

java.lang.OutOfMemoryError: Java heap space Hadoop