Hadoop on Mac with IntelliJ IDEA-5 solving Java heap space problems

This article describes the resolution process for job recalculation when submitting jobs in CentOS 6.5 to Hadoop 1.2.1 encountered Error:java heap space errors in the reduce phase. Workaround for Linux, Mac os X, and Windows operating systems.

Environment: Mac OS X 10.9.5, IntelliJ idea 13.1.4, Hadoop 1.2.1

Hadoop is placed in a virtual machine, and the host is connected via SSH, IDE and data files on the host. Idea itself runs on JDK 1.8,idea engineering and Hadoop using JDK 1.6.

When dealing with the patent data referenced in the Hadoop in Action Chapter 4th, there was a shortage of jvm heap space, as shown in.

It appears that the Java Virtual machine heap memory is too small to cause. [1] can be resolved as follows.

mode one , temporary processing, before each commit task, execute the statement export hadoop_opts=[size], such as set the JVM heap size to 4G size, export hadoop_opts= "-xmx4096m" Execute the Hadoop command again.

mode two , permanent processing, configure the Mapred-site.xml file in the Conf directory under the Hadoop directory, add the following, and restart Hadoop to take effect.

 1  <  property  >  2   >  mapred.child.java.opts</  name  >  3   > -xmx4096m</ value  >  4  </ property  >  

description , the heap settings are generally tried from a smaller value, gradually increasing, when the Java heap space hint is no longer present. 32-bit systems, generally limited to memory limit on 1.5g~2g;64-bit operating systems.

For JVM tuning information, refer to JVM tuning summary-xms-xmx-xmn-xss.

Reference documents

[1]http://stackoverflow.com/questions/15609909/error-java-heap-space

Hadoop on Mac with IntelliJ IDEA-5 solving Java heap space problems