Common problems with using Eclipse to run Hadoop 2.x mapreduce programs

1. When we write the MapReduce program and click Run on Hadoop, the Eclipse console outputs the following:

This information tells us that we did not find the Log4j.properties file. Without this file, when the program runs out of error, there is no print log, so it will be difficult to debug.

Workaround: Copy the Log4j.properties file under the $hadoop_home/etc/hadoop/directory to the MapReduce Project Src folder.

2. When executing a mapreduce program, Eclipse may report errors in heap benefits. At this point, the out directory executed by the MapReduce program has been created, but at this point the directory is empty, and we need to delete this output directory before running the program again. As shown in the following:

Analysis: First we can enter commands (java-client-xx:+unlockdiagnosticvmoptions-xx:+printflagsfinal-version | grep-i heapsize), To see the maximum heap supported by the current JDK. Then increase the heap size on this basis.

Solution: Set the VM arguments parameter in the running configuration of the currently running program, as shown in:

3.Hadoop Local library (Native Libraries) Introduction

Hadoop is developed in the Java language, but there are some requirements and actions that are not appropriate for Java, so the concept of a local library (Native Libraries) is introduced, and Hadoop can perform some operations more efficiently with local libraries.

Currently in Hadoop, the local library is applied on file compression:

• Zlib
• Gzip

With both of these compression modes, HADOOP loads the local library from the $hadoop_home/lib/native/linux-* directory by default.

If the load succeeds, the output is:

DEBUG util. Nativecodeloader-trying to load the custom-built Native-hadoop library ...
INFO util. Nativecodeloader-loaded The Native-hadoop Library

If the load fails, the output is:

INFO util. nativecodeloader-unable to load Native-hadoop library for your platform ... using Builtin-java classes where applicable

You can set whether to use a local library in the configuration file Core-site.xml of Hadoop:

<property>
<name>hadoop.native.lib</name>
<value>true</value>
<description>should Native Hadoop libraries, if present, be used.</description>
</property>

The default configuration for Hadoop is to enable the local library.

In addition, you can set the location in the environment variable to use the local library:

Export Java_library_path=/path/to/hadoop-native-libs

Sometimes you will find that the local library that comes with Hadoop does not work, and in this case you need to compile the local library yourself. Under the $hadoop_home directory, use the following command:

Ant Compile-native

After compiling, you can find the corresponding file in the $hadoop_home/build/native directory, then specify the path of the file or move the compiled file to the default directory.

Common problems with using Eclipse to run Hadoop 2.x mapreduce programs