Hadoop Local Library

Objective

Given the performance problems and the lack of some Java class libraries, Hadoop provides its own local implementation for some components. These components are stored in a separate dynamically linked library of Hadoop. This library is called libhadoop.so on the Nix platform. This article mainly describes how to use the local library and how to build the local library.

Component

Hadoop now has the following compression codecs local components:

zlib gzip Lzo

In the above components, the Lzo and gzip compression codecs must use the Hadoop local library to run.

Use Method

The use of the Hadoop local library is simple:

look at the supported platform. Download the pre-built 32-bit i386 Linux local Hadoop library (which can be found in the Lib/native directory of the Hadoop release) or build these libraries yourself. Make sure your platform is installed with zlib-1.2 above or lzo2.0 or both installed (depending on your needs).

The Bin/hadoop script confirms that the Hadoop local library is included in the library path through the system Properties-djava.library.path=<path>.

Check the Hadoop log file to see if the Hadoop Library is normal and will normally see:

DEBUG util. Nativecodeloader-trying to load the custom-built Native-hadoop library ...
INFO util. Nativecodeloader-loaded The Native-hadoop Library

If an error occurs, you will see:

INFO util. nativecodeloader-unable to load Native-hadoop library for your platform ... using Builtin-java classes where applicable

Supported Platforms

The Hadoop local library only supports Nix platforms and is already widely used on GNU platforms, but does not support Cygwin and Mac OS x.

GNU releases that have been tested:

Rhel4/fedora Ubuntu Gentoo

On the platform above, the 32/64 Hadoop local library works correctly with 32/64-bit JVMs.

builds Hadoop local library

The Hadoop local library is written using ANSI C and is built using the GNU Autotools tool Chain (autoconf, Autoheader, Automake, AutoScan, Libtool). That is to say, the platform that builds Hadoop library needs to have standard C compiler and GNU Autotools tool chain. See supported platforms.

Software packages you may need on your target platform:

C compiler (e.g. GNU C Compiler) GNU Autools tool chain: autoconf, Automake, Libtool zlib Development Kit (Stable version &gt;= 1.2.0) LZO Development package (Stable Version &gt;= 2.0)

If the above prerequisites are met, you can use the Build.xml file and set the compile.native to true so that the Hadoop local library can be generated:

$ ant-dcompile.native=true <target>

Because not all users need the Hadoop local library, Hadoop does not generate the library by default.

You can view the newly generated Hadoop local library in the following path:

$ build/native/<platform>/lib

Where <platform> is a combined ${os.name}-${os.arch}-${sun.arch.data.model} for the following system properties; linux-i386-32.

Note that the zlib and Lzo development packs must be installed on the target platform where the Hadoop local library is generated, but if you only want to use one of them, it is sufficient to install either of them at deployment time. When you build and deploy the Hadoop local library on the target platform, you need to select the corresponding 32/64-bit Zlib/lzo package based on the 32/64-bit JVM. Loading local libraries using Distributedcache

Users can load a local shared library through Distributedcache and distribute and establish symbolic links to the library files.

This example describes how to distribute a library file and load the library file from the Map/reduce task.

first copy the library file to HDFs.


bin/hadoop fs-copyfromlocal mylib.so.1/libraries/mylib.so.1 Startup job contains the following code:


distributedcache.createsymlink (conf);


distributedcache.addcachefile ("hdfs://host:port/libraries/mylib.so.1#mylib.so", conf); The Map/reduce task contains the following code:


system.loadlibrary ("mylib.so");