Install hadoop cluster lzo

Main steps:

1. Install and update GCC and ant (if the system has been installed, skip the following steps)

Yum-y install GCC gcc-C ++ Autoconf automake

Wget

Http://labs.renren.com/apache-mirror//ant/binaries/apache-ant-1.8.2-bin.tar.gz

Tar-jxvf apache-ant-1.8.2-bin.tar.bz2

Export ant_home =/usr/local/Apache-ant-1.8.2

VI/etc/profile

Source/etc/profile

Export Path = $ path: $ ant_home/bin

2. Install lzo on each node

Wget

Http://www.oberhumer.com/opensource/lzo/download/lzo-2.04.tar.gz

Tar-zxvf lzo-2.04.tar.gzcd lzo-2.04

./Configure -- enable-shared

Make & make install

Copy the lzo library file in the/usr/local/lib directory to/usr/lib (32-bit platform) or/usr/lib64 (64-bit platform)

3. Install lzo encoding/Decoder

Wget

Https://download.github.com/kevinweil-hadoop-lzo-2ad6654.tar.gz

Tar-zxvf kevinweil-hadoop-lzo-2ad6654.tar.gz

CD kevinweil-hadoop-lzo-2ad6654

Ant compile-native tar

There are errors during the installation and compilation process. Please consider installing the following two RPM packages

Wget http://packages.sw.be/lzo/lzo-devel-2.04-1.el5.rf.i386.rpm

Wget http://packages.sw.be/lzo/lzo-2.04-1.el5.rf.i386.rpm

Rpm-IVH lzo-2.04-1.el5.rf.i386.rpm

Rpm-IVH lzo-devel-2.04-1.el5.rf.i386.rpm

(Depending on the system) after compilation is successful, copy the kevinweil-hadoop-lzo-2ad6654/bulid/hadoop-lzo-0.4.14.jar to hadoop/lib

Copy the files under build/native/Linux-amd64-64/lib to hadoop/lib/native/Linux-amd64-64

4. modify the configuration file and synchronize the configuration files of each node.

Add in core-site.xml

<Property>
<Name> Io. Compression. codecs </Name>
<Value> org. Apache. hadoop. Io. Compress. gzipcodec, org. Apache. hadoop. Io. Compress. defaultcodec, Com. hadoop. Compression. lzo. lzocodec, Co
M. hadoop. Compression. lzo. lzopcodec, org. Apache. hadoop. Io. Compress. bzip2codec </value>
</Property>
<Property>
<Name> Io. Compression. codec. lzo. Class </Name>
<Value> com. hadoop. Compression. lzo. lzocodec </value>
</Property>

Add in mapred-site.xml

<Property>
<Name> mapred. Compress. Map. Output </Name>
<Value> true </value>
</Property>
<Property>
<Name> mapred. Map. Output. Compression. codec </Name>
<Value> com. hadoop. Compression. lzo. lzocodec </value>
</Property>

5. Install the lzop compression software and download it to download.pdf at http://www.lzop.org/. you can select precompiled binaries are available for download, directly configure available, or download source ,. /configure make & makeinstall. You can use the lzop file name to generate lzo compressed files and upload them to the hadoop platform. The Platform Supports file splitting.