Flume Lzo compressed output to Hadoop

1. Hadoop supports Lzo compression dependencies:

The Lzo:unix/linux system does not have a Lzo library by default, so it needs to be installed, sudo yum install lzo-devel.x86_64

sudo yum install lzo.x86_64 sudo yum install lzop.x86_64

2. Prepare MAVEN,ANT,GCC, etc.

3. Compiling Hadoop-lzo

Download from Https://github.com/twitter/hadoop-lzo, unzip into directory, MVN clean package

Under Target, there are libgplcompression libraries under Hadoop-lzo-0.4.20-snapshot.jar target/native/linux-amd64-64/lib.

The jar package is copied to the Hadoop Lib directory, and the Gplcompression library is copied to Lib/native/linux-amd64-64/lib

In Hadoop client or individual node, do not need to install Lzo library, can directly copy the Lzo library to Lib/native/linux-amd64-64/lib under Hadoop

Summary: Compile Hadoop-lzo, LZO-DEVL Library also can not be less.

4. Modify the Hadoop client configuration file (we only need to configure on the machine using flume)

Http://heylinux.com/archives/2078.html

Flume Lzo compressed output to Hadoop