1. Hadoop supports Lzo compression dependencies:
The Lzo:unix/linux system does not have a Lzo library by default, so it needs to be installed, sudo yum install lzo-devel.x86_64
sudo yum install lzo.x86_64 sudo yum install lzop.x86_64
2. Prepare MAVEN,ANT,GCC, etc.
3. Compiling Hadoop-lzo
Download from Https://github.com/twitter/hadoop-lzo, unzip into directory, MVN clean package
Under Target, there are libgplcompression libraries under Hadoop-lzo-0.4.20-snapshot.jar target/native/linux-amd64-64/lib.
The jar package is copied to the Hadoop Lib directory, and the Gplcompression library is copied to Lib/native/linux-amd64-64/lib
In Hadoop client or individual node, do not need to install Lzo library, can directly copy the Lzo library to Lib/native/linux-amd64-64/lib under Hadoop
Summary: Compile Hadoop-lzo, LZO-DEVL Library also can not be less.
4. Modify the Hadoop client configuration file (we only need to configure on the machine using flume)
Http://heylinux.com/archives/2078.html
Flume Lzo compressed output to Hadoop