Hadoop, the alternative solution above
Compression/
ExtractPlugin lzo installation instructions
1. Required Software
Apache-ant-1.8.0-bin.tar The ant compilation tool must be later than version 1.7. Otherwise, some attributes are not supported.
Kevinweil-hadoop-lzo-23e8370.tar Used to compile a hadoop-lzo-0.4.13.jar File
Hadoop-gpl-
CompressIon-0.1.0-rc0.tar.gz above the alternative, tested this solution is better, it is recommended to use
Lzo-2.04.tar Lzo dynamic library Compilation
Lzop-1.03.tar Lzo client tool, used to compress and decompress to generate a. lzo file for testing
2. Install ant Tool
Download apache 1.7 or later
Decompress the package and add it to/usr/local/apache-ant-1.8.0.
ANT_HOME =/usr/local/apache-ant-1.8.0
PATH = $ PATH: $ ANT_HOME/bin
Source/etc/pro
File
3. Compile and install the lzo Library 【
ClusterYou must copy the library file to install each machine.] Many exception errors are encountered during the installation process. Therefore, we recommend that you compile and install each machine.
Wget http://www.oberhumer.com/opensource/lzo/download/lzo-2.04.tar.gz
Tar-zxvf lzo-2.04.tar.gz
./Configure --
Enable-Shar
Ed
Make
Make install
Library files are installed in the/usr/local/lib directory by default.
Any of the following operations is required:
A. Copy the lzo library in the/usr/local/lib directory to/usr/lib [/usr/lib64] According to the system's decision.
B. Create the lzo. conf file under the/etc/ld. so. conf. d/directory, write the path of the file into the database, and run/sbin/ldconfig-v to make the configuration take effect.
4. Install the decoder
Https ://
GitHub.com/kevinweil/hadoop-lzo. the latest download source code is currently kevinweil-hadoop-lzo-23e8370.tar.gz.
Tar-zxvf kevinweil-hadoop-lzo-2ad6654.tar.gz
CdKevinweil-hadoop-lzo-2ad6654
Ant compile-native tar
If the operation is correct, the corresponding jar file hadoop-lzo-0.4.13.jar will be produced for deployment to the Cluster
Note the following points:
A. if you encounter the following exceptions
/Usr/bin/build-classpath: error: cocould not
FindXml-commons-apis Java
ExTension for this JVM
/Usr/bin/build-classpath: error: Some specified jars were not found
The dependency plug-in is not correctly installed. You can use the following
CommandInstall
Yum-y remove xml-commons-apis
Yum-y install xml-commons-apis
B. Class org. apache. too
Ls. Ant. taskdefs. ConditionTask doesn't
SuPport the nested "typefound" element.
Cause: the ant version is earlier. upgrade to version 1.7.0 or later to view the version.
If the upgrade is complete, the environment variable is also set or the above exception occurs. Try to run ant in the absolute path.
For example:/usr/local/apache-ant-1.8.0/bin/ant compile-native tar
If the above process encounters an error, the following solution is feasible:
Download: http://code.google.com/a/apache-ex
TrAs.org/p/hadoop-gpl-compression/downloads/list
Download: hadoop-gpl-compression-0.1.0-rc0.tar.gz
Compile according to the above compilation process. If it runs correctly, the hadoop-gpl-compression-0.1.0.jar will be generated
[Note:] this jar package is suitable for non-cdh3 hadoop versions. Because cdh3 has modified the compression class in hadoop source code, if you are a cdh user, refer to the last step.
5. Copy the decoder and native Library to the hadoop cluster.
CpBuild/hadoop-lzo-0.4.10.jar/usr/local/cdh3u0/hadoop-0.20.2-CDH3B4/lib/
Tar-cv-c build/native. | tar-xBvf--C/usr/local/cdh3u0/hadoop-0.20.2-CDH3B4/lib/native
If there is a document on the Internet, you can use cp directly.
Cd kevinweil-hadoop-lzo-2ad6654/build/native/Linux-amd64-64/lib
Cp * $ HADOOP_HOME/lib/native/Linux-amd64-64
Cp * $ HBASE_HOME/lib/native/Linux-amd64-64
6. Configure hadoop cluster configuration file core-site.xml, mapred-site.xml
Core-site.xml
Io. compression. codecs
Org. apache. hadoop. io. compress. DefaultCodec, org. apache. hadoop. io. compress. G
ZipCodec, org. apache. hadoop. io. compress. BZip2Codec, com. hadoop. compression. lzo. LzopCodec
Io. compression. codec. lzo. class
Com. hadoop. compression. lzo. LzoCodec
Mapred-site.xml
Mapred. map. output. compress
True
Mapred. child. env
JAVA_LIBRARY_PATH =/usr/local/cdh3u0/hadoop-0.20.2-cdh3u1/lib/native/Linux-amd64-64
Mapred. map. output. compress. codec
Com. hadoop. compression. lzo. LzoCodec
7. Install the lzo file generation tool Used to generate some lzo compressed files for testing
Wget http://www.lzop.org/download/lzop-1.03.tar.gz
Tar-zxvf lzop-1.03
Cd lzop-1.03
ExportLD_LIBRARY_PATH = $ LD_LIBRARY_PATH:/usr/local/lib
./Configure
Make & make install
Run the following command after successful installation:
Lzop-U-9 The lzo-test.txt compresses the .txt file into a. lzo File
Hadoop fs-copyFromLocal/home/hadoop/likl/lzo-test.txt.lzo Lzo-test-in
[A crucial step]
Http://code.google.com/p/hadoop-gpl-compression/download the native (hadoop-gpl-compression-0.1.0-rc0.tar.gz)
Unpackage, which has a hadoop-gpl-compression-0.1.0.jar root directory, lib directory has the corresponding native library,
Copy the hadoop-gpl-compression-0.1.0.jar to the lib directory of hadoop
Copy the libraries under the lib/native directory to hadoop/lib/native.
* ******* If hadoop uses the cdh3 version, cloudora will rewrite the compression class. Therefore, you need to modify the source code and re-compile it based on the above.
Add the following code snippet:
Import org. apache. hadoop. conf. Configuration;
Public vo
IdReinit (Configuration conf ){
// Do nothing
}
Ant recompile and copy the generated hadoop-gpl-compression-0.1.0-dev.jar to/hadoop/lib to replace the previous package
[FAQ] http://code.google.com/a/apache-extras.org/p/hadoop-gpl-compression/wiki/FAQ? Redir = 1
| [Content navigation] |
| Page 1: lzo |
Page 1: chukwa |
| Page 6: chukwa Client |
Page 1: cloudera |
| Page 5th: rsyNcD |
Page 7: shell |
