Hadoop 2.2.0 and HBase-0.98 installation snappy

1. Installation of dependent packages and software required

The dependent packages that need to be installed are:

GCC, C + +, autoconf, Automake, Libtool

The necessary software for installation is:

JAVA6, Maven

For the above dependency package, assume that under Ubuntu, use the sudo apt-get install * command installation. Let's say under CentOS. Install using the sudo yum install * command.

For the installation of the accompanying Java and Maven, refer to the installation of Java, MAVEN, Tomcat under Linux Kauboven.

2. Download snappy-1.1.2

Address available for download:

Address One: https://code.google.com/p/snappy/wiki/Downloads?tm=2

Address two: http://download.csdn.net/detail/iam333/7725883

3. Compile and install dynamically

Unzip to a folder after downloading, if you unzipped the address bit home folder. Again run such as the following commands such as the following:

$ cd ~/snappy-1.1.2$ sudo./configure$ sudo./make$ sudo make install

Then run a command such as the following to see if the installation was successful.

$ cd/usr/local/lib$ ll libsnappy.*-rw-r--r--1 root root 233506 7 11:56 libsnappy.a-rwxr-xr-x 1 root root    953 7 11:56 libsnappy.lalrwxrwxrwx 1 root root     7 11:56 libsnappy.so libsnappy.so.1.2.1lrwxrwxrwx 1 root root     7 11:56 libsnappy.so.1-libsnappy.so.1.2.1-rwxr-xr-x 1 root root 147758 7 11:56 libsnappy.so.1.2.1

It is assumed that no errors were encountered during the installation and that the/usr/local/lib folder has the above file indicating a successful installation.

4, Hadoop-snappy source code compilation

1) Download the source code. Two different ways

A, install SVN, assuming Ubuntu, use sudo apt-get install subversion, assuming it is centos. Install the Subversion command using sudo yum.

B. Use SVN to checkout the source code from Google's SVN repository, using the following command, for example:

$ svn Checkout Http://hadoop-snappy.googlecode.com/svn/trunk/hadoop-snappy

This copies the source code of the Hadoop-snappy into the Hadoop-snappy folder under the folder where the command is run.

Just because Google's services are always problematic on the mainland. So you can also choose to download directly, address: http://download.csdn.net/detail/iam333/7726023

2) Compile Hadoop-snappy source code

Switch to the Hadoop-snappy source code folder and run the following command, for example:

A, assume that the above installation snappy use the default path, the command is:

MVN Package

b, assuming that the snappy installed above uses its own defined path, the command is:

MVN Package [-dsnappy.prefix=snappy_installation_dir]

The SNAPPY_INSTALLATION_DIR bit snappy installation path.

Issues that may occur during the compilation process:

A)/root/modules/hadoop-snappy/maven/build-compilenative.xml:62:execute Failed:java.io.IOException:Cannot run Program ' autoreconf ' (in Directory "/ROOT/MODULES/HADOOP-SNAPPY/TARGET/NATIVE-SRC"): java.io.ioexception:error=2, No Such file or directory

Workaround: Description missing file. But this file is under Target, is the compilation process of their own initiative generated, originally should not exist, this is asked what? In fact, the fundamental problem is not the lack of documentation, but the need for Hadoop snappy to have a certain preconditions.

So please refer to the top install dependencies package for installation dependent packages.

b) such as the following error message appears:

Workaround: The official documentation for Hadoop snappy only lists the need for GCC. There is no GCC listing what version number is required. In fact, Hadoop snappy needs to be gcc4.4. If the GCC version number is higher than the default 4.4 version number, an error will be given.

If you are using a system for CentOS, use such as the following command: (note: Ubuntu needs to change sudo yum install to sudo apt-get install)

$ sudo yum install gcc-4.4$ sudo rm/usr/bin/gcc$ sudo ln-s/usr/bin/gcc-4.4/usr/bin/gcc

Use the following command, for example, to see if the substitution succeeded:

$ gcc--VERSIONGCC (GCC) 4.4.7 20120313 (Red Hat 4.4.7-3) Copyright (C) free software Foundation, inc.this are free sof Tware; See the source for copying conditions.  There is nowarranty; Not even to merchantability or FITNESS for A particular PURPOSE.

c) such as the following error message appears:

[exec]/bin/bash./libtool--tag=cc   --mode=link gcc-g-wall-fpic-o2-m64-g-o2-version-info 0:1:0-l/usr/local//l Ib-o libhadoopsnappy.la-rpath/usr/local/lib Src/org/apache/hadoop/io/compress/snappy/snappycompressor.lo src/org/ Apache/hadoop/io/compress/snappy/snappydecompressor.lo  -LJVM-LDL [exec]/usr/bin/ld:cannot find-ljvm[exec] Collect2:ld returned 1 exit status[exec] Make: * * [libhadoopsnappy.la] Error 1[exec] libtool:link:gcc-shared  -fPIC- Dpic  SRC/ORG/APACHE/HADOOP/IO/COMPRESS/SNAPPY/.LIBS/SNAPPYCOMPRESSOR.O src/org/apache/hadoop/io/compress/ SNAPPY/.LIBS/SNAPPYDECOMPRESSOR.O   -l/usr/local//lib-ljvm-ldl  -o2-m64-o2   -wl,-soname-wl, Libhadoopsnappy.so.0-o. libs/libhadoopsnappy.so.0.0.1

This is due to the fact that the libjvm.so symbolic installing the JVM is not linked to Usr/local/lib. Assuming your system is 64-bit, you can go to/root/bin/jdk1.6.0_37/jre/lib/amd64/server/view libjvm.so link to the place, here changes such as the following, using the command:

$ sudo ln-s/usr/local/jdk1.6.0_45/jre/lib/amd64/server/libjvm.so/usr/local/lib/

Problem can be solved.

5. Hadoop 2.2.0 Configuration Snappy

Hadoop-snappy after the compilation is successful. Some files are generated in the target folder under the Hadoop-snappy folder. One of the file names is: hadoop-snappy-0.0.1-snapshot.tar.gz

1) Unzip target under Hadoop-snappy-0.0.1-snapshot.tar.gz, unzip, copy lib file

$ sudo cp-r ~/snappy-hadoop/target/hadoop-snappy-0.0.1-snapshot/lib/native/linux-amd64-64/* $HADOOP _home/lib/ native/linux-amd64-64/

2) Copy the Hadoop-snappy-0.0.1-snapshot.jar under Target to $hadoop_home/lib.

3) Configure $hadoop_home/etc/hadoop/hadoop-env.sh, add:

Export ld_library_path= $LD _library_path: $HADOOP _home/lib/native/linux-amd64-64/:/usr/local/lib/

4) Configure $hadoop_home/etc/hadoop/mapred-site.xml. In this file, all the configuration options related to compression are:

<property> <name>mapred.output.compress</name> <value>false</value> <description  >should the job outputs be compressed?  </description></property> <property> <name>mapred.output.compression.type</name>               <value>RECORD</value> <description>if The job outputs is to compressed as sequencefiles, how should They be compressed?  Should be one of NONE, RECORD or BLOCK.  </description></property> <property> <name>mapred.output.compression.codec</name> <value>org.apache.hadoop.io.compress.DefaultCodec</value> <description>if the job outputs is  Compressed, how should they is compressed? </description></property> <property> <name>mapred.compress.map.output</name> < Value>false</value> <description>should The outputs of the maps be compressed before being S ENT across the network. UseS sequencefile compression. </description></property> <property> <name>mapred.map.output.compression.codec</name > <value>org.apache.hadoop.io.compress.DefaultCodec</value> <description>if the map outputs is Compressed, how should they is compressed?
</description></property>

Can be configured according to their own needs.

Of The types of codec such as the following:

<property>    <name>io.compression.codecs</name>    <value>      Org.apache.hadoop.io.compress.GzipCodec,      Org.apache.hadoop.io.compress.DefaultCodec,      Org.apache.hadoop.io.compress.BZip2Codec,      org.apache.hadoop.io.compress.SnappyCodec    </value>  </property>

Snappycodec represents the snappy compression method.

5) After the configuration is ready. Restarting the Hadoop cluster is possible.

6, HBase 0.98 configuration Snappy

1) Configure the Lib file in HBase lib/native/linux-amd64-64/.

For simplicity, we just need to $hadoop_home/lib/native/linux-amd64-64/the Lib file, all copied to the corresponding HBase folder:

$ sudo cp-r $HADOOP _home/lib/native/linux-amd64-64/* $HBASE _home/lib/native/linux-amd64-64/

2) Configure HBase environment variable hbase-env.sh

Export ld_library_path= $LD _library_path: $HADOOP _home/lib/native/linux-amd64-64/:/usr/local/lib/export HBASE_ Library_path= $HBASE _library_path: $HBASE _home/lib/native/linux-amd64-64/:/usr/local/lib/export CLASSPATH=$ CLASSPATH: $HBASE _library_path

Note: Don't forget to configure Hadoop_home and Hbase_home at the beginning of the habase-env.sh.

3) After the configuration is done, you can restart HBase.

4) Verify the installation is successful

Under the installation folder for HBase. Run such as the following statement:

$ bin/hbase shell2014-08-07 15:11:35,874 INFO  

Then run the CREATE statement:

To view the created Test_snappy table:

hbase (main):002:0> describe ' test_snappy ' DESCRIPTION                                                                     ENABLED ' Test_snappy ', {NAME = ' cf ', data_block_encoding = ' NONE ', Bloomfilte                                                                        R = ' ROW ', Replication_scope = ' 0 ', VERSIONS = ' 1 ', COMPRESSIO true N = ' SNAPPY ', min_versions = ' 0 ', TTL = ' 2147483647 ', keep_deleted_cells = ' F                                                                             Alse ', BLOCKSIZE = ' 65536 ', in_memory = ' false ', BLOC                                                                                                                                                                                                   Kcache = ' true '} 1 row (s) in 0 .0420 seconds 

can see, COMPRESSION = ' SNAPPY '.

Next. Insert Data try:

HBase (main):003:0> put ' test_snappy ', ' key1 ', ' cf:q1 ', ' value1 ' 0 row (s) in 0.0790 secondshbase (main):004:0>

To traverse the Test_snappy table try:

HBase (main):004:0> scan ' test_snappy ' ROW                                                    column+cell                                                                                                                                                    key1                                                  column=cf:q1, timestamp=1407395814255, Value=value1                                                                                                           

The above process can be run correctly, indicating the correct configuration.

Error Resolution Method:

A) after configuration, start HBase appears for example the following exception:

WARN [main] util.CompressionTest:Can ' t instantiate Codec:snappyjava.io.IOException:java.lang.UnsatisfiedLinkError: Org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy () Zat Org.apache.hadoop.hbase.util.CompressionTest.testCompression (compressiontest.java:96) at Org.apache.hadoop.hbase.util.CompressionTest.testCompression (compressiontest.java:62) at Org.apache.hadoop.hbase.regionserver.HRegionServer.checkCodecs (hregionserver.java:660) at Org.apache.hadoop.hbase.regionserver.hregionserver.<init> (hregionserver.java:538) at Sun.reflect.NativeConstructorAccessorImpl.newInstance0 (Native Method) at Sun.reflect.NativeConstructorAccessorImpl.newInstance (nativeconstructoraccessorimpl.java:57) at Sun.reflect.DelegatingConstructorAccessorImpl.newInstance (delegatingconstructoraccessorimpl.java:45) at Java.lang.reflect.Constructor.newInstance (constructor.java:526)

The instructions are not configured yet, check the configuration in hbase-env.sh to see if you are configured correctly.

Reprint Please specify Source:http://blog.csdn.net/iAm333

Hadoop 2.2.0 and HBase-0.98 installation snappy