CDH4.5.0下安裝snappy

標籤：blog   http   java   使用   os   檔案   

編譯原始碼 http://www.cnblogs.com/chengxin1982/p/3862289.html

測試參考 http://blog.jeoygin.org/2012/03/java-compression-library-test.html
1 snappy
參考地址
 http://sstudent.blog.51cto.com/7252708/1405485(主)
 http://wzxwzx2011.blog.51cto.com/2997448/1111619

snappy庫: wget http://pkgs.fedoraproject.org/repo/pkgs/snappy/snappy-1.1.1.tar.gz/8887e3b7253b22a31f5486bca3cbc1c2/snappy-1.1.1.tar.gz
tar -zxvf  snappy-1.0.5.tar.gz
cd snappy-1.0.5
./configure
make
sudo make install
或者  sudo yum install snappy snappy-devel(centos)


安裝hadoop-snappy包 ：https://github.com/electrum/hadoop-snappy
    sudo apt-get install automake libtool
    cd hadoop-snappy
    mvn package


修改設定檔core-site.xml
 <property>
    <name>io.compression.codecs</name>
        <value>
                org.apache.hadoop.io.compress.GzipCodec,
                org.apache.hadoop.io.compress.DefaultCodec,
                org.apache.hadoop.io.compress.BZip2Codec,
                org.apache.hadoop.io.compress.SnappyCodec
        </value>
</property>


錯誤1 Cannot run program "autoreconf"
解決辦法: apt-get install autoconf，automake，libtool(參考 http://www.cnblogs.com/shitouer/archive/2013/01/05/2845954.html)

錯誤 2
java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
    at org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native Method)
    at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:62)
    at org.apache.hadoop.io.compress.SnappyCodec.createCompressor(SnappyCodec.java:138)
    at org.apache.hadoop.io.compress.SnappyCodec.createOutputStream(SnappyCodec.java:93)
    at org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.getRecordWriter(TextOutputFormat.java:136)
    at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.<init>(ReduceTask.java:562)
    at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:636)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:404)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:443)

解決方案
mdkir -p $HADOOP_HOME/lib/native/Linux-amd64-64

cp -r hadoop-snappy-0.0.1-SNAPSHOT/lib/native/Linux-amd64-64/* $HADOOP_HOME/lib/native/Linux-amd64-64
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_HOME/sbin/Linux-amd64-64/:/usr/local/lib/

錯誤3 DefaultCodec/GZipCodec not found
修改core-site.xml
  <property>
        <name>io.compression.codecs</name> <value>org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.BZip2Codec,org.apache.hadoop.io.compress.SnappyCodec</value>
去掉空格



配置hbase
mkdir -p $HBASE_HOME/lib/native/Linux-amd64-64/
cp -r $HADOOP_HOME/lib/native/Linux-amd64-64/* $HBASE_HOME/lib/native/Linux-amd64-64/
cp hadoop-snappy-0.0.1-SNAPSHOT.jar $HBASE_HOME/lib/

使用
conf.setBoolean("mapreduce.map.output.compress", true);
    conf.setClass("mapreduce.map.output.compression.codec", SnappyCodec.class, CompressionCodec.class);
    conf.setBoolean("mapreduce.output.fileoutputformat.compress", true);
    conf.setClass("mapreduce.output.fileoutputformat.compress.codec", SnappyCodec.class, CompressionCodec.class);