Hadoop Installation Lzo Experiment

Reference http://blog.csdn.net/lalaguozhe/article/details/10912527

Environment: hadoop2.3cdh5.0.2

Hive 1.2.1

Target: Install Lzo Test job run with Hive table creation using LZO format store

Before installing the trial snappy, it was found that the CDH extracted native contains a local library such as Libsnappy, but does not contain lzo.

Therefore, the use of Lzo, in addition to installing the LZO program, but also to compile the installation Hadoop-lzo.

1. Install the Lzo. Yum can be installed, or you can download the source code according to the links provided above to compile the installation.

2. git download install Hadoop-lzo, compile and install

git clone https://github.com/twitter/hadoop-lzo.git?
Export cflags=-m64
Export cxxflags=-m64
MVN Clean Package-dmaven.test.skip=true

CP linux-amd64-64/lib/app/cdh23502/lib/native/
CP hadoop-lzo-0.4.20-snapshot.jar/app/cdh23502/share/hadoop/common/

When I compile the problem is the Mave library URL domain Name resolution failed two or three times, I have tried several times, compile generally no problem.

Place the native and jar packages in the right place and distribute them to the cluster.

Because you have previously configured the use of snappy, you only need to change the two,

One is the core-site.xml added Lzo: org.apache.hadoop.io.compress.lz4codec,com.hadoop.compression.lzo.lzopcodec

< Property>      <name>Io.compression.codecs</name>      <value>Org.apache.hadoop.io.compress.defaultcodec,org.apache.hadoop.io.compress.gzipcodec, Org.apache.hadoop.io.compress.bzip2codec,org.apache.hadoop.io.compress.snappycodec, Org.apache.hadoop.io.compress.lz4codec,com.hadoop.compression.lzo.lzopcodec</value>          <Description>A comma-separated List of the compression codec classes that can is used for compression/decompression. In addition-to-classes specified with this property (which take precedence), codec classes on the Classpa Th is discovered using a Java serviceloader.</Description>    </ Property>

The second is to replace the snappy in Mapred-site.xml:

< Property>      <name>Mapred.compress.map.output</name>      <value>True</value>  </ Property>  < Property>      <name>Mapred.map.output.compression.codec</name>      <value>Com.hadoop.compression.lzo.LzoCodec</value>  </ Property>

To create a hive table:

Seq 1 > Nums.txt

hive -e "create table nums(num int) row format delimited stored as textfile;"

hive -e "load data local inpath ‘/yourpath/nums.txt‘ overwrite into table nums;"

And then

CREATE TABLE lzo_test (   col String  )  as  InputFormat " Com.hadoop.mapred.DeprecatedLzoTextInputFormat "OutputFormat" Org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat ";

Insert into Table lzo_test (COL) select Num from Nums;

Select COUNT (*) from lzo_test;

Hadoop Installation Lzo Experiment