Hadoop Compression Codec

Brief introduction

Codec is actually coder and decoder two words of the prefix composition of the acronym. Compressioncodec defines the compression and decompression interface, we are talking about the codec is implemented Compressioncodec interface some of the compressed format of the class, the following is a list of these classes:

Unzip using Compressioncodecs

Compressioncodec has two ways to easily compress and decompress:

compression: obtains the Compressionoutputstream object through the Createoutputstream (OutputStream out) method.

extract: Obtain the Compressioninputstream object through the Createinputstream (InputStream in) method.

Sample code for compression

[Java]View PlainCopy

2. Package com.sweetop.styhadoop;
6. Import org.apache.hadoop.conf.Configuration;
8. Import Org.apache.hadoop.io.IOUtils;
10. Import Org.apache.hadoop.io.compress.CompressionCodec;
12. Import Org.apache.hadoop.io.compress.CompressionOutputStream;
14. Import Org.apache.hadoop.util.ReflectionUtils;
18. /**
20. * Created with IntelliJ idea.
22. * User:lastsweetop
24. * date:13-6-25
26. * Time: 10:09
28. * To change this template use File | Settings | File Templates.
30. */
32. Public class Streamcompressor {
34. public static void Main (string[] args) throws Exception {
36. String codecclassname = args[0];
38. class<?> Codecclass = Class.forName (codecclassname);
40. Configuration conf = new configuration ();
42. Compressioncodec codec = (COMPRESSIONCODEC) reflectionutils.newinstance (codecclass, conf);
46. Compressionoutputstream out = Codec.createoutputstream (System.out);
48. Ioutils.copybytes (system.in, out, 4096, false);
50. Out.finish ();
52. }
54. }

Receives a COMPRESSIONCODEC implementation class parameter from the command line, and then instantiates the class through reflectionutils, calls the Compressioncodec interface method to encapsulate the standard output stream, encapsulates it into a compressed stream, The standard input stream is copied into the compressed stream by the Ioutils method, and finally the Compressioncodec finish method is called to complete the compression.

Then look at the command line:

[HTML]View PlainCopy

1. echo "Hello lastsweetop" | ~/hadoop/bin/hadoop Com.sweetop.styhadoop.StreamCompressor Org.apache.hadoop.io.compress.GzipCodec | Gunzip-

Use the Gzipcode class to compress "Hello lastsweetop" and then unzip it with the Gunzip tool.

Let's take a look at the output:

[HTML]View PlainCopy

1. [exec] 13/06/26 20:01:53 INFO util. Nativecodeloader:loaded the Native-hadooplibrary
2. [exec] 13/06/26 20:01:53 INFO zlib. Zlibfactory:successfully Loaded & initialized Native-zlib Library
3. [EXEC] Hello Lastsweetop

Unzip using Compressioncodecfactory

If you want to read a compressed file, first your first through the extension to determine which kind of codec, of course, there is a more convenient way, Compressioncodecfactory has helped you to do this, by passing in a path to call its Getcodec () Method can obtain the corresponding codec. Let's look at the source code:

[Java]View PlainCopy

2. Public class Filedecompressor {
6. public static void Main (string[] args) throws Exception {
10. String uri = args[0];
12. Configuration conf = new configuration ();
14. //Get File system
16. FileSystem FileSystem = Filesystem.get (Uri.create (URI), conf);
20. //Build input path
22. Path InputPath = new Path (URI);
24. //Create Compressioncodecfactory Object
26. Compressioncodecfactory factory = new Compressioncodecfactory (conf);
28. //Get the compressed format of the file
30. Compressioncodec codec = Factory.getcodec (InputPath);
34. //If the compression format does not exist, exit
36. if (codec = = null) {
38. System.out.println ("No codec found for" + URI);
40. System.exit (1);
42. }
44. //Remove suffix of file, key Outputuri as output path of decompression
46. String Outputuri = Compressioncodecfactory.removesuffix (URI, Codec.getdefaultextension ());
50. //define input/output stream
52. InputStream in = null;
54. OutputStream out = null;
58. try {
60. //Create an input/output stream
62. in = Codec.createinputstream (Filesystem.open (InputPath));
64. out = Filesystem.create (new Path (Outputuri));
66. //Unzip
68. Ioutils.copybytes (in, Out, Conf);
70. } catch (Exception e) {
72. E.printstacktrace ();
74. }finally{
76. Ioutils.closestream (in);
78. Ioutils.closestream (out);
80. }
82. }
84. }

Note the Removesuffix method, which is a static method that removes the suffix of the file, and then we use the path as the extracted output path. The codec that compressioncodefactory can find is also limited, By default there are only three types of Org.apache.hadoop.io.compress.gzipcodec;org.apache.hadoop.io.compress.bzip2codec;org.apache.hadoop.io.compress.defa Ultcodec. If you want to add additional codec you need to change the Io.compression.codecs property and register codec.

Native Library

Now more and more native library concept, HDFs codec is no exception, the native library can greatly improve performance such as gzip extraction of the original library increased by 50%, compression increased by 10%, but not all codec have a native library, and some codec only the native library. Let's look at the following table:

Under Linux, Hadoop had previously compiled 32-bit native libraries and 64-bit native repositories, and we looked at:

[HTML]View PlainCopy

1. [[email protected] native] $pwd
2. /home/hadoop/hadoop/lib/native
3. [[email protected] native] $ls-ls
4. Total 8
5. 4 drwxrwxrwx 2 root root 4096 Nov linux-amd64-64
6. 4 drwxrwxrwx 2 root root 4096 Nov linux-i386-32

If it is a different platform, you will need to compile, detailed steps please see here: Http://wiki.apache.org/hadoop/NativeHadoop
Java Native Library path can be specified by Java.library.path, in the bin directory, Hadoop startup script has been specified, if you do not use this script, then you need to specify in your program.

[HTML]View PlainCopy

2. If [-D "${hadoop_home}/build/native"-o-d "${hadoop_home}/lib/native"-o-e "${hadoop_prefix}/lib/libhadoop.a"]; Then
6. If [-D "$HADOOP _home/build/native"]; Then
8. Java_library_path=${hadoop_home}/build/native/${java_platform}/lib
10. Fi
14. If [-D "${hadoop_home}/lib/native"]; Then
16. If ["x$java_library_path"! = "X"]; Then
18. Java_library_path=${java_library_path}:${hadoop_home}/lib/native/${java_platform}
20. Else
22. Java_library_path=${hadoop_home}/lib/native/${java_platform}
24. Fi
26. Fi
30. If [-E "${hadoop_prefix}/lib/libhadoop.a"]; Then
32. Java_library_path=${hadoop_prefix}/lib
34. Fi
36. Fi

Hadoop will go to find the corresponding native library and automatically load, you don't need to care about these settings. But sometimes you don't want to use a native library, such as debugging some bugs, you can do this by setting the Hadoop.native.lib to False.

If you do a lot of compression and decompression with the native repository, consider using Codecpool, which is a bit like connection pooling, so you don't have to create codec objects frequently.

[Java]View PlainCopy

2. Public class Pooledstreamcompressor {
6. public static void Main (string[] args) throws classnotfoundexception {
10. String codecclassname = args[0];
12. //Gets the byte code of the compression class for reflection
14. class<?> Codecclass = Class.forName (codecclassname);
16. //Create configuration information
18. Configuration conf = new configuration ();
20. //Create a compression class by reflection mechanism
22. Compressioncodec codec = (COMPRESSIONCODEC) reflectionutils.newinstance (codecclass, conf);
24. //define compression classes
26. Compressor Compressor = null;
30. try {
32. //Create compressor via Codecpool
34. Compressor = Codecpool.getcompressor (codec);
36. //Create a compression class object
38. Compressionoutputstream out = Codec.createoutputstream (System.out, compressor);
40. //Compression
42. Ioutils.copybytes (system.in, out,4096, false);
44. //Completed
46. Out.finish ();
48. } catch (Exception e) {
50. E.printstacktrace ();
52. } finally {
54. Codecpool.returncompressor (compressor);
56. }
60. }
62. }

The code is easier to understand, and the compressor object is obtained through the Codecpool Getcompressor () method, which needs to pass in a codec and then compressor the object to be used in Createoutputstream. After use, put it back through Returncompressor ().
The output results are as follows:

[HTML]View PlainCopy

1. [exec] 13/06/27 12:00:06 INFO util. Nativecodeloader:loaded The Native-hadoop Library
2. [exec] 13/06/27 12:00:06 INFO zlib. Zlibfactory:successfully Loaded & initialized Native-zlib Library
3. [exec] 13/06/27 12:00:06 INFO compress. Codecpool:got brand-new Compressor
4. [EXEC] Hello Lastsweetop

Originally from: http://blog.csdn.net/lastsweetop/article/details/9173061

Code from: Https://github.com/lastsweetop/styhadoop

Hadoop Compression Codec