Hadoop detailed (ix) compression in MapReduce

As input

When the compressed file is MapReduce input, MapReduce will automatically extract the corresponding codec from the extension.

As output

When the MapReduce output file requires compression, you can change mapred.output.compress to True, Mapped.output.compression.codec the class name for the codec you want to use

Yes, of course you can specify in the code, by calling the Fileoutputformat static method to set these two properties, let's look at the code:

Package com.sweetop.styhadoop;  
Import Org.apache.hadoop.fs.Path;  
Import org.apache.hadoop.io.IntWritable;  
Import Org.apache.hadoop.io.Text;  
Import Org.apache.hadoop.io.compress.GzipCodec;  
Import Org.apache.hadoop.mapreduce.Job;  
Import Org.apache.hadoop.mapreduce.lib.input.FileInputFormat;  
      
Import Org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;  
      
Import java.io.IOException; 
 /** * Created with IntelliJ idea. * User:lastsweetop * date:13-6-27 * Time: PM 7:48 * To change this template use File | Settings | 
 File Templates.  
        */public class Maxtemperaturewithcompression {public static void main (string[] args) throws Exception {  
            if (args.length!=2) {System.out.println ("Usage:maxtemperature <input path> <out");  
        System.exit (-1);  
        Job Job=new job ();  
        Job.setjarbyclass (Maxtemperature.class);  
      
Job.setjobname ("Max temperature");        Fileinputformat.addinputpath (Job, New Path (args[0));  
      
        Fileoutputformat.setoutputpath (Job, New Path (args[1));  
        Job.setmapperclass (Maxtemperatruemapper.class);  
        Job.setcombinerclass (Maxtemperaturereducer.class);  
      
        Job.setreducerclass (Maxtemperaturereducer.class);  
        Job.setoutputkeyclass (Text.class);  
      
        Job.setoutputvalueclass (Intwritable.class);  
        Fileoutputformat.setcompressoutput (Job, true);  
      
        Fileoutputformat.setoutputcompressorclass (Job, Gzipcodec.class);  
      
    System.exit (Job.waitforcompletion (true)? 0:1); }  
}

Input is also a compressed file

~/hadoop/bin/hadoop com.sweetop.styhadoop.MaxTemperatureWithCompression input/data.gz output/