The compressed final output file of the Hadoop map-reduce

Hadoop can change the behavior of a job by setting a series of parameters in the configuration object of the job, for example, we need to do a map-reduce job, and the result of the final job reduction process is output in a compressed format. We can do some customization on the general Map-reduce.

Realize

Or in the previous case of the deletion of the highest temperature as a reference:

Previous examples can be found in this blog post: http://supercharles888.blog.51cto.com/609344/878422

We now ask for the output to be compressed, so keep the map class (Maxtemperaturemapper) and the Reduce class (maxtemperaturereducer) unchanged. As long as the configuration of the job class makes some compressed configuration, see the 第45-49 line:

Package com.charles.parseweather.compression; 
Import org.apache.hadoop.conf.Configuration; 
Import Org.apache.hadoop.fs.Path; 
Import org.apache.hadoop.io.IntWritable; 
Import Org.apache.hadoop.io.Text; 
Import Org.apache.hadoop.io.compress.CompressionCodec; 
Import Org.apache.hadoop.io.compress.GzipCodec; 
Import Org.apache.hadoop.mapreduce.Job; 
Import Org.apache.hadoop.mapreduce.lib.input.FileInputFormat; 
Import Org.apache.hadoop.mapreduce.lib.input.TextInputFormat; 
Import Org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; 
     
     
Import Org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
     
/** * * * Description: This class defines and runs the job, compact version * * @author Charles.wang * @created May, 5:29:12 PM * * * * public class Maxtemperaturewithcompression {/** * @param args/public static void main ( String[] args) throws exception{//TODO auto-generated Method stub if (args.length !=2) {System.err.println ("usage:maxtemperature <input path> <output path>"); 
        System.exit (-1); 
        //Create a map-reduce job Configuration conf = new Configuration (); 
             
        Conf.set ("Hadoop.job.ugi", "Hadoop-user,hadoop-user"); Here we configure some of the parameters related to the compression//We set the reduce output result using the GZIP compression form Conf.setboolean ("Mapred.output.compress", 
        true); 
             
             
             
        Conf.setclass ("Mapred.output.compression.codec", Gzipcodec.class, Compressioncodec.class); 
             
           
             
             
             
        Job Job = new Job (conf, "Get Maximum Weather information with compression! ^_^"); Set the Startup class/Job.setjarbyclass for the job (maxtemperaturewithcompre 
             
        Ssion.class); 
        Parse input and output parameters, respectively, as the input and output of the job, are file Fileinputformat.addinputpath (job, New Path (args[0)); Fileoutputformat.setoutputpath (Job,New Path (args[1])); 
        Configure the job, set Mapper class, Reducer class Job.setmapperclass (Maxtemperaturemapper.class); 
        Job.setreducerclass (Maxtemperaturereducer.class); 
       Job.setoutputkeyclass (Text.class); 
            

        Job.setoutputvalueclass (Intwritable.class); 
            

    System.exit (Job.waitforcompletion (true)? 0:1); } 
     
}