"Hadoop Authority" learning note five: MapReduce application

First, the configuration of the API---configclass

• API configuration: Hadoop provides a dedicated API to configure resources,Configuration ClassInstances of (inorg.apache.hadoop.conf Bag) includes a collection of configuration properties and their values. Each property is a string type, and the type of the value may be of several types, including Java primitives (such as Boolean, int, long, float), and some other useful types. Configure the properties that read them from a given resource, usually the structure of a name-value pair is simpleXML File

  1 <?XML version= "1.0"?>2 <Configuration>3     < Property>4         <name>Color</name>5         <value>Yellow</value>6         <Description>Color</Description>7     </ Property>8 9     < Property>Ten         <name>Size</name> One         <value>10</value> A         <Description>Size</Description> -     </ Property> -  the     < Property> -         <name>Weight</name> -         <value>Heavy</value> -         <Final>True</Final> +         <Description>Weight</Description> -     </ Property> +  A     < Property> at         <name>Size-weigth</name> -         <value>${size},${weight}</value> -         <Description>Size and Weight</Description> -     </ Property> - </Configuration>

  1 New Configuration (); 2 conf.addresource ("Configuration-1.xml"); 3 System.out.println (conf.get ("Color")); 4 System.out.println (conf.getint ("size", 0)); 5 System.out.println (Conf.get ("breadth", "wide"));

• Merge profiles: In Hadoop, thecore-default.xml file defines the default properties of the system, while Core-site.xml defines the properties of the specific overrides. If more than one configuration file is defined at the same time, subsequent attributes will overwrite the previously defined property, but if final is marked as true, the property will not be overwritten and the system will report a configuration error.
• Extended Properties

1. 1. The configured properties can be defined by other properties or system properties, such as ${size},${weigth}
   2. The system attribute defines a property, that is, system.setproperty ("size"), and the precedence is higher than the attribute defined by the file resource
   3. To override a property by using the JVM parameter -dproperty=value on the command line

Second, configure the development environment

• CONF option: Easy to switch configuration files
• Genericoptionsparser,tool and Toolrunner: Genericoptionsparser A class that explains common Hadoop command-line options, which can be set in the configuration object depending on the application, but is usually not used, but instead uses toolrunnner because it implements the tool interface, Easier to run.

  1  Public Interface extends configurabel{2     int throws Exception; 3 }

  1  Public classConfigurationprinterextendsConfiguredImplementsTool {2     Static{3Configuration.adddefaultresource ("Hdfs-default.xml");4Configuration.adddefaultresource ("Hdfs-site.xml");5Configuration.adddefaultresource ("Mapred-default.xml");6Configuration.adddefaultresource ("Mapred-site.xml");7     }8 9 @OverrideTen      Public intRun (string[] args)Throwexception{ OneConfiguration conf =getconf (); A           for(entry<string,string>entry:conf) { -System.out.printf ("%s=%s\n", Entry.getkey (), Entry.getvalue ()); -          } the          return0; -     } -  -      Public Static voidMain (string[] args)ThrowException { +          intExitCode = Toolrunner.run (NewConfigurationprinter,args); - System.exit (exitCode); +     } A  at  -}

III. Writing Unit Tests

Four, local run test data

V. Running on a cluster

Vi. Operation Tuning

Vii. Workflow for MapReduce

"Hadoop Authority" learning note five: MapReduce application