2.2 Hadoop Configuration Detailed

2.2 Hadoop Configuration Detailed

Hadoop does not use the Java.util.Properties management profile, nor does it use the Apache Jakarta Commons configuration to manage the configuration files, but instead uses a unique set of configuration file management systems and provides its own API , which uses org.apache.hadoop.conf.Configuration to process configuration information.

Format of the 2.2.1 Hadoop configuration file

The Hadoop configuration file is in XML format, and the following is an example of the Hadoop configuration file:

<?xml version= "1.0"?>  <?xml-stylesheet type= "text/xsl"  href= "configuration.xsl"? >  <configuration>    <property>       <name >io.sort.factor</name>       <value>10</value>        <description>the number of streams to merge at  once while sorting        files.  this determines  the number of open file handles.</description>    </ property>     <property>       <name>dfs.name.dir< /name>       <value>${hadoop.tmp.dir}/dfs/name</value>        <description>determines where on the local filesystem  the&nBsp;dfs name        nodeshould store the name table ( Fsimage) .  ......</description>    </property>     <property >       <name>dfs.web.ugi</name>        <value>webuser,webgroup</value>       <final>true</ final>       <description>the user account used by  the web interface.        Syntax: USERNAME,GROUP1,GROUP2, ......</description>    </property>  </configuration> 

The root element of the Hadoop configuration file is configuration, which typically contains only child element property. Each property element is a configuration item, and the configuration file does not support layering or grading. Each configuration item typically includes the name of the configuration property, value values, and a description of the configuration item description; The element final is similar to the keyword final in Java, meaning that the configuration item is "fixed". Final generally does not appear, but when merging resources, you can prevent the values of the configuration items from being overwritten.

In the example file above, the value of the configuration item Dfs.web.ugi is "Webuser,webgroup", which is a final configuration item, and from description, this configuration item configures the user account for the Hadoop Web interface, including user name and user group information. This information can be accessed through the methods provided by the configuration class.

In configuration, each property is of type string, but a value type may be of the following types, including the basic types in Java, such as Boolean (Getboolean), int (getInt), Long (Getlong), Float (getfloat) can also be other types, such as String (get), Java.io.File (getFile), string array (getstrings), and so on. As an example of the above configuration file, GetInt ("Io.sort.factor") returns an integer 10, while Getstrings ("Dfs.web.ugi") returns an array of strings with two elements, WebUser and Webgroup, respectively.

Merging resources refers to merging multiple profiles to produce a configuration. If you have two profiles, which are two resources, such as Core-default.xml and core-site.xml, merge them into one configuration through the Loadresources () method of the configuration class. The code is as follows:

configurationconf = new Configuration ();   Conf.addresource ("Core-default.xml"); Conf.addresource ("Core-site.xml");

If both configuration resources contain the same configuration items, and the configuration items for the previous resource are not marked final, then the subsequent configuration overrides the previous configuration. In the example above, the configuration in Core-site.xml overrides the configuration in Core-default.xml with the same name. If a configuration item is marked final in the first resource (Core-default.xml), a warning prompt is available when the second resource is loaded.