Hadoop technology insider HDFS-Note 2

To improve the adaptability and scalability of any complicated software system, a configuration module or system is usually used to complete the expansion and customization of the system.

1.1. Windows configuration file

(This is the first time I know about it ):

Text files with the INI suffix can be processed through the dynamic link library.

Initialize a file or profile Profile

1.2. Java configuration file:

Properties File

XML file

Tool: Apache commons configuration is used to perform common operations on configuration files.

Hadoop Configuration

Hadoop uses the configuration class to manage its own configuration files

The root element of the hadoop configuration file is configuration, which only contains the sub-element property. Each property is a configuration item with the configuration property name, value, description and final (meaning this configuration item is "fixed". When merging resources, it can prevent the value of the configuration item from being overwritten)

Each attribute name belongs to the string type, and the value type can be multiple. boolean (getboolean) int (getint) Long (getlong) float (getfloat) string (get) file (GetFile) string Array (getstrings, where values are separated by commas)

Merge resources:

Configuration conf = new configuration ()

Conf. addresource (core-default.xml ");

Conf. addresource (core-site.xml ");

If the configuration item is not marked as final, the subsequent configuration will overwrite the previous configuration. If there is final, there will be a warning when overwriting.

Property extension:

The value of DFS. Name. DIR is $ (hadoop. tmp. DIR)/dfs/name, where $ (Dfs. Name. DIR) is the value of the corresponding attribute in Conf.

Configuration class member variables

When quietmode is true, the default value is true. No log information is output when the parsing configuration file is loaded.

Array resources stores all the resources of the configuration object added by addresource.

Boolean loaddefaults: whether to load default resources. The default resources are saved in defaultresources. You can use adddefaultresource to add default system resources. In HDFS, use hdfs-default.xml and hdfs-site.xml as the default resource, and in Mr, use mapred-default.xml and mapred-site.xml as the default resource.

For example, in the HDFS datanode class (data node), there is a static Block

Static {

Configuration. adddefaultresource (hdfs-default.xml ");

Configuration. adddefaultresource (hdfs-site.xml ");

}

Properties has all parsed key-value pairs

Overlay records the configuration items set by the application through the set method (overwrite or not overwrite)

Finalparameters, all configuration items declared as final

You can use the class loader to load resources under the classpath path,

URL classloader. getresource ("name"); these resources can be of any type. Images, images, files, etc., return the read URL object

Loading files: by loading files in a delayed manner, you can load the Analysis configuration file only when you really need data.

XML parsing format:

Sax: provides a stream and event-driven XML processing method, but the processing logic is complicated and suitable for processing large XML files.

DOM: Different, first load the file into memory to form a tree structure, Document Object Model. Hadoop does not have a large configuration file, so Dom is used to process XML (search for data by yourself)

Configuration loadresouce class (which contains the loaded code ).

Public String get (name, defaultvalue); you can get any configuration attribute value.

Property extension:Use the regular expression \ $ \ {[^ \] \ $] + \} to escape the three characters $, at the same time, [^] matches all characters except the values in it. + The number of occurrences exceeds 1

The retriable interface implements this interface, which means that this class can be configured, that is, it can be passed into the system's conf instance. In this way, you can obtain the configuration information useful to you in the conf file.

Setconf (configuration) and getconf ()

When to call setconf? Using the reflection mechanism, you can use the setconf method in the reflectionutils class to configure the object. Whether the IF (OBJ instanceof configrable) class implements the retriable Interface