Part 2 Common implementation chapter 1 Hadoop Configuration Information Processing Section 2nd configuration text

Hadoop technology: in-depth analysis of HadoopCommon and HDFS Architecture Design and Implementation Principles chapter 1 Hadoop configuration information processing, this chapter begins with Windows and JavaProperties-based configuration files, it analyzes the XML Configuration files consisting of key-pairs used by Hadoop with relatively simple structure, and the corresponding processing class Configuration, especially the C

Hadoop technology Insider: in-depth analysis of Hadoop Common and HDFS Architecture Design and Implementation Principles chapter 1 Hadoop configuration information processing, this chapter begins with Windows and Java Properties-based configuration files, it analyzes the XML Configuration files consisting of key-pairs used by Hadoop with relatively simple structure, and the corresponding processing class Configuration, especially the C

Hadoop technology Insider: in-depth analysis of Hadoop Common and HDFS Architecture Design and Implementation Principles chapter 1 Hadoop configuration information processing, this chapter begins with Windows and Java Properties-based configuration files, analyzes the XML Configuration files consisting of key-value pairs used by Hadoop with relatively simple structure, and corresponding processing class Configuration, in particular, resource loading, resource merging, and attribute scaling are important processes in the Configuration class. This section describes the configuration file.

Part 2 Implementation of Common

This part

Hadoop Configuration Information Processing

Serialization and Compression

Hadoop remote process call

Hadoop File System

Chapter 2 Hadoop Configuration Information Processing

To improve the adaptability and scalability of any complicated software system, a configuration module or system is usually used as a means and method for its expansion and customization. Hadoop uses the configuration file to persist important attributes in the system in the form of files, so that these attributes can be used by restart processes or different processes.

2.1 configuration file Overview

The configuration file is an indispensable part of a flexible system. Although the configuration file is very important, there is no standard. This section describes the configuration files in the Windows operating system and Java environment.

2.1.1 Windows operating system configuration file

Windows systems use a special ASCII file (with "ini" as the file extension) as its main configuration file standard. The following is a snippet of the INI file:

 
 

  
  
Last modification time: 2012.10.12
  
  
[Owner]
  
  
Name = John Doe
  
  
Organization = Acme Widgets Inc.
  
  
 
  
  
[Database]
  
  
Server = 192.0.2.62; the IP address can work normally when domain name resolution is unavailable
  
  
Port = 143
  
  
File = "payroll. dat"
  
  
 
  
  
[Ftp]
 
 

This File is also called the Initialization File. Its extension is the first three letters of initialization) or profile. The application can have its own configuration File, stores application settings. You can also access Windows's basic system configuration file win. configuration information stored in ini. The INI file divides the configuration information into "sections", and the section title is placed in square brackets. For example, [database] in the preceding example is the section title of the database section. The section is used to classify the configuration data. Each section can contain some related "items" and assign values to it by equal signs ). The general form is as follows:

 
 

  
  
[p] 
  
  
ENTRY=VALUE 
 
 

There are two types of VALUE: numeric or string. In the preceding INI file segment, the database section contains three items: server, port, and file. The configuration item port can be read in the form of a number.

The comments in the INI file start with a semicolon and end at the end of the line.

Windows also provides APIs for reading and writing configuration files. If you use the GetProfileString () function, you can obtain the string type configuration from the configuration file win. ini. You can use the GetPrivateProfileInt () function to read a configuration Integer type item from the private configuration file. The function is prototype as follows:

 
 

  
  
UINT WINAPI GetPrivateProfileInt( 
  
  
 __in LPCTSTR lpAppName, 
  
  
 __in LPCTSTR lpKeyName, 
  
  
 __in INT nDefault, 
  
  
 __in LPCTSTR lpFileName 
  
  
); 
 
 

The lpFileName parameter is the name of the INI file, and the lpAppName and lpKeyName parameters are the preceding sections and items respectively. INT nDefault is the default value, that is, if the configuration information cannot be found in the configuration file, the default value is returned.

2.1.2 Java configuration file

JDK provides the java. util. Properties class for processing simple configuration files. Properties has been introduced into Java class libraries for a long time and has never changed. It inherits from Hashtable, as shown in 2-1, indicating a persistent attribute set, which can be saved in a stream or loaded from a stream. In the attribute list, each key and its corresponding value are of the string type.

Compared with the INI file, the configuration file format processed by Properties is very simple. It only supports key-value pairs. The left side of the equal sign "=" is the key, and the right side is the value. The format is as follows:

 
 

  
  
ENTRY=VALUE 
 
 

Because Properties is based on Hashtable, it does not support "sections" in the INI file and classifies configuration items.

Java. util. in Properties, the main method for processing the attribute list is as follows. getProperty () is used to obtain the attribute corresponding to the specified key (parameter key) in the attribute list. It has two forms, one does not provide default values, and the other provides default values. Properties. setProperty () is used to set/update attribute values in the property list. The related code is as follows:

 
 

  
  
// Use the specified key to search for attributes in this attribute list
  
  
Public String getProperty (String key)
  
  
 
  
  
// Functions are the same as above. The default value is provided for the defaultValue parameter.
  
  
Public String getProperty (String key, String defaultValue)
  
  
 
  
  
// Call the Hashtable method put.
  
  
Public synchronized Object setProperty (String key, String value)
 
 

Properties is loaded using the load () method. This method reads key-value pairs from the input stream, while the store () method writes the attribute list in the Properties table to the output stream. Using the input stream and output stream, the Properties object can be saved not only in files, but also in other stream-supported systems, such as Web servers. After J2SE 1.5, data in Properties can also be saved in XML format. The corresponding loading and writing methods are loadFromXML () and storeToXML ().

The following is an example of the Properties configuration file in XML format.

 
 

  
  
  
  
  

  
  
 "http://java.sun.com/dtd/properties.dtd"> 
  
  
  
  
  
  Hi  
  
  
  bar 
  
  
  baz 
  
  
 
 
 

Java. util. Properties provides limited capabilities. A large number of Configuration information read/write solutions have emerged in the Java Community. Among them, the most famous is the Commons Configuration provided by the Apache Jakarta Commons tool set. The PropertiesConfiguration class in Commons Configuration provides a wide range of methods to access Configuration parameters. Commons Configuration supports text and XML Configuration file formats; supports loading multiple Configuration files; supports hierarchical or multilevel Configuration; and provides type-based access to single-value or multi-value Configuration parameters. Commons Configuration is a powerful Configuration file processing tool.