HDFs Source Code Analysis first: Hadoop configuraion

Hadoop is a huge source of code, the first part of the first analysis of configuration.

A configuration in a Hadoop system can be seen as a bunch of key-value pairs, stored in the properties of the configuration, which are derived from the Resources, A resource can be an XML file, it can be another configuration, and so on. The configuration will load Core-default.xml and core-site.xml two resource by default. The configuration can invoke the AddResource method to dynamically load the resource, which, after loading, triggers the property refactoring of all the configuration in the Hadoop system.

In order to know what configuration is in the Hadoop system, the configuration class consists of a static REGISTRY, which is a weakhashmap, but value is not used, All existing configuration is saved.

Weakhashmap is a more interesting thing.

HashMap when put (Key,value), the key is added to a reference, when the key is deleted elsewhere, if the HashMap does not delete the key, the GC will not recycle the key.

But when Weakhashmap put (key, value), the key is not added to a reference, and when the key is deleted elsewhere, after the GC, Weakhashmap will automatically delete the key, so, Weakhashmap is ideal for managing resources, and Weakhashmap will automatically delete this key when the resource is deleted:

1.     new weakhashmap " key value pair " added to weakhashmap weakhashmap table save entry ( key-value pairs Span style= "Color:black", each entry entry

2.     when a " weak key " GC when recovering. In GC recycle the weak key " " weak key " Span style= "Color:black" will also be added to referencequeue (queue) in the queue.

3.     Now we need to operate weakhashmap table queue table queue GC The key-value pairs that are recycled, or they are deleted table GC The key-value pairs that are recycled.

this is " Weak keys " how to be automatically removed from Weakhashmap The steps removed in the

After the property is triggered to refactor, the external call to the configuration's Get method will re-load the Key-value from various resource to the properties when it wants a property value.

In addition to AddResource, there are adddefaultresource methods, typically used when configuration is initialized, such as The configuration will load Core-default.xml and core-site.xml two resource as Defaultresource, And its subclass hdfsconfiguration will load Hdfs-default.xml and hdfs-site.xml as Defaultresource

Defaultresource is a static type, that is, all the configuration in the Hadoop system will share the Defaultresource, so consider thread safety. Defaultresource is the copyonwritearraylist type

Copyonwritearraylist is a thread-safe ArrayList type that uses copy onwrite in a way that is generally suitable for reading and writing less, and if you write more, copy on write will copy too many times, affecting performance

In addition to AddResource, you can also modify the properties using the Set method of the configuration, in order to preserve these modifications, there is overlay in the configuration

In general, the configuration of Hadoop is not complex, and it mainly maintains some properties for Hadoop computing.

HDFs Source Code Analysis first: Hadoop configuraion