Hadoop source Reading

1. Function Analysis of Hadoop package

2. Because of the need for communication between the MapReduce and HDFs of Hadoop, the object of communication needs to be serialized. Instead of using Java serialization, Hadoop introduces its own system. A large number of serializable objects are defined in Org.apache.hadoop.io, and they all implement the writable interface.

3, after the introduction of Org.apache.hadoop.io, we began to analyze the ORG.APACHE.HADOOP.IPC. RPC takes client/server mode.

4, since it is RPC, naturally with the client and the server, of course, ORG.APACHE.HADOOP.IPC also has the class client and Class Server. Here we examine the org.apache.hadoop.ipc.Client carefully. The following diagram contains the key classes and key methods in Org.apche.hadoop.ipc.Client.

5, chat The client Chat server, according to the Convention, first put the class diagram. It is important to note that the server class here is an abstract class, and the only abstraction is the call method.

6, with the client and server, it is natural to RPC. The next round is Rpc.java.

7. A typical HDFS system consists of a namenode and multiple datanode. Namenode maintains the namespace, while Datanode stores data blocks.

Let's take a concrete look at the implementation of Datanode.

8, before continuing to analyze the datanode, we need to look at the system's working state. First of all understand the above state diagram, it is below we want to introduce the basis of Datanode.

9. Let's take a look at what happens on Datanode when upgrading, rolling back, and committing.

10, after analysis of storage related classes, we look at the next big guy, fsdataset related classes.

When we introduce storage, we do not involve block operations, and all data-block related operations are handled in Fsdataset related classes. The following is a class diagram.

11, through the above series of introductions, we know the Datanode work when the file structure and file structure in memory corresponding to the object. Below we can analyze the dynamic behavior on Datanode. First we analyze dataxceiverserver and dataxceiver. Datanode the acceptance of the data block, send does not take the RPC mechanism we described earlier, the reason and simple, RPC is a command-style interface, and datanode processing data part, is often a streaming mechanism. Dataxceiverserve and Dataxceiver are the implementation of this mechanism. Among them, Dataxceiver also relies on two auxiliary classes: Blocksender and Blockreceiver. The following is a class diagram.

12, continue dataxceiver analysis, the next piece of nut is to write data. The write data operation of HDFs, the pen reads the data to copy n many times. Let's look at the request to write the data first. The next process is data processing and validation, which are written to block files and data block meta-files, respectively.

13, Dataxceiver supported six operations, we have analyzed the most important two. The rest is: Read the data block meta-file, replace a block of data, copy a block of data, read the data block check code. Let's discuss it one by one.

14, through the above discussion, Datanoda on the reading, writing process has been basically clear, we look at the next non-main process, Datablockscanner used to periodically check the data block. The class diagram is as follows:

15, the surrounding obstacles cleared, we can begin to analyze class Datanodde. The class diagram is as follows:

16, the introduction of Datanode. We started analyzing namenode. Compared to Datanode,namenode is more complex, the system only a namenode, do

Is the manager of the System files directory and the "Inode" table. In order to improve usability, there are still namenode from the system.

When we analyzed Datanode earlier, we focused on data blocks. Namenode as the manager of the file directory and file allocation in HDFs, the most important information it holds is the following two mappings:

Data block, file name

Data Block->datanode List

17, we first analyze the Inode.java, the class Inode abstract file hierarchy. If we do object-oriented abstraction of the filesystem, we will get the same NVC as the following (class Inode): The Inode is an abstract class whose two subclasses correspond to the directory (inodedirectory) and the file (Inodefile) respectively.

18, before we mentioned the relationship: file name, data block persistence on disk, all changes to the directory tree update and file name-to-block relationship must be persisted, in order to ensure that each modification does not need to re-save the entire structure, HDFS uses the operation log, save the update.

19, we analyze Fseditlog.java, this class provides namenode operation log and log file related methods, the related class diagram is as follows: through the above analysis, we should know what information is recorded in the log file.

20, we began to analyze the lease lease, the following is the class diagram. Lease is a file can be considered a write lock, when the client needs to write a file, it needs to apply for a lease,namenode responsible for recording which file on which the Lease,lease customer is who, time-out, etc., all of these work by the following three classes to complete

21, below we analyze fsdirectory. In fact, the best place to analyze fsdirectory, should be introduced after the Inode, fsdirectory on the basis of the Inode, the file directory status of HDFs saved. When the system loads the Fsimage, Fsimage will re-establish the file directory status on the Fsdirectory object, change the status of the HDFs file directory, and also have the fsdirectory write log, and it holds the mapping of the data block.

22, down the turn to Fsnamesystem appearances. Fsnamesystem.java a total of 4573 lines, and the entire Namenode directory under all the Java program altogether has 16876 lines, the Fsnamesystem has done, Namenode also basically has done.

Fsnamesystem is the place where Namenode actually records information, and the data stored in Fsnamesystem are:

Data list by file name (stored in fsimage and logs)

List of valid data blocks (inverse relationship above)

Block->datanode (only available in memory, based on information sent by DataNode)

Block of data stored on the Datanode (inverse relationship of the above relationship)

Datanode (LRU) that recently sent heartbeat information

Let's start by analyzing the member variables of the Fsnamesystem.

23, continue to analyze the fsnamesystem. The threads on the Namenode, respectively, correspond to the datanode heartbeat check, lease check, Safe mode check, and block copy, and we will describe the functions that these threads correspond to later. Next is a bunch of system parameters, such as the maximum allowable number of data blocks per Datanode node, heartbeat interval, etc...

The Safemodeinfo class diagram is as follows,

After discussing Safemodeinfo, let's analyze Safemodemonitor, which is used to periodically check if the system is able to leave Safe mode (Smmthread is an instance of it). When the system leaves Safe mode, the Smmthread is re-assigned to NULL.

24, private Host2nodesmap, defines the network topology of HDFs,

25, we then analyze the member variables of Namenode.java, then two classes together, analyze the interface it provides, and with the interface to explain the request corresponding processing flow.

Let's look at the start-up process for namenode. The main method is the entrance to the system.

26, everything is ready, we can analyze the process on the Namenode.

First we look at the implementation of Namenode on the ClientProtocol, the client through this interface can operate on the directory tree, open, close files and so on.

27, soft persimmon all pinched, we began to chew nut. Getblocklocation,create,append,setreplication,setpermission and SetOwner have been analyzed before, and we'll discuss the actions related to file content.

28. The following are the operations associated with the directory tree.

Here are the methods for maintaining management with other systems.

29, Fix ClientProtocol, Next is the Datanodeprotocol part. The interface is as follows:

With these basics, let's look at the process of fssystem.handlerheartbeat.

The following is an analysis of how an answered imperative is constructed.

30, continue to namenode implementation of the interface to do analysis

So far, we have completed the analysis of ClientProtocol and Datanodeprotocol on Namenode, Namenodeprotocol, we will analyze the namenodeprotocol when we understand it.

31, in addition to the external interface, Namenode also defined a series of threads, constantly check the state of the system, the following are the functional analysis of these threads.

32, moved into secondary NameNode, the previous analysis we also call him from NameNode, from NameNode in HDFs is a small supporting role.

The classes that are related to secondary namenode are not many, such as.

The first thing to discuss is the communication between Namenode and secondary namenode. Interface Namenodeprotocol is implemented on Namenode, which is used for command communication between Namenode and secondary namenode.

33, the Secondarynamenode member variable is very few, mainly has:

The specific process is:

34. We can begin to understand HDFs from the outside of the system, and Dfsclient provides the basic functionality to connect to the HDFS system and perform file operations. Dfsclient is also a big guy, let's analyze some of its inner classes first. Let's look at Leasechecker first. Lease is a client to the file write operation time a credential, before analysis Namenode, already understood the lease, inodefileunderconstruction relationship, inodefileundercontruction only in the file write Time exists. The client's lease management is simple, including the addition of the Add and remove methods, the Run method is executed on a regular basis, and the lease is automatically extended through the ClientProtocol renewlease.

Next we analyze the classes introduced internally for file reads.

35. Next, of course, the analysis output stream

36, with the above basis, we can come to dissect dfsoutputstreamle. Look at the constructor first.

37, the previous analysis of the dfsclient inner class, occupy the implementation of this class part of the 2/3, we look at the rest.

=================================

1. Introduction to MapReduce

2. Next we analyze the two sub-classes of task, Maptask and Reducetask. Maptask's related

Maptask is not complex, but it is complex to support maptask work of some auxiliary classes.

The most important method of Maptask is run.

Next we analyze Runoldmapper

3, Maptask auxiliary class mainly for the input and output of mapper. First, let's look at the mapper input in Maptask.

4, with the above mapper output memory storage structure and hard disk storage structure discussed. Let's carefully analyze the mapoutputbuffer process.

The first is the member variable.

5, the next discussion is the output of the Key,value, this part is more complex, but with the front kvstart,kvend and kvindex with the analysis, to help us understand this part of the code.

6. Internal classes and auxiliary classes of tasks. From the previous diagram, we can see that the task has many internal classes and has a large number of member variables that work with the task to do the job

7, map results, will be distributed through partition to reducer, reducer after the reduce operation, through the OutputFormat, output, the following we will analyze the class involved in the process.

8, mapper output, before sending to reducer is stored in the local file system, IFile provides management of the mapper output. We already know that the mapper output is the <key,value> key value pair, ifile stored this data in the form of record <key-len,value-len,key,value>. In order to preserve the bounds of the key-value pairs, it is natural ifile to save Key-len and Value-len.

9. We started to analyze the internal operating mechanism of Hadoop MapReduce.

Let's start by analyzing some auxiliary classes, first the class with ID, and the number of IDs inherited as follows:

10, the previous analysis of the org.apache.hadoop.mapreduce has been completed, this package provides a Hadoop mapreduce part of the application API for users to implement their own mapreduce application. But these interfaces are for future mapreduce applications, and the current MapReduce framework is still used by the old system. Let's analyze org.apache.hadoop.mapred. Starting with the mapred's MapReduce framework, the following class diagram

11. Next we install the sequence of data flow in the MapReduce process to decompose the relevant content of org.apache.hadoop.mapreduce.lib.* and introduce the corresponding function of the base class. The first is the input section, which implements the data entry portion of the MapReduce. The class diagram is as follows:

The more important methods of class are

12. In the MapReduce framework of Hadoop, map actions are abstracted through the Mapper class. In general, we will implement our own special mapper, and register to the system, when executed, our mapper will be called by the MapReduce framework. The Mapper class is simple, including an inner class and four methods, and the static structure diagram is as follows

13, with the analysis of the previous section, we look at the specific interface, they are in the package org.apache.hadoop.mapreduce.

Hadoop source Reading