Hadoop source code analysis (1)

 

GoogleIts core competitive technology is its computing platform.GoogleThe following are some examples5ArticleArticleDescribes their computing facilities.
Googlecluster
Chubby

GFS

Bigtable
Mapreduce

Soon,ApacheThere is a similar solution. Currently, they all belongApacheOfHadoopProject, corresponding:
Chubby --> zookeeper
GFS --> HDFS
Bigtable --> hbase
Mapreduce --> hadoop
CurrentlyOpen SourceThere are still many projects, suchFacebookUsed for User AnalysisHive.
HDFSAs a distributed file system, it is the foundation of all these projects. Good AnalysisHDFSTo help you understand other systems. BecauseHadoopOfHDFSAndMapreduceIt is the same project, so we put them together for analysis.

 

Is the top-level package diagram of the entire mapreduce project and their dependencies. The dependency between hadoop packages is complex because HDFS provides a distributed file system that provides APIs to shield local and distributed file systems, even online storage systems like Amazon S3. This results in the implementation of the Distributed File System or the underlying implementation of the Distributed File System, depending on some seemingly high-level functions. The mutual reference of functions leads to Spider-type dependency. A typical example is the package Conf. conf is used to read system configurations. It depends on FS. It is mainly used to read configuration files, and file systems are used to read some file system functions, it is abstracted in the package FS.

The key part of hadoop is focused on the blue part in the figure, which is also the focus of our research.