Let You Know What Hadoop Is in One Minute

What is hadoop?

Hadoop is a software platform for analyzing and processing big data. It is a framework of Apach's open source software implemented in Java language. It implements distributed computing for massive amounts of data in a cluster of large numbers of computers.

 

The core design of the Hadoop framework is: HDFS and MapReduce. HDFS provides storage for massive amounts of data, and MapReduce provides calculations for massive amounts of data.

 

The process of big data processing in Hadoop can be understood by referring to the following simple diagram: Data is the result of processing through the Hadoop cluster.


 

HDFS: Hadoop Distributed File System, Hadoop's distributed file system.

 

Large files are divided into data blocks of 64M by default and stored in cluster machines. The file data1 in the following figure is divided into 3 blocks, which are distributed in different machines in a redundant mirroring manner.



MapReduce: Hadoop creates a task for each input split to call Map calculation. In this task, each record in this split is processed in turn. Map will output the result in the form of key-value, and hadoop is responsible for pressing the key value. The output of the map is sorted and used as the input of Reduce, and the output of the Reduce Task is the output of the entire job, which is stored on HDFS.


The Hadoop cluster is mainly composed of NameNode, DataNode, Secondary NameNode, and JobTracker.


The NameNode records how the files are split into blocks and these blocks are stored in those DateNode nodes. The NameNode also saves the status information of the file system operation. The DataNode stores the split blocks. The Secondary NameNode helps the NameNode collect files The status information of the system operation. The JobTracker is responsible for the operation of the Job when a task is submitted to the Hadoop cluster and is responsible for scheduling multiple TaskTrackers. The TaskTracker is responsible for a map or reduce task.