Hadoop officially learns---Hadoop

One: Course structure

II: What is Hadoop

Hadoop is the platform for distributed storage and computing for big data

Three: Distributed storage of data

Four: Concepts in Hadoop

In distributed storage System, the data scattered in different nodes may belong to the same file, in order to organize a large number of files, the files can be placed in different folders, folders can be included at a level. We call this organization name space (namespace). The namespace manages all the files in the entire server cluster.

Different nodes in the cluster assume different responsibilities. The node responsible for the namespace is called the Master node, and the node responsible for storing the real data is called the slave node (slave nodes). The master node is responsible for managing the file structure of the filesystem, which is responsible for storing the real data, called the master-slave structure (master-slaves). When a user operates, it should first deal with the master node, querying what data is stored on the node and then reading from the node. In the master node, in order to speed up user access, the entire namespace information will be placed in memory, when the more files stored, the master node needs more memory space. When storing data from a node, some raw data files may be very large, some may be small, the size of the file is not easy to manage, then you can abstract a separate storage file units, called blocks (block). The data is stored in the cluster, possibly because of network reason or the node hardware causes the access failure, it is best to use the copy (replication) mechanism, the data is backed up to multiple nodes at the same time, so that the data is safe, the probability of data loss or access failure is small.

V: Architecture of the HDFS2

Responsible for distributed storage of data

Master-Slave structure
Master node, there can be 2: namenode
From the node, there are a number of: Datanode

Namenode is responsible for:
Receives user action request, is the user operation's entrance
Maintaining the directory structure of a file system, called a namespace

Datanode is responsible for:
Storing files

VI: Yarn's architecture

Scheduling and management platform for resources

Master-Slave structure
Master node, there can be 2: ResourceManager
From the node, there are a number of: NodeManager

ResourceManager is responsible for:
Allocation and scheduling of cluster resources
For applications such as MapReduce, Storm, and Spark, the Applicationmaster interface must be implemented to be managed by RM

NodeManager is responsible for:
Management of single node resources

VII: The architecture of MapReduce

Batch computing model with disk IO dependent

Master-Slave structure
Master node, only one: Mrappmaster
Mrappmaster is responsible for:
Receive client-submitted calculation tasks
Assign compute tasks to tasktrackers execution, which is task scheduling
Monitor the implementation of Tasktracker

Eight: Features of Hadoop

Capacity (scalable): can reliably (reliably) store and process gigabytes (PB) of data.
Low cost (Economical): You can distribute and process data through a server farm consisting of common machines. The total of these server farms is up to thousands of nodes.
High efficiency (efficient): By distributing data, Hadoop can process them in parallel (parallel) on the node where the data resides, which makes processing very fast.
Reliability (Reliable): Hadoop can automatically maintain multiple copies of data and automatically redeploy (redeploy) compute tasks after a task fails.

Hadoop formally learns---hadoop