Design of HADOOP HDFs

Hadoop provides a way to handle data on its HDFs, in the following ways:

1 batch processing, MapReduce

2 Real-time processing: Apache storm, spark streaming, IBM streams

3 Interactive: Like pig, spark Shell can provide interactive data processing

4 Sql:hive, Impala provides interfaces that can be used in SQL standard language for data query analysis

5 iterative processing: In particular, machine learning-related algorithms, which require repeated data on data, MapReduce does not adapt to this method of calculation, it always output its intermediate results to disk, so it is inefficient to handle, new spark well support this way.

6 Search: Use SOLR to index files stored on HDFs to help them quickly document searches

HPC (high performance computing) typically distributes tasks to a clustered computer using a SAN (shared storage network), which is beneficial for computationally intensive jobs, but is limited for IO-intensive because the SAN does not provide good IO concurrency.

Hadoop streaming is the use of similar unix/linux above the standard input and output stream ideas, suitable for the text of the Mr Processing, without writing Java programs, you can use Shell,python,ruby and so on.

Its principle is to use Java to implement a wrapper user program Mr Program, the program is responsible for invoking the MapReduce Java interface to get Key/value to the input, create a new process to start the wrapper user program, the data passed through the pipeline to the packaging User program processing, The MapReduce Java interface is then called to cut the output of the user program into key/value pairs of outputs.

1 Hadoop streaming can only process text data by default and cannot process binary data directly

The mapper and reducer in 2Streaming can only write data to the standard output by default and cannot be easily multiplexed out

Hadoop streaming is a programming tool provided by Hadoop that allows users to use any executable file or script file as mapper and reducer, for example:

Use some of the commands in the Shell scripting language as Mapper and reducer (cat as MAPPER,WC as reducer)

$HADOOP _home/bin/hadoop jar $HADOOP _home/contrib/streaming/hadoop-*-streaming.jar \

-input myinputdirs \

-output myoutputdir \

-mapper cat \

-reducer WC

The purpose of HDFS design is

Large files: Hundreds of M,GB,PB?

Streaming data access: Suitable for write-once, multiple-read scenarios

General commercial hardware: can accept damage from normal hardware without high-cost storage

Not applicable to:

1. Low-latency data access recommendations using HBase

2. A large number of small files

Consumes a lot of NN memory to store

For example, 1 million files (one block per file) will take approximately 300M of memory. A file uses 150bytes.

Question: How do I check if there are a large number of small files in your HDFs cluster?

Why are files cut into chunks?

1. Files that can hold file sizes larger than a single disk

2. Block is a fixed logical size, simplifying the design of the storage system

3. Improved availability and fault tolerance

Distributed processing according to the block, the file does not block to improve the efficiency of distributed computing, a piece of error, only need to process a single block, not to process the entire file.

Why are the blocks so big?

Reduce the time wasted on finding files and improve execution efficiency. (If a block is too small, for example, in mapreduce processing, the overhead of other operations exceeds the block data processing itself, the storage is also)

Design of HDFS HA

1. There is a shared storage directory to store edit log. Via QJM or NFS

2. Datanodes to send the block mapping to the two namenodes, because the block mapping is not serialized on the disk and is stored in the memory of the NN.

3. The client needs to have a mechanism to handle the switchover of the NN, which is transparent to the user, who does not care about the current active NN when requesting access to the file.

4. Secondnamenode's role is taken over by the standby NN, which periodically generates checkpoints (that is, merge the current fsimage with edit log and pass it on to active NN)

Read the document in detail

The client-generated Dfs object accesses Namenode,namenode through RPC to return some file blocks. In other words, a file block will have multiple backups, which one to return?

The Namenode returns an ordered Datanodes based on the configured network topology. The first one recently, generally read the first datanode. A file may have multiple blocks,

Is the wholesale sent to the client, after reading a batch, and then return to the next batch. Read blocks sequentially.

At the current design time, the client is under the guidance of Namenode, according to the nearest Datanode returned, and then the client establishes a connection directly with Datanode to read the data.

Write a detailed document

1. Client sends request to NN to create file via RPC

2. NN check if file exists and check user write permissions

3. If the file is created by checking, and returned to the client Fsdataoutputstream object for the client to write the data

4. Datastreamer to the NN request block, and then be assigned block datanodes form pipeline to write the data

5. What happens if there is a datanode failure in pipe line during the transfer of data?

The current pipeline is closed, the problematic datanode will be removed from the Pipelie, and the last block will be deleted the next time it is started.

The remaining Dataodes form a new pipeline, and the remaining data is closed, and the file is identified as not meeting the replica requirements and will be synchronized later.

In the case of multiple dataodes failures, if the minimum dfs.namenode.replication.min copy is met (by default, 1), even if the file is written successfully.

It is then replicated asynchronously to meet the requirements of the replica.

Consistency model

That is, the visibility of files in the file system when reading and writing files.

When we write a file to HDFs, we can see the file in the file system, but at the beginning, the file is 0 in length.

The contents are not visible until the first block is written.

A block that is being written is not visible.

If you do not call the HSync command, the data is stored in Datanode memory, which is a point to note.

Design for HADOOP HDFs