HDFS copy Mechanism & Load Balancing & Rack Awareness & access methods & robustness & deletion recovery mechanism & HDFS disadvantages

Source: Internet
Author: User
Tags hadoop fs

Copy Mechanism

1. Copy placement policy

The first copy is placed on the datanode of the uploaded file. If it is submitted outside the cluster, a node with a low disk speed and a low CPU usage will be randomly selected;
The second copy is placed on nodes in different racks of the first copy;
Third copy: different nodes in the same rack as the second copy;
If there are more copies: randomly placed in the node;

2. Copy Coefficient

1) When uploading a file to HDFS, the copy coefficient of hadoop at that time is a few, then the number of block copies of this file will be several, no matter how to change the system copy coefficient in the future, the number of copies of this file will not change, that isThe number of file copies uploaded to the HDFS system is determined by the number of system copies at that time and will not be changed due to the change of the copy coefficient.;

2) You can specify the copy coefficient when uploading files,DFS. replication is a client attribute and does not specify the default number of copies used for replication. After the file is uploaded, the number of backups is determined and DFS is modified. replication does not affect the previous files or the files with the specified number of backups. It only affects the files with the default number of backups.;

3)Replication is determined by the client by default.If the client is not set, it will be read from the configuration file;

hadoop fs setrep 3 test/test.txthadoop fs -ls test/test.txt

The copy coefficient of replicated test.txt is 3, but the number of backup blocks is still 1 (assuming the default value of DFS. replication is 1 ).

4) if you have set DFS in the hdfs-site.xml. replication = 1, this is not necessarily the number of backups of the block is 1, because the hdfs-site.xml may not be added to the classpath of the project, then our program runs the DFS. replication may be a DFS in the hdfs-default.xml. replication. The default value is 3. This may be why you are running DFS. replication is always 3.

 

 

Server Load balancer

Hfds data is not evenly distributed on each DN. A common reason is that a dn node is often added to an existing cluster, when a new data block is added (the data of a file is stored in a series of blocks), NN takes into account many factors before choosing DN to receive the data block:

1) place a copy of the data block on the node where the data block is being written;

2) try to place different copies of data blocks on different racks, so that the cluster can continue to work even if a rack is completely lost;

3) Distribute the HDFS data evenly on the DN of the cluster as much as possible;

 

Role of Server Load balancer:Data is evenly distributed across various datanode to balance Io performance and balance Io, average data, and cluster to prevent hotspot occurrence.;

 

Why does hadoop suffer from load imbalance?

Hadoop only takes into account the chunks instead of the chunks;

For example, assume that there are 10 pieces of data on datanode1, and datanode2 only has 2 pieces of data, which may cause high Io of datanode1, resulting in unbalanced Cluster load;

 

How does hadoop implement load balancing?

Copying data between hadoop balancer datanode instances consumes a large amount of bandwidth. By default, 1 Mbit/s of bandwidth is used for data copying, which does not affect the cluster bandwidth to a large extent; when the cluster is busy, do not perform load balancing. The execution time of Load Balancing may be long and there is no time to remind you until the execution is complete;

 

 

Rack awareness

Generally, large hadoop clusters are organized in the form of racks. The network conditions of different nodes on the same Rack are more ideal than those between different racks; namenode tries to save data block copies on different racks to improve fault tolerance;

HDFS cannot automatically determine the network topology of each DN in the cluster. hadoop allows the Cluster Administrator to configure DFS. network. script Parameters are used to determine the rack of the node. The configuration file provides the translation from IP to rackid. Nn knows the rackid of each DN machine in the cluster through this configuration. If topology. Script. file. Name is not set, each IP address is translated into/default-rack.

On the way, D and R are vswitches and H are DN;

Then, rackid =/D1/R1/H1 of H1 has the rackid information (these rackid information can be obtained through topology. script. file. name configuration) to calculate the distance between the two datanode:
Distance (/D1/R1/H1,/D1/R1/H1) = 0 same datanode
Distance (/D1/R1/H1,/D1/R1/H2) = 2 different datanode from Rack
Distance (/D1/R1/H1,/D1/R1/H4) = 4 different datanode from IDC
Distance (/D1/R1/H1,/D1/R1/H7) = 6 datanode under different IDCs

Note: 1) When no rack information is configured, hadoop of all machines is named "/default-rack" under the same default rack by default. In this case, any datanode machine, whether physically or not in the same rack, is considered to be in the same rack. 2) Once topology is configured. script. file. name, which searches for datanode and topology based on the network topology. script. file. the value of the configuration option name is specified as an executable program, usually a script.

 

 

HDFS access method

1) Shell

2) Java

3) webui

4) restful

 

 

HDFS file deletion and recovery mechanism

When a file is deleted from HDFS, the file is not immediately deleted from HDFS, but renamed to the/trash directory, as long as the file is still in the/trash directory, this file can be quickly recovered;

The file storage time in the/trash directory can be configured (6 hours by default). When this time is exceeded, NN will delete the file from the namespace;

Deleting a file will release data blocks related to the file. Note: there will be a certain delay between the increase in the number of files deleted by the user and the increase in HDFS free space;

Configure the Time of the recycle bin: hdfs-site.xml

<property>    <name>fs.trash.interval</name>    <value>1440</value></property>

The time unit is seconds. the recycle bin location is/user/$ user/. Trash/current/on HDFS/

 

 

HDFS system disadvantages: not suitable for processing small files

1) nn places the metadata of the file system in the memory, so the number of files that the file system can accommodate is determined by the memory size of NN. The more small files, the larger the memory occupied by NN;

Generally, each file/folder and block occupy about 150 bytes of space. Therefore, if there are million files and each occupies one block, at least MB of memory is required;

At present, millions of files are still feasible. When it is extended to billions, it is difficult to achieve the previous hardware level;

2) The number of map tasks is determined by the split. When mapreduce processes a large number of small files, too many map tasks will be generated, and the thread management overhead will be the job time;

For processing ipvm files, if each split is 1 MB, there will be 10000 map tasks with a large thread overhead; if each split is 100 MB, there are only 100 map tasks, and each task processes more data, reducing the thread management overhead;

 

HDFS copy Mechanism & Load Balancing & Rack Awareness & access methods & robustness & deletion recovery mechanism & HDFS disadvantages

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.