Hadoop Study Notes (7): Using distcp to copy big data files in parallel

Previously we introduced that the methods for accessing HDFS are single-threaded. hadoop has a tool that allows us to copy a large number of data files in parallel. This tool is distcp.

A typical application of distcp is to copy files in two HDFS clusters. If the two clusters use the same hadoop version, you can use the HDFS identifier:
% Hadoop distcp HDFS: // namenode1/Foo HDFS: // namenode2/Bar
This command copies the/Foo directory in the first cluster (namenode is the namenode1 specified in the Command) to the/bar directory in the second cluster, therefore, the directory structure such as/BAR/foo is obtained in the second cluster. We can also specify multiple copy sources, but there is only one copy destination. Note that the absolute path must be used to specify the copy path.

The distcp command is implemented in the form of a mapreduce job, but this job does not have a reduce task. Each file is copied by a map task. distcp tries its best to import each file with the same size to the same map task. In this way, the data volume copied by each map task is roughly the same.
The number of map tasks is determined as follows:
1) Considering the overhead of creating each map task, each map task should process at least 256 MB of data (if the total input file size is less than MB, then, all the input data is handed over to a map task for execution ). For example, a 1 GB input data is assigned four map tasks to copy.
2) If the data to be copied is large, it cannot be divided by only the MB input data of each map task, because it may need to create many map tasks. This can be divided by every datanode20 map tasks. For example, if there are 100 GB of input data and 100 nodes, 2000*20 = map tasks will be started to copy data, each map task copies mb data. You can also use the-M option to specify the number of maps to be used. For example,-M 1000 only starts 1000 map tasks, and each map task copies 1 GB of data.

By default, if a file with the same name already exists at the copied destination, the files are skipped by default. You can use the-overwrite option to overwrite files of the same name, or use the-update option to update files of the same name.
For more usage of distcp, run the "hadoop distcp" command without adding parameters to view its usage.

If the hadoop versions of the two clusters are inconsistent, HDFS identifiers cannot be used to copy files because the RPC systems of the two clusters are incompatible. You can use a read-only HTTP-based hftp File System to read source data, as shown below (note that this command is executed on the second cluster to Ensure RPC version compatibility ):
% Hadoop distcp hftp: // namenode1: 50070/Foo HDFS: // namenode2/Bar
Note that the network port of namenode1 needs to be specified in the preceding command. It is specified by DFS. http. Address. The default value is 50070.

Another option is to use the webhdfs protocol (replacing the hftp protocol), so that HTTP can be used in both the source and destination of the copy, without worrying about version incompatibility:
% Hadoop distcp webhdfs: // namenode1: 50070/Foo webhdfs: // namenode2: 50070/Bar

We can also use the hdfs http proxy as the source and destination of the copy, so that we can set the firewall and implement bandwidth control.

Reprinted please indicate the source:Http://www.cnblogs.com/beanmoon/archive/2012/12/18/2823581.html