Hadoop detailed (d) distcp

The first few articles we talked about were single-threaded operations, if you want to copy a lot of files in parallel, Hadoop provides a gadget distcp, the most common use of which is to copy files between two Hadoop clusters and help the document in detail, without explaining that there are no two clusters in the development environment, Demo with the same cluster:

Hadoop distcp hdfs://namenode:9000/user/hadoop/input HDFS://NAMENODE:9000/USER/HADOOP/INPUT1

Complete list of options:

distcp [Options] <srcurl>* <desturl> OPTIONS:-p[rbugp] Preserve status  
                       R:replication number B:block size U:user                      G:group p:permission-p alone is equivalent to-prbugp-i          Ignore Failures-log <logdir> Write logs to <logdir>-M <num_maps> Maximum number of simultaneous copies-overwrite overwrite destination-update overwr   
                       ITE if src size different from DST Size-skipcrccheck don't use CRC check to determine if SRC different from Dest. Relevant only if-update are SPECIFIED-F <urilist_uri> use list at <urilist_uri& Gt As SRC list-filelimit <n> Limit The total number of files to be <= n-sizelimit <n&Gt Limit the total size of to IS <= n Bytes-delete delete the files existing in the DST but don't in src-m apredsslconf <f> Filename of SSL Configuration for mapper task

Look at the results of Distcp's execution and you'll find that DISTCP is a mapreduce task, but only map has no reducer.

<PRE>13/06/18 

10:59:19 INFO tools. Distcp:srcpaths=[hftp://namenode:50070/user/hadoop/input]  
13/06/18 10:59:19 INFO tools. DISTCP:DESTPATH=HDFS://NAMENODE:9000/USER/HADOOP/INPUT1  
13/06/18 10:59:20 INFO tools. DISTCP:HDFS://NAMENODE:9000/USER/HADOOP/INPUT1 does not exist.  
13/06/18 10:59:20 INFO Tools. Distcp:sourcepathscount=3
13/06/18 10:59:20 INFO tools. distcp:filestocopycount=2
13/06/18 10:59:20 INFO tools. Distcp:bytestocopycount=1.7m  
13/06/18 10:59:20 INFO mapred. Jobclient:running job:job_201306131134_0009  
13/06/18 10:59:21 INFO mapred. Jobclient:  map 0% reduce 0%  
13/06/18 10:59:35 INFO mapred. Jobclient:  map 100% reduce 0%</pre>

Distcp a large stack of documents evenly distributed to map to execute, each file a single map task. So the default is divided into a few map appropriate? First by 256MB average, if the total size below 256mb,distcp will only assign a map. But if the result of the split is more than 20 of the map number of nodes, each node map number will be 20, look at the flowchart:

You can manually set through-M, if for HDFs balance, it is best to set the maps more, the block spread out.

If the version is inconsistent between two clusters, using HDFS may cause an error because the RPC system is incompatible. Then you can use the HFTP protocol based on the HTTP protocol, but the destination address must also be hdfs, like this:

Hadoop distcp hftp://namenode:50070/user/hadoop/input HDFS://NAMENODE:9000/USER/HADOOP/INPUT1

It is recommended to use the HFTP Alternative protocol Webhdfs, both the source and destination addresses can be WEBHDFS and fully compatible

Hadoop distcp webhdfs://namenode:50070/user/hadoop/input WEBHDFS://NAMENODE:50070/USER/HADOOP/INPUT1