Distcp usage minutes

Distcp is mainly used to copy data between hadoop clusters.

1. If the haboop version is the same, you can use the following format:

Hadoop distcp HDFS: //

2. If you copy data between different versions of hadoop clusters, you can use the following format:

Hadoop distcp-I hftp: //

Note: At this time, you need to run distcp on the target cluster.-I is a ignore error.

Note that hftp has nothing to do with ftp. It is a protocol packaging that accesses the HDFS file system through http to support data copying between different versions. Its port is not a DFS port, but an http port.

In my application, one is the hadoop1.0.0 cluster and the other is the cloudera cdh3u0 cluster. In this case, copy the data in hadoop1.0.0 to the hdfs of cloudera cdh3u0. Therefore, the hftp distcp is used.

Furthermore, if you only create data without a logical relationship, distcp does not only write teragen or slive fast. In my tests, the hybrid write speed between teragen and slive can reach 300 MB/S, and the network I/O speed can reach 100 + MB/s. In distcp, the disk write speed is 100 Mb/s, and the network Io speed is also 100 + MB/s.

In addition, if you are upgrading the HDFS hadoop version, you can start-DFS-upgrade at startup to upgrade the file system to the new hadoop version. Such as from hadoop-0.19 to hadoop-0.20, but if not the same version, the upgrade is also problematic. For example, I cannot upgrade the hadoop-1.0.0 TO THE cloudera version.

Related Links

Http://hadoop.apache.org/hdfs/docs/current/hftp.html

Http://www.linezing.com/blog? P = 452