The most bull hadoop2.2 in history. Pseudo-distributed HDFS data migration

1. Requirements :

HDFs cannot upload files due to 99% occupancy on disk where Hadoop data files reside

Exception : Org.apache.hadoop.ipc.remoteexception:java.io.ioexception:file/user/root/input could only being replicated to 0 nodes, instead of 1

[Email protected] hadoop]# df-h

Filesystem Size used Avail use% mounted on

/dev/mapper/vg_greenbigdata4-lv_root

50G 49.2G 880M 99%/

Tmpfs 7.8G 0 7.8G 0%/dev/shm

/DEV/SDA1 485M 64M 396M 14%/boot

/dev/mapper/vg_greenbigdata4-lv_home

860G 36G 781G 5%/Home

can see / under directory 50G and /home under 860G, so migrate directory

2. Original Hadoop configuration

<configuration>

<property>

<name>fs.defaultFS</name>

<value>hdfs://idata-slave3:9010</value>

</property>

<property>

<name>hadoop.tmp.dir</name>

<value>/usr/dfs/tmp</value>

</property>

</configuration>

3. Stop the Hadoop cluster

stop-all.sh

4. New Hadoop configuration

<configuration>

<property>

<name>fs.defaultFS</name>

<value>hdfs://idata-slave3:9010</value>

</property>

<property>

<name>hadoop.tmp.dir</name>

<value>/home/dfs/tmp</value>

</property>

</configuration>

5. Data Migration

Raw Data Directory :

[Email protected] ~]# cd/usr/dfs/tmp/

[Email protected] tmp]# pwd

/usr/dfs/tmp

[[email protected] tmp]# ls

DFS hadoop-unjar5031704250755207519 hadoop-unjar7185041879037748832

hadoop-unjar2609195906982650560 hadoop-unjar5034268358752828831 Nm-local-dir

The new path :

[Email protected] hadoop]# Mkdir/home/dfs

Copy :

[Email protected] hadoop]# cp-r/usr/dfs/tmp//home/dfs/

6. Start the Hadoop cluster

start-all.sh

Output Information :

This script is Deprecated. Instead Use start-dfs.sh and start-yarn.sh

Slf4j:class path contains multiple slf4j bindings.

Slf4j:found Binding in [jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j /impl/staticloggerbinder.class]

Slf4j:found Binding in [jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/mahout-core-0.9-job.jar!/org/ Slf4j/impl/staticloggerbinder.class]

Slf4j:found Binding in [jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/mahout-examples-0.9-job.jar!/ Org/slf4j/impl/staticloggerbinder.class]

Slf4j:see http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

Slf4j:actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

Starting namenodes on [Idata-slave3]

Idata-slave3:starting Namenode, logging to/usr/lib/hadoop-2.2.0/logs/hadoop-root-namenode-idata-slave3.out

Idata-slave3:slf4j:class path contains multiple slf4j bindings.

Idata-slave3:slf4j:found Binding in [jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/ Slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/staticloggerbinder.class]

Idata-slave3:slf4j:found Binding in [jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/ Mahout-core-0.9-job.jar!/org/slf4j/impl/staticloggerbinder.class]

Idata-slave3:slf4j:found Binding in [jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/ Mahout-examples-0.9-job.jar!/org/slf4j/impl/staticloggerbinder.class]

Idata-slave3:slf4j:see http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

Idata-slave3:slf4j:actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

Idata-slave3:starting Datanode, logging to/usr/lib/hadoop-2.2.0/logs/hadoop-root-datanode-idata-slave3.out

Idata-slave3:slf4j:class path contains multiple slf4j bindings.

Idata-slave3:slf4j:found Binding in [jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/ Slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/staticloggerbinder.class]

Idata-slave3:slf4j:found Binding in [jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/ Mahout-core-0.9-job.jar!/org/slf4j/impl/staticloggerbinder.class]

Idata-slave3:slf4j:found Binding in [jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/ Mahout-examples-0.9-job.jar!/org/slf4j/impl/staticloggerbinder.class]

Idata-slave3:slf4j:see http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

Idata-slave3:slf4j:actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

Starting secondary namenodes [0.0.0.0]

0.0.0.0:reverse mapping checking getaddrinfo for localhost [127.0.0.1] failed-possible break-in attempt!

0.0.0.0:starting Secondarynamenode, logging to/usr/lib/hadoop-2.2.0/logs/ Hadoop-root-secondarynamenode-idata-slave3.out

0.0.0.0:slf4j:class path contains multiple slf4j bindings.

0.0.0.0:slf4j:found Binding in [jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/ Org/slf4j/impl/staticloggerbinder.class]

0.0.0.0:slf4j:found Binding in [Jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/mahout-core-0.9-job.jar !/org/slf4j/impl/staticloggerbinder.class]

0.0.0.0:slf4j:found Binding in [jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/ Mahout-examples-0.9-job.jar!/org/slf4j/impl/staticloggerbinder.class]

0.0.0.0:slf4j:see http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

0.0.0.0:slf4j:actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

Slf4j:class path contains multiple slf4j bindings.

Slf4j:found Binding in [jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j /impl/staticloggerbinder.class]

Slf4j:found Binding in [jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/mahout-core-0.9-job.jar!/org/ Slf4j/impl/staticloggerbinder.class]

Slf4j:found Binding in [jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/mahout-examples-0.9-job.jar!/ Org/slf4j/impl/staticloggerbinder.class]

Slf4j:see http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

Slf4j:actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

Starting Yarn Daemons

Starting ResourceManager, logging to/usr/lib/hadoop-2.2.0/logs/yarn-root-resourcemanager-idata-slave3.out

Slf4j:class path contains multiple slf4j bindings.

Slf4j:found Binding in [jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j /impl/staticloggerbinder.class]

Slf4j:found Binding in [jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/mahout-core-0.9-job.jar!/org/ Slf4j/impl/staticloggerbinder.class]

Slf4j:found Binding in [jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/mahout-examples-0.9-job.jar!/ Org/slf4j/impl/staticloggerbinder.class]

Slf4j:see http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

Slf4j:actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

Idata-slave3:starting NodeManager, logging to/usr/lib/hadoop-2.2.0/logs/yarn-root-nodemanager-idata-slave3.out

Startup success

7. Testing

[Email protected]ave3 hadoop]# HDFs dfs-ls/

Slf4j:class path contains multiple slf4j bindings.

Slf4j:found Binding in [jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j /impl/staticloggerbinder.class]

Slf4j:found Binding in [jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/mahout-core-0.9-job.jar!/org/ Slf4j/impl/staticloggerbinder.class]

Slf4j:found Binding in [jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/mahout-examples-0.9-job.jar!/ Org/slf4j/impl/staticloggerbinder.class]

Slf4j:see http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

Slf4j:actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

Found 6 Items

Drwxr-xr-x-root supergroup 0 2014-10-24 15:57/flume

Drwxr-xr-x-root supergroup 0 2014-11-14 14:41/hbase

Drwxr-xr-x-root supergroup 0 2014-11-11 12:58/test

Drwxr-xr-x-root supergroup 0 2014-11-27 17:11/tmp

Drwxr-xr-x-root supergroup 0 2014-10-11 11:14/tmp-output101

Drwxr-xr-x-root supergroup 0 2014-10-28 16:02/user

Upload :

[Email protected] ~]# HDFs dfs-put sparkdata/.

Slf4j:class path contains multiple slf4j bindings.

Slf4j:found Binding in [jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j /impl/staticloggerbinder.class]

Slf4j:found Binding in [jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/mahout-core-0.9-job.jar!/org/ Slf4j/impl/staticloggerbinder.class]

Slf4j:found Binding in [jar:file:/usr/lib/hadoop-2.2.0/share/hadoop/mapreduce/lib/mahout-examples-0.9-job.jar!/ Org/slf4j/impl/staticloggerbinder.class]

Slf4j:see http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

Slf4j:actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

Drwxr-xr-x-root supergroup 0 2014-11-04 15:14 _sqoop

Drwxr-xr-x-root supergroup 0 2014-10-23 17:08 Flume

Drwxr-xr-x-root supergroup 0 2014-10-16 16:22 Forest

Drwxr-xr-x-root supergroup 0 2014-10-16 16:25 od

Drwxr-xr-x-root supergroup 0 2014-11-11 15:10 PFP

Drwxr-xr-x-root supergroup 0 2014-11-07 16:07 recommend

Drwxr-xr-x-root supergroup 0 2014-11-27 17:44 Sparkdata

Drwxr-xr-x-root supergroup 0 2014-11-18 17:19 Temp

This allows data to be migrated without the need to format HDFs to prevent loss of data

The most bull hadoop2.2 in history. Pseudo-distributed HDFS data migration