Install HDFS 2.7.1 on CentOS 6.6

Install HDFS 2.7.1 on CentOS 6.6

This article tries to build 10 HDFS clusters on CentOS, instead of YARN and Hive, because Spark will be used later. Install jdk 1.8 first, which is not described here.

The server has 12 disks, so this is a real scenario where the cluster is built, but the size is small.

Download

First download the hadoop binary Package

[Plain] view plaincopyprint?

1. Wgethttp: // apache.mesi.com. ar/hadoop/common/hadoop-2.7.1/hadoop-2.7.1.tar.gz

Decompress the package and put it in the/data/slot0/directory, and create a soft link/data/slot0/hadoop

Create an account

Create an account for managing a hadoop Cluster

[Plain] view plaincopyprint?

1. Useraddlisa

The/home/lisa directory appears. The lisa account does not need a password because you cannot log on directly.

You can create lisa accounts for all 10 servers in batches, mainly through ssh-t.

Modify directory owner

[Plain] view plaincopyprint?

1. Chown-Rlisa: lisahadoop-2.7.1/
2. Chown-Rlisa: lisahadoop
3. Ll
4. Total8
5. Drwxr-xr-x3rootroot4096Sep2518: 02 download
6. Lrwxrwxrwx1lisalisa13Sep2517: 42hadoop-> hadoop-2.7.1/
7. Drwxr-xr-x9lisalisa139Jun2914: 15hadoop-2.7.1.
8. Lrwxrwxrwx1rootroot24Sep2516: 50java->/data/slot0/jdk1.8.0 _ 45/
9. Drwxr-xr-x8rootroot4096Sep2514: 01jdk1. 8.0 _ 45

Enable SSH authentication between accounts

Switch to the account used by the hadoop system and generate an ssh key

[Plain] view plaincopyprint?

1. Su-lisa
2. Ssh-keygen-trsa-p''
3. Generatingpublic/privatersakeypair.
4. Enterfileinwhichtosavethekey (/home/lisa/. ssh/id_rsa ):
5. Createddirectory '/home/lisa/. ssh '.
6. Youridentificationhasbeensavedin/home/lisa/. ssh/id_rsa.
7. Yourpublickeyhasbeensavedin/home/lisa/. ssh/id_rsa.pub.
8. Thekeyfingerprintis:
9. 15: 44: c6: 4e: bb: cd: b9: 09: e7: 50: c7: AE: dd: 92: 56: 57lisa @ 10-149-11-152
10. Thekey'srandomartimageis:
11. + -- [RSA2048] ---- +
12. | + = |
13. |. O. |
14. | O... |
15. |. O. oE |
16. | S = +. |
17. | + =. O |
18. | = +. |
19. | =. |
20. |... |
21. + ----------------- +

Configure/etc/hosts

The/etc/hosts of the 10 servers must be the same, so that they can communicate with each other through hostname.

[Plain] view plaincopyprint?

1. 127.0.0.1localhostlocalhost.localdomainlocalhost4localhost4.localdomain4
2. : 1localhostlocalhost. localdomainlocalhost6localhost6. localdomain6
3. 10.149.11.15210-149-11-152
4. 10.149.11.15310-149-11-153
5. 10.149.11.15410-149-11-154
6. 10.149.11.15510-149-11-155
7. 10.149.11.15610-149-11-156
8. 10.149.11.15710-149-11-157
9. 10.149.11.15810-149-11-158
10. 10.149.11.15910-149-11-159
11. 10.149.11.16010-149-11-160
12. 10.149.11.16110-149-11-161

Configure core-site.xml

Switch to the lisa account and edit the core-site.xml File

[Plain] view plaincopyprint?

1. <Configuration>
2. <Property>
3. <Name> fs. default. name </name>
4. <Value> hdfs: // 10-149-11-152: 9000 </value>
5. </Property>
6. <Property>
7. <Name> io. file. buffer. size </name>
8. <Value> 131072 </value>
9. </Property>
10. <Property>
11. <Name> ha. zookeeper. quorum </name>
12. <Value> 10.149.11.146: 2181, 10.149.11.147: 2181, 10.149.11.148: 2181 </value>
13. </Property>
14. </Configuration>

Three zookeeper IP addresses and port numbers are configured here.

Configure hdfs-site.xml

First, create a series of directories to store namenode, journalnode, and datanode data. These directories do not need to be created on every server. I have created them to save time, but I just don't need them.

[Plain] view plaincopyprint?

1. Mkdir-p/data/slot0/lisa/hdfs/namenode
2. Chown-Rlisa: lisa/data/slot0/lisa
3. Mkdir-p/data/slot0/lisa/hdfs/journalnode
4. Mkdir-p/data/slot0/lisa/hdfs/datanode
5. Mkdir-p/data/slot1/lisa/hdfs/datanode
6. Mkdir-p/data/slot2/lisa/hdfs/datanode
7. Mkdir-p/data/slot3/lisa/hdfs/datanode
8. Mkdir-p/data/slot4/lisa/hdfs/datanode
9. Mkdir-p/data/slot5/lisa/hdfs/datanode
10. Mkdir-p/data/slot6/lisa/hdfs/datanode
11. Mkdir-p/data/slot7/lisa/hdfs/datanode
12. Mkdir-p/data/slot8/lisa/hdfs/datanode
13. Mkdir-p/data/slot9/lisa/hdfs/datanode
14. Mkdir-p/data/slot11/lisa/hdfs/datanode
15. Mkdir-p/data/slot10/lisa/hdfs/datanode
16. Chown-Rlisa: lisa/data/slot1/lisa
17. Chown-Rlisa: lisa/data/slot2/lisa
18. Chown-Rlisa: lisa/data/slot3/lisa
19. Chown-Rlisa: lisa/data/slot4/lisa
20. Chown-Rlisa: lisa/data/slot5/lisa
21. Chown-Rlisa: lisa/data/slot6/lisa
22. Chown-Rlisa: lisa/data/slot7/lisa
23. Chown-Rlisa: lisa/data/slot8/lisa
24. Chown-Rlisa: lisa/data/slot9/lisa
25. Chown-Rlisa: lisa/data/slot10/lisa
26. Chown-Rlisa: lisa/data/slot11/lisa