"Go" Hadoop cluster add disk step

Transferred from: http://blog.csdn.net/huyuxiang999/article/details/17691405

First, the experimental environment:

1, Hardware: 3 Dell Servers, CPU:2.27GHZ*16, Memory: 16GB, one for master, and the other 2 for slave.

2, the system: all CentOS6.3

3, Hadoop version: CDH4.5, the use of the MapReduce version is not yarn, but Mapreduce1, the entire cluster under Cloudera Manager monitoring, configuration is also through the manager configuration (by changing the configuration file is the same).

Second, the background:

With the increase in data volume, the lab's HDFS storage is not enough, so the new 2*2TB hard drive, installed on 2 datanode server. Hadoop does not require RAID (HDFs has other machines that guarantee data redundancy), so plug the hard drive directly into the server and find a new hard drive in the BIOS.

Three, hard disk partition, format:

Each drive is divided into 5 zones, three of which are primary, and the other two logical partitions.

1. Check the drive Mount condition

Enter the command fdisk-l to see how many hard drives are in the system. Generally the first hard disk is/dev/sda, the second block is/dev/sdb, and so on. If you do not find the installed hard disk, the system is not recognized, you need to restart the server to enter the BIOS settings.

2. Hard disk partition

Input command Fdisk/dev/sdb (because this is the second hard drive, so the input is/dev/sdb, this needs to be based on the situation)

Here you can enter m to see what the specific parameters are.

Enter p to display the partition of this hard disk, because it is a new hard disk, so there is no partition information.

The first primary partition is created below.

Enter N to create the partition, and then enter p in the next prompt to create the primary partition, and then enter 1, which means to create the first primary partition (you can create 4 primary partitions altogether), then enter the starting cylinder, select the default, and then enter the end cylinder, where you can also enter the size, such as +400g, Indicates that the size of this partition is 400G, so that the first primary partition is created.

Then follow the steps above to create a 2nd, 3rd primary partition, all 400G.

Next, create 2 logical partitions.

Enter N, and then enter E in the next prompt to create an extended partition, enter the starting cylinder, generally default, and then enter the end bar, which is also the default, indicating that the extended partition uses all the remaining space. In this way, an extended partition is ready, and then we will create two logical partitions on this extended partition, creating a method that is similar to creating a primary partition, entering n first, and then believing that the reader will do the next step.

After all the partitions have been created, remember to enter W to save the partition, otherwise it is not done.

3. Format the partition

The input command mkfs-t EXT4/DEV/SDB1, which means that the partition/DEV/SDB1 formatted as a EXT4 file system, and the same method is used to format the other partitions.

4. Mount Partition to Directory

You can choose to mount a partition to an existing directory or mount to a new directory. Note If you mount to an existing directory, you must first back up the files in that directory. This article is for the new directory: Mkdir-p/hd/sdb1, and then mount/dev/sdb1/hd/sdb1, same mount other partitions.

5, modify the/etc/fstab

If not modified, every time you turn on the manual to do the 4th step, more trouble. Open the Fstab file, add 5 new partitions according to an existing entry, and the last two data for each entry are 0 0

Iv. Expansion of HDFs

I add all of the above 5 partitions to HDFs. First create a new subdirectory in the Mount directory for each partition/dfs/dn, such as Mkdir-p/hd/sdb1/dfs/dn, and then modify the permissions of the DN directory:

Chown-r Hdfs:hadoop/hd/sdb1/dfs/dn

This allows HDFs to use this directory (each partition will have to create a new/DFS/DN).

Open Cloudera Manager and find "service"-"HDFs"

Select "Configure"-"View and edit", then on the left to find Datanode (default), the first property on the right, the original only one entry (my is/dfs/dn), in order to make HDFS use the new hard disk, so to add the partition we created, the following 5 is my directory.

Then click Open Datanode (Default)

Find "Advanced", in the Datanode volume selection policy on the right, select "Free space" so that the next two properties ("10G bytes" and "0.75") will work to balance the space usage in each volume.

You can expand HDFs by saving the above configuration and then restarting HDFs.

"Go" Hadoop cluster add disk step