Hadoop namenode for NFS disaster recovery

Hadoop namenode for NFS disaster recovery

1. Select a disaster recovery machine as the NFS server.

(1) the installation is as follows:

Apt-get install nfs-kernel-server (NFS server, providing nfs service)

Apt-get install portmap (Port ing server, mainly used to convert RPC program numbers to Internet port numbers)

(2) set the directory to be mounted as nfs service in the/etc/exports file

Note:/home/Hadoop/name indicates the mounted directory

* Indicates any IP address. You can configure it as needed, for example, 10. 10 .*.*

(Rw, sync, no_subtree_check) This refers to the settings of the Mount directory (For details, refer to the remarks below). To grant the write permission to namenode, you should configure it to rw.

(3) restart the NFS service (enable the exports modification to take effect. Of course, there are other commands that won't be mentioned here)

Sudo/etc/init. d/nfs-kernel-server restart or sudo service nfs-kernel-server restart. In addition, start, stop, and status are not detailed.

The commands related to portmap are the same as those above.

(4) check whether the mounted directory is successful

Showmount-e

2. Install NFS client on namenode

(1) apt-get install nfs-common (NFS client, access nfs)

(2) mount the NFS directory to the local device.

Mount IP:/home/hadoop/name/home/hadoop/nfs

Mount command NFS service IP address and path local directory (must be an existing Directory)

Umount/home/hadoop/nfs unmount directory. If the device is busy, add the-l parameter to force unmount the directory.

3. Configure the dfs. name. dir configuration of the hdfs-site.xml file under namenode with the Mount path/home/hadoop/nfs

For example

Uploading...

4. Restart hadoop

Then, go to the NFS server/home/hadoop/name directory to check whether there are namenode fsimage and other files.

If yes, the configuration is successful.

If the specified directory (/home/hadoop/nfs) cannot be accessed, check whether the file permission on the NFS server is readable (ro)

Note: For more information about NFS configuration, see other articles on the Internet. This article does not focus on NFS configuration, so we will not describe it more.

The following are some configuration parameters and meanings of the mount file, which are for reference only.

Ro read-only access

Rw read/write access

Sync all data written for sharing upon request

Async NFS can request data before writing data

Secure NFS is sent through the secure TCP/IP ports below 1024

Insecure NFS is sent over port 1024

Wdelay if multiple users want to write data to the NFS Directory, group write (default)

No_wdelay if multiple users want to write data to the NFS Directory, write the data immediately. This setting is not required when async is used.

Hide does not share its subdirectories in the NFS shared directory.

No_hide shares the subdirectory of the NFS Directory

Subtree_check if sub-directories such as/usr/bin are shared, force NFS to check the permissions of the parent directory (default)

No_subtree_check is opposite to the above. The parent directory permission is not checked.

All_squash: The UID and GID of the shared file are mapped to the anonymous user anonymous, which is suitable for public directories.

No_all_squash retains the UID and GID of the shared file (default)

Root_squash all requests of the root user are mapped to the same permissions as those of the anonymous user (default)

The no_root_squas root user has full management access permissions to the root directory.

Anonuid = xxx specifies the UID of an anonymous user in the NFS server/etc/passwd file

Anongid = xxx specifies the GID of anonymous users in the NFS server/etc/passwd file

You may also like the following articles about Hadoop:

Tutorial on standalone/pseudo-distributed installation and configuration of Hadoop2.4.1 under Ubuntu14.04

Install and configure Hadoop2.2.0 on CentOS

Build a Hadoop environment on Ubuntu 13.04

Cluster configuration for Ubuntu 12.10 + Hadoop 1.2.1

Build a Hadoop environment on Ubuntu (standalone mode + pseudo Distribution Mode)

Configuration of Hadoop environment in Ubuntu

Detailed tutorial on creating a Hadoop environment for standalone Edition