Configure hadoop 2.6.0 pseudo-distributed in CentOS 6.5

Configure hadoop 2.6.0 pseudo-distributed in CentOS 6.5

Operating System: CentOS 6.5 32-bit

Java environment: jdk 1.7.0 _ 71

Hadoop: http://apache.fayea.com/hadoop/common/stable/hadoop-2.6.0.tar.gz

1. create user groups and users

Use the su command to switch the user to root.

[Html] view plaincopyprint?

1. Groupaddhadoop
2. Useraddhadoophadoop
3. Passwdhadoop # You can add a password to a user without setting a password.

2. Install ssh

[Plain] view plaincopyprint?

1. Rpm-qa | grepssh # Check whether an SSH package is installed
2. Yuminstallopenssh-server # Install ssh
3. Chkconfig -- listsshd # Check whether SSHD is set to start upon startup
4. Chkconfig -- level2345sshdon # Set it if startup is not set.
5. Servicesshdrestart # restart

3. Configure ssh login without a password

Switch to hadoop user

[Html] view plaincopyprint?

1. Suhadoop

Generate key

[Html] view plaincopyprint?

1. Ssh-keygen-trsa-P ""

The id_rsa and id_rsa.pub files are generated under the. ssh directory.

Enter the. ssh directory and append id_rsa.pub to the authorized_keys file.

[Html] view plaincopyprint?

1. Cd./. ssh
2. Catid_rsa.pub> authorized_keys
3. Chmod600authorized_keys # modify user permissions

Test whether logon is allowed

[Html] view plaincopyprint?

1. Sshlocalhost # After the command is executed, a prompt is displayed, indicating that yesorno. If yes is entered, the last logon time indicates that the command is successful.

4. Install hadoop

Decompress the downloaded hadoop file and move it to the desired installation directory to modify its access permissions.

[Html] view plaincopyprint?

1. Tar-xvfhadoop-2.6.0.tar.gz
2. Mvhadoop-2.6.0/usr/opt/hadoop
3. Chmod-R775/usr/opt/hadoop
4. Chown-Rhadoop: hadoop/usr/opt/hadoop

Configure hadoop environment variables (all environment variables are required)

[Html] view plaincopyprint?

1. Su
2. Vim/etc/profile
3. ExportHADOOP_INSTALL =/usr/opt/hadoop
4. ExportPATH =$ {HADOOP_INSTALL}/bin: $ {HADOOP_INSTALL}/sbin $ {PATH}
5. ExportHADOOP_MAPRED_HOME =$ {HADOOP_INSTALL}
6. ExportHADOOP_COMMON_HOME =$ {HADOOP_INSTALL}
7. ExportHADOOP_HDFS_HOME =$ {HADOOP_INSTALL}
8. ExportYARN_HOME =$ {HADOOP_INSTALLL}
9. ExportHADOOP_COMMON_LIB_NATIVE_DIR =$ {HADOOP_INSTALL}/lib/natvie
10. ExportHADOOP_OPTS = "-Djava. library. path =$ {HADOOP_INSTALL}/lib :$ {HADOOP_INSTALL}/lib/native"

Set java environment variables in hadoop-env.sh

[Html] view plaincopyprint?

1. Cd/usr/opt/hadoop
2. Vim ../etc/hadoop/hadoop-env.sh

[Html] view plaincopyprint?

1. ExportJAVA_HOME = {your java environment variable}

5. Configure pseudo-distributed

Hadoop configuration files mainly include core-site.xml, hdfs-site.xml, yarn-site.xml three files.

[Html] view plaincopyprint?

1. Cd/usr/opt/hadoop/etc/hadoop

Core-site.xml

[Html] view plaincopyprint?

1. <Configuration>
2. <Property>
3. <Name> hadoop. tmp. dir </name>
4. <Value>/usr/opt/hadoop/tmp </value> <! -- Make sure to configure the default cache folder of the system. It will be deleted after each restart. -->
5. </Property>
6. <Property>
7. <Name> fs. defaultFS </name>
8. <Value> hdfs :/// localhost: 9000 </value>
9. </Property>
10. </Configuration>

Hdfs. xml

[Html] view plaincopyprint?

1. <Configuration>
2. <Property>
3. <Name> dfs. replication </name>
4. <Value> 1 </value>
5. </Property>
6. <Property>
7. <Name> dfs. namenode. name. dir </name>
8. <Value> file:/usr/opt/hadoop/dfs/name </value>
9. </Property>
10. <Property>
11. <Name> dfs. datanode. data. dir </name>
12. <Value> file:/usr/opt/hadoop/dfs/data </value>
13. </Property>
14. <Property> <! -- This attribute node is used to prevent the following eclipse from rejecting read/write settings -->
15. <Name> dfs. permissions </name>
16. <Value> false </value>
17. </Property>
18. </Configuration>

Yarn-site.xml

[Html] view plaincopyprint?

1. <Configuration>
2. <Property>
3. <Name> mapreduce. framework. name </name>
4. <Value> yarn </value>
5. </Property>
7. <Property>
8. <Name> yarn. nodemanager. aux-services </name>
9. <Value> mapreduce_shuffle </value>
10. </Property>
11. </Configuration>

Next, create the required folder.

[Html] view plaincopyprint?

1. Cd/usr/local/hadoop
2. Mkdirtmpdfsdfs/namedfs/data

All configurations have been completed so far.

6. Run

First, format the file system.

[Html] view plaincopyprint?

1. . Bin/hdfsnamenode-format

Start

[Html] view plaincopyprint?

1. ../Sbin/start-dfs.sh
2. ../Sbin/start-yarn.sh

If the following prompt is displayed, the operation is successful.

[Html] view plaincopyprint?

1. Startingnamenodeson [localhost]
2. Localhost: startingnamenode, loggingto/usr/opt/hadoop-2.6.0/logs/hadoop-namenode-. out
3. Localhost: startingdatanode, loggingto/usr/opt/hadoop-2.6.0/logs/hadoop-datanode-. out
4. Startingsecondarynamenodes [0.0.0.0]
5. 0.0.0.0: startingsecondarynamenode, loggingto/usr/opt/hadoop-2.6.0/logs/hadoop-secondarynamenode-. out

Enter 127.0.0.1: 50070 to view the hadoop webpage.

PS: If a warning is displayed:

[Html] view plaincopyprint?

1. Unabletoloadnative-hadooplibraryforyourplatform

You can also see the webpage, but hadoop is not fully installed.

There are two problems:

1. The environment variable HADOOP_OPTS is not set.

2. This is caused by incompatibility between the operating system and hadoop. so files.

Hadoop 2.5 and later versions of release are 64-bit compiled. If your computer is 32-bit, incompatibility may occur. At this time, you need to re-compile the hadoop source code (when compiling the hadoop source code, do not use jdk1.8.0, use 1.6 and 1.7. Otherwise, this ERROR [ERROR] Failed to execute goal org. apache. maven. plugins: maven-javadoc-plugin: 2.8.1: jar (module-javadocs) on project hadoop-annotations: MavenReportException: Error while creating archive ):