CentOS 7.4 single-host Hadoop 3.0 installation tutorial

CentOS 7.4 single-host Hadoop 3.0 installation tutorial

Simple record of the Hadoop 7.4 standalone installation tutorial under CentOS 3.0.

Related downloads
Hive
Http://mirrors.hust.edu.cn/apache/hive/stable-2/apache-hive-2.3.2-bin.tar.gz
Hadoop
Https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/hadoop-3.0.0/hadoop-3.0.0.tar.gz

1. hive installation directory/opt/hive/apache-hive-2.3.2-bin
2. hadoop installation directory/opt/hadoop/hadoop-3.0.0
3. JDK installation directory/usr/java/jdk1.8.0 _ 65
4. Environment Variable Configuration
Export JAVA_HOME =/usr/java/jdk1.8.0 _ 65
Export HADOOP_HOME =/opt/hadoop/hadoop-3.0.0.
Export HADOOP_CONF_DIR =$ {HADOOP_HOME}/etc/hadoop
Export HADOOP_COMMON_LIB_NATIVE_DIR =$ {HADOOP_HOME}/lib/native
Export HADOOP_OPTS = "-Djava. library. path =$ {HADOOP_HOME}/lib"
Export HIVE_HOME =/opt/hive/apache-hive-2.3.2-bin
Export HIVE_CONF_DIR =$ {HIVE_HOME}/conf
Export CLASS_PATH =. :$ {JAVA_HOME}/lib :$ {HIVE_HOME}/lib: $ CLASS_PATH
Export PATH =. :$ {JAVA_HOME}/bin :$ {HADOOP_HOME}/sbin :$ {HIVE_HOME}/bin: $ PATH
5. Make environment variables take effect
Source/etc/profile
6. vim/opt/hadoop/hadoop-3.0.0/etc/hadoop/core-site.xml Modification
<Configuration>
<! -- Specify the communication address of the HDFS boss (namenode) -->
<Property>
<Name> fs. defaultFS </name>
<Value> hdfs :/// localhost: 9000 </value>
</Property>
<! -- Specify the storage path for files generated during hadoop running -->
<Property>
<Name> hadoop. tmp. dir </name>
<Value>/opt/hadoop/tmp </value>
</Property>
</Configuration>

7. vim/opt/hadoop/hadoop-3.0.0/etc/hadoop/hdfs-site.xml Add the following content
<Configuration>
<Property>
<Name> dfs. name. dir </name>
<Value>/opt/hadoop/hdfs/name </value>
<Description> store hdfs namespace metadata on namenode </description>
</Property>
<Property>
<Name> dfs. data. dir </name>
<Value>/opt/hadoop/hdfs/data </value>
<Description> physical storage location of data blocks on datanode </description>
</Property>
<! -- Set the number of hdfs replicas -->
<Property>
<Name> dfs. replication </name>
<Value> 1 </value>
</Property>
</Configuration>
8. Set password-free logon for SSH
Ssh-keygen-t dsa-p'-f ~ /. Ssh/id_dsa
Cat ~ /. Ssh/id_dsa.pub> ~ /. Ssh/authorized_keys
Chmod 0600 ~ /. Ssh/authorized_keys
9. Start the command
9.1 Initialization
Cd/opt/hadoop/hadoop-3.0.0
./Bin/hdfs namenode-format
9.2 start command
../Sbin/start-dfs.sh
9.3 Stop command
../Sbin/stop-dfs.sh

Error Handling
Starting namenodes on [localhost]
ERROR: Attempting to operate on hdfs namenode as root
ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation.
Starting datanodes
ERROR: Attempting to operate on hdfs datanode as root
ERROR: but there is no HDFS_DATANODE_USER defined. Aborting operation.
Starting secondary namenodes [bogon]
ERROR: Attempting to operate on hdfs secondarynamenode as root
ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting operation.
Processing 1
$ Vim sbin/start-dfs.sh
$ Vim sbin/stop-dfs.sh
Add the following content
HDFS_DATANODE_USER = root
HADOOP_SECURE_DN_USER = hdfs
HDFS_NAMENODE_USER = root
HDFS_SECONDARYNAMENODE_USER = root
Process 2
$ Vim sbin/start-yarn.sh
$ Vim sbin/stop-yarn.sh
Add the following content
YARN_RESOURCEMANAGER_USER = root
HADOOP_SECURE_DN_USER = yarn
YARN_NODEMANAGER_USER = root

10. Verify Installation
Http: // 192.168.50.48: 9870/dfshealth.html # tab-overview

Hadoop2.3-HA high-availability cluster environment construction

Hadoop project-Cloudera 5.10.1 (CDH) installation and deployment based on CentOS7

Detailed explanation of Hadoop2.7.2 cluster construction (high availability)

Use Ambari to deploy a Hadoop cluster (build an intranet HDP source)

Hadoop cluster installation in Ubuntu 14.04

Install Hadoop 2.7.2 In CentOS 6.7

Build a distributed Hadoop-2.7.3 cluster on Ubuntu 16.04

Build a Hadoop 2.6.4 distributed Cluster Environment in CentOS 7

Hadoop2.7.3 + Spark2.1.0 fully distributed cluster Construction Process