Build a pseudo-distributed environment for Hadoop-2.7.1

Build a pseudo-distributed environment for Hadoop-2.7.1

1. Prepare the Linux environment
1.0 click the VMware shortcut, right-click the file location, and choose vmnet.exe> VMnet1 host-only> modify the subnet ip address to set the CIDR Block: 192.168.8.0 subnet mask: 255.255.255.0-> apply-> OK.
Go back to windows --> open network and sharing center-> change adapter settings-> right-click VMnet1-> properties-> double-click IPv4-> set windows IP Address: 192.168.8.100 subnet mask: 255.255.255.0-> click OK
On the virtual software, choose My Computer> select Virtual Machine> right-click and choose settings> network adapter> host only> OK.
1.1 Modify host name
Vim/etc/sysconfig/network

NETWORKING = yes
HOSTNAME = FTH01
 
1.2 modify IP Address
Two methods:
First: Modify through the Linux graphic interface (highly recommended)
Go to the Linux graphical interface, right-click the two computers in the upper-right corner, and choose Edit connections from the shortcut menu. Select the current network System eth0, click edit, and select IPv4 from the shortcut menu. Choose manual from the shortcut menu-> click add to add an IP address: 192.168.8.118 subnet mask: 255.255.255.0 Gateway: 192.168.1.1-> apply
 
Method 2: modify the configuration file (dedicated to diaosi programmers)
Vim/etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE = "eth0"
BOOTPROTO = "static"
HWADDR = "00: 0C: 29: 3C: BF: E7"
IPV6INIT = "yeED =" yes"
ONBOOT = "yes"
TYPE = "Ethernet"
UUID = "ce22eeca-ecde-4536-8cc2-ef0dc36d4a8c"
IPADDR = "192.168.8.11"
NETMASK = "255.255.255.0"
GATEWAY = "192.168.8.1"
　　

Run the service network restart command to make the settings take effect.

1.3 modify the ing between host names and IP addresses
Vim/etc/hosts

192.168.8.11 FTH01
 
1.4 disable Firewall
# View the Firewall Status
Service iptables status
# Disable the Firewall
Service iptables stop
# View the firewall startup status
Chkconfig iptables -- list
# Disable firewall startup
Chkconfig iptables off
 
1.5 restart Linux
Reboot

Note !!!!!!
The number of CentOS digits must be the same as that of Hadoop jdk.

 

2. Install JDK
Upload 2.1
Upload files through filezilla software or use secureCRT
2.2 decompress jdk
# Creating folders
Mkdir/usr/java
# Decompress
Tar-zxvf jdk-7u55-linux-i586.tar.gz-C/usr/java/

2.3 add java to Environment Variables
Vim/etc/profile
# Add at the end of the file
Export JAVA_HOME =/usr/java/jdk1.7.0 _ 55
Export PATH = $ PATH: $ JAVA_HOME/bin
 
# Refresh Configuration
Source/etc/profile

3. Install hadoop2.7.1

Decompress the hadoop Installation File to/fth /.

Tar-zxvf hadoop file-C/fth/
Note: hadoop2.x configuration file $ HADOOP_HOME/etc/hadoop
Pseudo-distributed needs to modify 5 profiles/fth/hadoop-2.7.1/conf/
3.1 configure hadoop
First: hadoop-env.sh
Vim hadoop-env.sh.
# 27th rows
Export JAVA_HOME =/usr/java/jdk1.7.0 _ 65

Second: core-site.xml
<! -- Specify the address of the HDFS boss (NameNode) -->
<Property>
<Name> fs. defaultFS </name>
<Value> hdfs: // FTH01: 9000 </value>
</Property>
<! -- Specify the storage directory for files generated during hadoop running -->
<Property>
<Name> hadoop. tmp. dir </name>
<Value>/itcast/hadoop-2.7.1/tmp </value>
</Property>

Third: hdfs-site.xml
<! -- Specify the number of HDFS copies -->
<Property>
<Name> dfs. replication </name>
<Value> 1 </value>
</Property>

Fourth: mapred-site.xml (mv mapred-site.xml.template mapred-site.xml)
Mv mapred-site.xml.template mapred-site.xml
Vim mapred-site.xml.
<! -- Specify that mr runs on yarn -->
<Property>
<Name> mapreduce. framework. name </name>
<Value> yarn </value>
</Property>

Fifth: yarn-site.xml
<! -- Specify the ResourceManager address -->
<Property>
<Name> yarn. resourcemanager. hostname </name>
<Value> itcast01 </value>
</Property>
<! -- CER: how to obtain data -->
<Property>
<Name> yarn. nodemanager. aux-services </name>
<Value> mapreduce_shuffle </value>
</Property>
 
3.2 add hadoop to Environment Variables
 
Vim/etc/proflie
Export JAVA_HOME =/usr/java/jdk1.7.0 _ 65
Export HADOOP_HOME =/itcast/hadoop-2.7.1
Export PATH = $ PATH: $ JAVA_HOME/bin: $ HADOOP_HOME/sbin

Source/etc/profile

3.3 format namenode (initialize namenode)
Hdfs namenode-format (hadoop namenode-format)

3.4 start hadoop
Start HDFS first
Sbin/start-dfs.sh

Start YARN again
Sbin/start-yarn.sh

3.5 verify that startup is successful
Use jps command to verify
27408 NameNode
Jps 28218
27643 SecondaryNameNode
28066 NodeManager
27803 ResourceManager
27512 DataNode
 
Http: // 192.168.8.118: 50070 (HDFS Management Interface)
Http: // 192.168.8.118: 8088 (MR Management Interface)

4. Configure ssh Login-free
# Generate an ssh Login-free key
# Go to my home directory
Cd ~ /. Ssh

Ssh-keygen-t rsa (four carriage return)
After executing this command, two files, id_rsa (Private Key) and id_rsa.pub (Public Key), are generated)
Copy the public key to the machine on which you want to log on without logon. cp id_rsa.pub authorized_keys
Ssh-copy-id localhost
For example, if ssh-copy-id 192.168.8.99 copies the public key of 192.168.8.88 to 192.168.8.99, you can directly connect to 192.168.8.99 without entering the password.

5. error message

View the Linux system's uname-

1. q: Java HotSpot (TM) ClientVM warning: You have loaded library/usr/local/hadoop2.5/lib/native/libhadoop. so.1.0.0which might have disabled stack guard. the VM will try to fix the stack guardnow.

A: The cause of this error occurs on the 64-bit operating system because the local library files downloaded from hadoop are all compiled based on 32-bit, the preceding error occurs when running on a 64-bit system.

Solution 1: recompile hadoop on a 64-bit system;

Solution 2: Add the following two lines in the hadoop-env.sh, yarn-env.sh:

Export HADOOP_COMMON_LIB_NATIVE_DIR =$ {HADOOP_HOME}/lib/native

Export HADOOP_OPTS = "-Djava. library. path = $ HADOOP_HOME/lib"

You may also like the following articles about Hadoop:

Tutorial on standalone/pseudo-distributed installation and configuration of Hadoop2.4.1 under Ubuntu14.04

Install and configure Hadoop2.2.0 on CentOS

Build a Hadoop environment on Ubuntu 13.04

Cluster configuration for Ubuntu 12.10 + Hadoop 1.2.1

Build a Hadoop environment on Ubuntu (standalone mode + pseudo Distribution Mode)

Configuration of Hadoop environment in Ubuntu

Detailed tutorial on creating a Hadoop environment for standalone Edition