Hadoop learning notes: hadoop pseudo-Distributed Environment Construction

Build a pseudo-distributed hadoop Environment

1. network connection between the host machine (Windows) and the client (Linux installed in a virtual machine.

A) The host-only host is connected to the client separately;

Benefits: Network isolation;

Disadvantage: the virtual machine cannot communicate with other servers;

B. The bridge host is in the same LAN as the client.

Benefit: the sinus can access each other in the same LAN;

Disadvantage: incomplete.

2. Distributed installation steps for hadoop

A) set static IP addresses

Right-click the icon in the upper-right corner of the lower-left corner of centos to modify it;

Restart the NIC service network restart;

Verify: Execute the command ifconfig

B) modify the Host Name (Original Name: casichadoop) hostname to view the current name

Modify the current Session host name: Set the name hostname hadoop

In the configuration file, set hostname VI/etc/sysconfig/network to hadoop.

C) bind the hostname and IP address

Run the command VI/etc/hosts to add 10.10.10.10.

Save and exit

Verification: Ping hadoop

 

D. Disable the firewall and view the service iptables status first.

Service iptables stop

E) Check chkconfig -- list | grep iptables when the firewall is automatically started.

Disable Automatic Start firewall chkconfig iptables off

Verification: chkconfig -- list | grep iptables

 

F) SSH (Secure Shell) password-free Login

<1> execute the command Shh-keygen-t rsa to generate the key, which is located in ~ /. SSH

<2> Execute Command CP ~ /. Ssh/id_rsa.pub ~ /. Ssh/authorized_keys

Verification: SSH localhost

G) install JDK

 

L open the directory CD/usr/local

L then delete all files Rm-RF *

 

L copy all files to the current directory CP/root/downloads /*.

 

L view the permissions of the current directory LS-l

L grant permission to execute chmod U + x jdk-6u24-linux-i586.bin

 

L execute JDK installation./jdk-6u24-linux-i586.bin

L mv jdk1.6.0 _ 24 JDK rename

L set the environment variable VI/etc/profile

N export java_home =/usr/local/JDK

N export Path =./$ java_home/bin: $ path

N source/etc/profile

N verification: Java-version

 

H) install hadoop

Hadoop-1.1.2.tar.zip I. Tar-zxvf

Ii. Mv hadoop-1.1.2 hadoop

Iii. Set Environment Variables

N export hadoop_home =/usr/local/hadoop

N export Path =.: $ hadoop_home/bin: $ java_home/bin: $ path

Iv. Modify four configuration files

1. Modify hadoop-env.sh

Export java_home =/usr/local/JDK

2. Modify core-site.xml

<?xml version="1.0"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Put site-specific property overrides in this file. --> <configuration><property><name>fs.default.name</name><value>hdfs://hadoop:9000</value><description>hadoop</description></property><property><name>hadoop.tmp.dir</name><value>/usr/local/hadoop/tmp</value></property></configuration>

3. Modify hdfs-site.xml

<?xml version="1.0"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Put site-specific property overrides in this file. --> <configuration><property><name>dfs.replication</name><value>1</value></property><property> <name>dfs.permissions</name><value>false</value></property></configuration> 

4. Modify mapred-site.xml

<?xml version="1.0"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Put site-specific property overrides in this file. --> <configuration><property><name>mapred.job.tracker</name><value>hadoop:9001</value><description>change you own hostname</description></property></configuration>

 

V. Run the hadoop namenoe-format command to format HDFS

Vi. Execute Command start-all.sh

VII. Execute the command JPs to view the process

 

Hadoop learning notes: hadoop pseudo-Distributed Environment Construction