Hadoop pseudo-Distributed build steps

Description: Build the environment is VMWARE10 with Linux CENTOS 32-bit, hadoop:hadoop-2.4.1 java:jdk7 32-bit This article is I collected on the network of the Hadoop series of videos attached to the class notes (here directly copied, hope to forgive, I am not a bad person ), record memo.

before we start , let's simply modify the JDK of the Linux system; In General, we will have to uninstall the Linux openjdk, and then install the Sun's JDK (the following code source network, please pay attention to the real environment)

<1> View the Java version information that is already installed

The following information is generally available:

" 1.6.0 "  1.6. 0-1.6. 0-b09, mixed mode)

<2> View information about the JDK

The following information is generally available:

java-1.4. 2-gcj-compat-1.4. 2.0-40jpp.  the Java-1.6. 0-openjdk-1.6. 0.0-1.7. B09.el5

<3> Uninstall

# yum-y Remove Java java-1.4. 2-gcj-compat-1.4. 2.0-40jpp.  the  -y remove java-1.6. 0-openjdk-1.6. 0.0-1.7. B09.el5

then , is the real installation of Hadoop pseudo-distribution process, I test success.

1. Preparing the Linux environment (IP for reference only)

1 Click the VMware shortcut, right-click the file location, double-clicking Vmnetcfg.exe, VMnet1 host-only, modify subnet IP settings network segment: 192.168.1.0 subnet mask: 255. 255.255.0, apply, OK

Back to Windows--open network and Sharing Center, change adapter settings, right-VMnet1 properties, double-click IPv4, set Windows ip:192.168.1.100 Subnet mask: 255.25 5.255.0-click OK
On the virtual software--my computer--------right-----Settings network adapter,

1.1 Modifying host Names

vim/etc/sysconfig/networknetworking=yeshostname=itcast # # # Here is not modified, also let the reader know the true author source

1.2 Modifying IP

Two different ways:

First: Modify with the Linux GUI (highly recommended)
Go to the Linux GUI-right click on the two small computers in the upper right-click Edit Connections, select the current network system eth0, click the Edit button, and select the option IPv4 Add ip:192.168.1.101 subnet Mask by clicking the Add button, Manual: 255.255.255.0 gateway: 192.168.1.1, apply

The second kind: Modify the configuration file way (The Cock Silk program ape dedicated)

Vim/etc/sysconfig/network-scripts/ifcfg-eth0

Device= "Eth0"
Bootproto= "Static" # # #
Hwaddr= "00:0c:29:3c:bf:e7"
ipv6init= "Yes"
Nm_controlled= "Yes"
onboot= "Yes"
Type= "Ethernet"
Uuid= "ce22eeca-ecde-4536-8cc2-ef0dc36d4a8c"
Ipaddr= "192.168.1.101" # # #
Netmask= "255.255.255.0" # # #
Gateway= "192.168.1.1" # # #

1.3 Modifying the mappings between host names and IPs

Vim/etc/hosts

192.168.1.101 Itcast

1.4 Shutting down the firewall

--list# off firewall boot chkconfig iptables off

1.5 restarting Linux reboot

2. Installing the JDK

2.1 The Sftp window appears after uploading Alt+p (using the SECURECRT), and then

2.2 Unpacking the JDK

/home/hadoop/-zxvf jdk-7u55-linux-i586.tar.gz-c/home/hadoop/app

2.3 Adding Java to an environment variable

vim/etc/profile# Add export java_home=/home/hadoop/app/jdk-7u_65-i585export PATH= $PATH at the end of the file: $JAVA _home//etc/profile

3. Installing hadoop2.4.1

Upload the Hadoop installation package to the server first/home/hadoop/ Note: The hadoop2.x configuration file $hadoop_home/etc/hadoop pseudo-distributed requires 5 configuration files to be modified

3.1 Configuring Hadoop

first one:hadoop-env.sh

Vim hadoop-env.sh# 27th Line export Java_home=/usr/java/jdk1. 7. 0_65

The second one:core-site.xml

<!--Specify the file system schema (URI) used by Hadoop, the address of the boss of HDFs (NameNode)--><property><name>fs.defaultfs</name ><value>hdfs://weekend-1206-01:9000</value></property><!-- Specifies the storage directory where the Hadoop runtime produces files--><property><name>hadoop.tmp.dir</name><value>/home/hadoop/ hadoop-2.4. 1/tmp</value></property>

The third one:hdfs-site.xml hdfs-default.xml

<!--Specify the number of HDFs replicas--><property>    <name>dfs.replication</name>    <value>1 </value></property>

Fourth:mapred-site.xml (mv mapred-site.xml.template mapred-site.xml)

Vim mapred-Site.xml<!--Designate Mr to run on yarn--><property>    <name>mapreduce.framework.name </name>    <value>yarn</value></property>

Fifth one:yarn-site.xml

<!--Specify the address of yarn's eldest (ResourceManager)--><property>    <name>yarn.resourcemanager.hostname</ name>    <value>weekend-1206-</value></property><!-- Reducer How to get Data--><property>    <name>yarn.nodemanager.aux-services</name>    <value >mapreduce_shuffle</value></property>

3.2 Adding Hadoop to an environment variable

vim/etc/proflieexport java_home=/usr/java/jdk1. 7 . 0_65export hadoop_home=/itcast/hadoop-2.4. 1 export PATH= $PATH: $JAVA _home/bin: $HADOOP _home/bin: $HADOOP _home//etc/profile

3.3 Formatting Namenode (initialization of Namenode)

HDFs Namenode-format (Hadoop namenode-format)

3.4 Starting Hadoop

#先启动HDFSsbin/start-dfs.sh        #再启动YARNsbin/start-yarn.sh

3.5 Verifying whether the startup was successful

using the JPS command to verify # JPS 27408 NameNode 28218 Jps 27643 Secondarynamenode 28066 NodeManager 27803 ResourceManager 27512 DataNode    #http://192.168.1.101:50070 (HDFs management Interface)#http://  192.168.1.101:8088 (Mr Management interface)

4. Configure SSH Free Login

~/. Sshssh-keygen-t RSA (four carriage return) after executing this command, a two file Id_rsa (private key), id_rsa.pub (public key) is generated to copy the public key to the machine to be free of login ssh- Copy-id localhost

finally , Hadoop can start up normally.

Here are some of the commands I need, please ignore

Service Network RESTARTCD/home/hadoop/app/hadoop-2.4.1/sbin//etc/udev/rules.d/ --persistent-Net.rules/etc/sysconfig/network-scripts/ifcfg-eth01. Remove Network Manager fromstartup Services.chkconfig NetworkManager off2. ADD Default Net managerchkconfig network on3. Stop NetworkManager firstservice NetworkManager stop4. And then start Default managerservice network STARTSCP-r/home/hadoop/jdk-7u65-linux-i586.tar.gz [email protected]192.168.2.202:/home/seasky/jdk-7u65-linux-I586.TAR.GZSCP-r/home/hadoop/hadoop-2.4.1. tar.gz [Email protected]192.168.2.202:/home/seasky/hadoop-2.4.1. tar.gz

Accessories: hadoop-shell.7z

Hadoop pseudo-Distributed build steps