Large Data Hadoop Platform (ii) Centos6.5 (64bit) Hadoop2.5.1 pseudo distributed installation record, WordCount run test __ Large data

Note: The following installation steps are performed in the Centos6.5 operating system, and the installation steps are also suitable for other operating systems, such as having classmates using other Linux operating systems such as Ubuntu, just note that individual commands are slightly different.

Note the operation of different user rights, such as the shutdown firewall, the need to use root permissions.

The problem with one-node Hadoop installation is as follows: Configuration of JDK environment, firewall shutdown, different operation of root user and Hadoop user, etc.

In the process of building a little more careful, follow the steps below to do, basically no problem.

I. Preparation (root user)

1. Close the firewall

Shutdown Firewall: Service iptables stop
shutdown boot: chkconfig iptables off

2. Create User

Create a Hadoop User: Useradd hadoop
Password: passwd hadoop
joins Sudoers:vim/etc/sudoers, writes Hadoop all= in the root line  (all)       All

3. Modify the Hosts file

On the last line of the/etc/hosts file, add:

127.0.0.1		Hadoop

Second, install JDK1.8 (root user)
1. View installed JDK

Rpm-qa |grep java
rpm-qa |grep jdk

2. Uninstall the program shown in the previous step

RPM-E--nodeps  Program name 					
(for example: Rpm-e--nodeps tzdata-java-2013g-1.el6.noarch rpm-e--nodeps
java-1.7.0-openjdk-1.7.0.45-2.4.3.3.el6.x86_64
rpm-e--nodeps java-1.6.0-openjdk-1.6.0.0-1.66.1.13.0.el6.x86 _64)

3. Install JDK1.8

RPM-IVH jdk-8-linux-x64.rpm  (executes the instruction in the directory where the installation files are located, before installing the. rpm file in any directory, and the left post JDK is installed in/usr/java/jdk1.8.0)

4. Modifying environment Variables

Modify the/etc/profile file and add the following lines to the end of the file:

Export java_home=/usr/java/jdk1.8.0
export jre_home=/usr/java/jdk1.8.0/jre
export path= $JAVA _home/bin:$ Jre_home/bin: $PATH
export classpath=.: $JAVA _home/lib/dt.jar: $JAVA _home/lib/tools.jar

5. Make the environment variables that have just been modified take effect

Source/etc/profile

6. Verify that the JDK is installed successfully

Java-version
Echo $JAVA _home

ssh Password-free login (Hadoop user) 1. Generate Key

Ssh-keygen-t DSA (and then always press ENTER) automatically generates an. ssh folder with two files in it

2. Generate Authorized_keys

Enter/home/hadoop/.ssh Directory

Cat Id_dsa.pub >> Authorized_keys

3. Granting executive authority to Authorized_keys

chmod Authorized_keys

4. Test if you can log on locally without a password

SSH localhost

If you do not need to enter the password again, the success of the installation of Hadoop (Hadoop user)
1. Extract to the specified directory (for example, in the/home/hadoop directory)

TAR-ZXVF hadoop-2.5.1.tar.gz

2. configuration file

Configuration file 2.1.core-site.xml file in/home/hadoop/hadoop-2.5.1/etc/hadoop/directory

Add the following content between <configuration> and </configuration>

<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value >
</property>
 
<property>
<name>hadoop.tmp.dir</name>
<value> /home/hadoop/hadoop-2.5.1/tmp</value>
</property>

2.2.hdfs-site.xml File

<property>
<name>dfs.namenode.name.dir</name>  
<value>/home/hadoop/hadoop-2.5.1 /name</value>        
</property>        
 
<property>
<name>dfs.datanode.data.dir</name >  
<value>/home/hadoop/hadoop-2.5.1/data</value>                                                                  
</property>          
 
<property>
<name>dfs.replication</name>  
<value>1</value>
</property>

Note: Both the/home/hadoop/hadoop-2.5.1/data and/home/hadoop/hadoop-2.5.1/name directories should be present.

2.3.mapred-site.xml File

<property>
<name>mapreduce.framework.name</name>  
<value>yarn</value>    

2.4.mapred-env.sh File

Export java_home=/usr/java/jdk1.8.0
Export hadoop_mapred_pid_dir=/home/hadoop/hadoop-2.5.1/tmp

2.5.hadoop-env.sh File

Export java_home=/usr/java/jdk1.8.0
export hadoop_pid_dir=/home/hadoop/hadoop-2.5.1/tmp
export HADOOP_ Secure_dn_pid_dir=/home/hadoop/hadoop-2.5.1/tmp

2.6.yarn-site.xml File

<property>
<name>yarn.nodemanager.aux-services</name>  
<value>mapreduce_shuffle </value>    
</property>

2. Adding Hadoop to environment variables

sudo vim/etc/profile
joins the following two lines of
export hadoop_home=/home/hadoop/hadoop-2.5.1
export path= $HADOOP _home/ Bin: $HADOOP _home/sbin: $PATH

v. Start (Hadoop user)

1. Format Namenode

HDFs Namenode-format

If successful, the current folder is generated in/home/hadoop/hadoop-2.5.1/name/

2. Start Namenode and Datanode

hadoop-daemon.sh start Namenode
hadoop-daemon.sh start Datanode

Ability to verify successful startup through JPS

3. Start Yarn

start-yarn.sh

Input JPS Validation

4. View at the end of the page

Enter ip:50070 (for example: http://192.168.56.103:50070/)

Vi. Running wordcount example (Hadoop user)

WordCount example in the/home/hadoop/hadoop-2.5.1/share/hadoop/mapreduce of the Hadoop-mapreduce-examples-2.5.1.jar 1. Upload local files to HDFs

Hadoop fs-put file/test (such as: Hadoop fs-put 1/test is to upload local file 1 to HDFs in/test directory)

2. Run

Hadoop jar Hadoop-mapreduce-examples-2.5.1.jar WORDCOUNT/TEST/1/TEST/OUTPUT/1

Note:/TEST/OUTPUT/1 must be a non-existent directory

3. View Results

Hadoop fs-cat/test/output/1/part-r-00000