Hadoop construction and wordcount instance running in CentOS

Hadoop construction and wordcount instance running in CentOS

I have encountered a lot of troubles in learning to get hadoop recently. It is recorded here to facilitate future viewing. CentOS7 installed in VMwareWorkstation10.0.4

Install and configure jdk

Java-version to check whether java is installed

Use env | grepJAVA_HOME or echo $ JAVA_HOME $ PATH to check whether the environment variable configuration is correct. If not, download it from the official website. Because the built-in jdk has some problems, I first uninstalled the built-in jdk:

First, check that jdk is installed in linux.

# Rpm-qa | grepjdk

Java-1.7.0-openjdk-headless-1.7.0.51-2.4.5.5.el7.x86_64

Java-1.7.0-openjdk-1.7.0.51-2.4.5.5.el7.x86_64

Then uninstall the built-in jdk

# Rpm-e -- nodepsjava-1.7.0-openjdk-headless-1.7.0.51-2.4.5.5.el7.x86_64

# Rpm-e -- nodepsjava-1.7.0-openjdk-1.7.0.51-2.4.5.5.el7.x86_64

Environment variable configuration

Run the following command to configure the environment variable:

Vi/etc/profile (switch to root to edit the file)

After opening the edited file, add the following content at the end of the file:

JAVA_HOME =/usr/java/jdk1.7.0 _ 71 (this is my installation directory, which can be customized)

JRE_HOME =/usr/java/jdk1.7.0 _ 71/jre

PATH = $ PATH: $ JAVA_HOME/bin

CLASSPATH =.: $ JAVA_HOME/lib/dt. jar: $ JAVA_HOME/lib/tools. jar

ExportJAVA_HOMEJRE_HOMEPATHCLASSPATH

Save and exit. Finally, run the following command to make the configuration take effect.

Source/etc/profile

Echo $ JAVA_HOME $ PATH to check whether the configuration is correct

Install the ssh service

# Rpm-qa | grepssh check whether the SSH package is installed. If no SSH package is installed, yuminstallopenssh-server

# Chkconfig -- listsshd check whether SSHD is set to start at the current running level

# Chkconfig -- level2345sshdon. If it is not set to start, set it.

# Servicesshdrestart restart

# Netstat-antp | grepsshd check whether port 22 is enabled. Confirm.

# Iptables-nL check whether 22 ports are allowed.

# Setup ----> set the firewall to allow access if it is not enabled.

Configure the hadoopssh key

# Su-hadoop: switch to a hadoop user;

$ Ssh-keygen-trsa

$ Cat ~ /. Ssh/id_rsa.pub> ~ /. Ssh/authorized_keys

$ Chmod0600 ~ /. Ssh/authorized_keys

$ Exit

Install hadoop

Download the source file in the corresponding format from the hadoop official website. I use hadoop2.6.0. After decompression, grant the current user the execution permission to the obtained folder.

Add executable permissions to all user groups of files in chmoda + x Files

Chown-Rliu/usr/local/hadoop-2.6.0/(liu is the current user name)

Then the three installation modes of hadoop2.6.0 can refer to the official document http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html#YARN_on_Single_Node

I installed it in pseudo-distributed mode.

Run the Wordcount instance to create two files in a folder

For example, I create two text files in the/home/liu/practice folder.

$ Echo "HelloWorld"> file1.txt

$ Echo "HelloHadoop"> file2.txt

$ Ll (you can view the created file. You can use cat to view the file content)

Create Input Folder

$ Hadoopfs-mkdir/user/input

$ Hadoopfs-ls/user (displays the created input directory information)

Copy the text file created above to the input directory

$ Hadoopfs-put/home/liu/practice/file *. txt/user/input

$ Hadoopfs-ls/user/input (displays the copied content)

Run and view results

$ Hadoopjar/usr/local/hadoop/hadoop-examles-2.6.0.jarwordcount/user/input/user/output

$ Hadoopfs-ls/user/output (view the output content and the result is in the part-r-00000)

$ Hadoopfs-cat/user/output/part-r-00000 (view output results)