Centos7 install and deploy the hadoop Environment

Source: Internet
Author: User
Tags hdfs dfs

Hadoop2.7.7 deployment

1. Install centos7 in vmware14 (Procedure omitted)

Ii. Configure Linux

1. Disable the Firewall:



2. view the current Java version, which is



3. Delete openjdk

CommandRpm-Qa | grep JavaSearch for Java-Related Files

Command:Rpm-e-nodepsDelete the built-in JAVA



4. Install JDK

Download the official 1.8jdk

Command rpm-IVH + file name to install JDK

Installation Complete



4. Download hadoop:

Download hadoop from the official website and upload it to the Virtual Machine




5. Continue to create two VMS

Use the clone function of the virtual machine to clone the cmaster in two copies: slave0 and slave1.





Cloning completed




6. Modify the host names cmaster, slave0, and slave1 respectively.

vim /etc/hostname

7. Add domain name ing:

Ifconfig # view the IP addresses of the three VMS
Vim/etc/hosts # Add domain name ing for three machines



Ping different machines separately



Ping successful

3. Install hadooop

1. Decompress hadoop

1 tar -zxvf hadoop-2.7.7.tar.gz

Decompress hadoop on three machines respectively

2. Configure hadoop (required for all three machines)

2.1 edit a file

1 vim /home/krysent/hadoop-2.7.7/etc/hadoop/hadoop-env.sh

Add Java path



2.2 In the/home/krysent/hadoop-2.7.7/etc/hadoop/directory


<property>       <name>hadoop.tmp.dir</name>       <value>/home/krysent/cloudData</value>  </property>  <property>       <name>fs.defaultFS</name>       <value>hdfs://cMaster:8020</value>  </property>

2.3 In the/home/krysent/hadoop-2.7.7/etc/hadoop/directory


<property>    <name>yarn.resourcemanager.hostname</name>    <value>cMaster</value></property><property>    <name>yarn.nodemanager.aux-services</name>    <value>mapreduce_shuffle</value></property>

2.4 rename mapred-site-. xml. Template under the/home/krysent/hadoop-2.7.7/etc/hadoop/directory to mapred-site-. xml and add in the configuration Tag:

<property>       <name>mapreduce.farmework.name</name>       <value>yarn</value></property>

4. Start hadoop

1. master node cmaster format:

/home/krysent/hadoop-2.7.7/bin/hdfs namenode -format

2. The master node cmaster starts the storage master service namenode and Resource Management Master service ResourceManager:

1 /home/krysent/hadoop-2.7.7/sbin/hadoop-daemon.sh start namenode3 /home/krysent/hadoop-2.7.7/sbin/yarn-daemon.sh start resourcemanager

3. Start the storage slave service datanode and Resource Management slave service nodemanager from the node (both slave0 and slave1 are required ):

/home/krysent/hadoop-2.7.7/sbin/hadoop-daemon.sh    start  datanode
/home/krysent/hadoop-2.7.7/sbin/yarn-daemon.sh start nodemanager

V. Test hadoop:

[[email protected] hadoop-2.7.7]$ /usr/java/jdk1.8.0_191-amd64/bin/jps17826 Jps9942 ResourceManager8908 NameNode
[[email protected] hadoop]$ /usr/java/jdk1.8.0_191-amd64/bin/jps15890 Jps8501 DataNode8684 NodeManager
[[email protected] hadoop-2.7.7]$ /usr/java/jdk1.8.0_191-amd64/bin/jps8578 NodeManager8707 DataNode15764 Jps



Firefox address bar inputCmaster: 50070View:



Vi. Use

1. Create an in directory in the Cluster

/home/krysent/hadoop-2.7.7/bin/hdfs dfs -mkdir /in

2. upload local files to HDFS

/home/krysent/bin/hdfs dfs -put /home/krysent/hadoop-2.7.7/etc/hadoop/* /in

3. Use the wordcount sample program to calculate data

bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.7.jar wordcount /in /out/wc-01

View Firefox:


Centos7 install and deploy the hadoop Environment

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.