hadoop2.6 Virtual machine building and application to the win platform Eclipse Development

The journey is a hard, brain-free work of toil ~

First of all, my platform is built on the CENOS7 virtual machine, using hadoop2.6, the deployment plan:

Host	Role
Namenode1:192.168.80.101	Namenode,resourcemanager
datanode1:192.168.80.102	Datanode,nodemanager
datanode2:192.168.80.103	Datanode,nodemanager

0. Need hadoop2.6, Jdk1.7,hadoop-eclipse-plugin-2.6.0.jar, some files in the wrong time

1. About the virtual machine's network settings, use NAT mode: File/etc/sysconfig/network-scripts/ifcfg-eno*

Lo network card setting off, change the onboot=yes to No

Type=ethernethwaddr=00:0c:29:9e:66:9c#bootproto=dhcpbootproto=staticdefroute=yespeerdns=yespeerroutes=yesipv4_ Failure_fatal=noipv6init=yesipv6_autoconf=yesipv6_defroute=yesipv6_peerdns=yesipv6_peerroutes=yesipv6_failure_ Fatal=noname=eno16777736uuid=98f69bcb-c555-40a4-ac50-d6cf6a2fab73device=eno16777736#onboot=noonboot=yesipaddr0 =192.168.80.101prefix0=24gateway0=192.168.80.1

sudo service network restart

2. For SSH-free login, refer to other materials, or a previous essay. In fact, the password-free login is not necessary to build a Hadoop platform, but can easily open the node

3.JDK configuration is also very simple, after unzipping, write the configuration file path sudo vi/etc/profile

(Export java_home=

Export path= $PATH: $JAVA _home)

Update Source Vi/etc/profile

After success, enter Java on the command console to run successfully.

4. Configure Hadoop

4.1 Writing Hadoop paths to system files

4.2 Changing the configuration file Hadoop_home\etc\hadoop\core-site.xml

The dir of the cache file needs to be created beforehand,

To remember the HDFs port number here,

< Property><name>Fs.defaultfs</name><value>hdfs://namenode1:9000</value></ Property>< Property><name>Hadoop.tmp.dir</name><value>/home/boy/hadoop/hadoop-2.6.0/tem</value></ Property>< Property>  <name>Hadoop.proxyuser.root.hosts</name>  <value>Namenode1</value></ Property>< Property>  <name>Hadoop.proxyuser.root.groups</name>  <value>*</value></ Property>

Premise: The mapping of hosts and URLs on all machines is available, file/etc/hosts

127.0.0.1   localhost www # localhost.localdomain localhost4 localhost4.localdomain4#::1         localhost Localhost.localdomain Localhost6 localhost6.localdomain6192.168.80.101 namenode1192.168.80.102 datanode1192.168.80.103 Datanode2

Configuration files should be based on their own platform to make the corresponding changes, about the explanation, please refer to the other. There's a lot of information online

4.3 Configuration file Hdfs-site.xml

< Property><name>Dfs.replication</name><value>2</value></ Property>< Property><name>Dfs.permissions</name><value>False</value></ Property>< Property><name>Dfs.name.dir</name><value>/home/boy/hadoop/hadoop-2.6.0/tem/name1</value>#hadoop的name目录路径<Description> </Description></ Property>< Property><name>Dfs.data.dir</name><value>/home/boy/hadoop/hadoop-2.6.0/tem/data1</value><Description> </Description></ Property>

Dfs.replication: Number of copies of data, in the cluster is two data nodes, the default is 3, the number can not be more than the number of data nodes

Dfs.permissions: Whether to verify visitor permissions

The next two is where the data files are stored, without having to create them beforehand

4.4yarn-site.xml

The ResourceManager port number 8032 Here is also important, and later eclipse will use

Port 8088 is a Web Access port for Hadoop marreduce

< Property>          <name>Yarn.nodemanager.aux-services</name>          <value>Mapreduce_shuffle</value>      </ Property>      < Property>          <name>Yarn.resourcemanager.address</name>          <value>namenode1:8032</value>      </ Property>      < Property>          <name>Yarn.resourcemanager.scheduler.address</name>          <value>namenode1:8030</value>      </ Property>      < Property>          <name>Yarn.resourcemanager.resource-tracker.address</name>          <value>namenode1:8031</value>      </ Property>      < Property>          <name>Yarn.resourcemanager.admin.address</name>          <value>namenode1:8033</value>      </ Property>      < Property>          <name>Yarn.resourcemanager.webapp.address</name>          <value>namenode1:8088</value>      </ Property> 

4.5mapred-sit.xml

< Property><name>Mapreduce.framework.name</name><value>Yarn</value></ Property>< Property>    <name>Mapreduce.jobtracker.http.address</name>    <value>namenode1:50030</value>  </ Property>< Property><name>Mapreduce.jobhistory.address</name><value>namenode1:10020</value></ Property>< Property><name>Mapreduce.jobhistory.webapp.address</name><value>namenode1:19888</value></ Property>< Property><name>Mapreduce.jobhistory.intermediate-done-dir</name><value>/mr-history/tmp</value></ Property>< Property><name>Mapreduce.jobhistory.done-dir</name><value>/mr-history/done</value></ Property>

4.6 Modify the Java_home in the hadoop-env.sh file for the java_home of the machine

4.7 To here configuration complete, send or copy all the configuration files to other hosts, must be the same, different words mast node will be error, or data node will not start up, but if the data node error, error information can be seen through the console log file location, you can view the exception.

hadoop2.6 Virtual machine building and application to the win platform Eclipse Development