Basic Hadoop tutorial

Basic Hadoop tutorial

This document uses the Basic Environment configuration of the K-Master server as an example to demonstrate user configuration, sudo permission configuration, network configuration, firewall shutdown, and JDK installation. Follow these steps to complete KVMSlave1 ~ The Basic Environment configuration of the KVMSlave3 server.

Development Environment

Hardware environment: Four CentOS 6.5 servers (one Master node and three Slave nodes)
Software environment: Java 1.7.0 _ 45, Hadoop-1.2.1

1. installation environment

Hardware environment: Four CentOS 6.5 servers (one Master node and three Slave nodes)

Software environment: Java 1.7.0 _ 45, hadoop-1.2.1

2. User Configuration

1) Add a user

[Hadoop @ K-Master hadoop] $ adduser hadoop # create a hadoop user [hadoop @ K-Master hadoop] $ passwd hadoop # Set a hadoop User Password

2) create a working group

[Hadoop @ K-Master hadoop] $ groupadd hadoop # create a hadoop workgroup

3) Add a Working Group to existing users

[hadoop@K-Master hadoop]$ usermod -G hadoop hadoop

2. sudo permission Configuration

1) create a user group admin

[hadoop@K-Master hadoop]# groupadd admin

2) add existing users to the admin user group

[hadoop@K-Master hadoop]# usermod -G admin,hadoop hadoop

3) grant write permission to the/etc/sudoers file

[hadoop@K-Master hadoop]# chmod u+w /etc/sudoers 

4) edit the/etc/sudoers file

[Hadoop @ K-Master hadoop] # vi/etc/sudoers by default, there is only one configuration: root ALL = (ALL) ALL add a configuration below: % admin ALL = (ALL) ALL

In this way, the admin user group has the sudo permission. hadoop users belonging to the admin user group also have the sudo permission.

5) Permission reduction after editing

[Hadoop @ K-Master hadoop] $ chmod u-w/etc/sudoers

3. Network Configuration

1) configure the IP address

The detailed configuration information is as follows:

[Hadoop @ K-Master hadoop] $ su hadoop # Switch to hadoop user [hadoop @ K-Master hadoop] $ sudo vi/etc/sysconfig/network-scripts/ifcfg-eth0HWADDR = 06: 8D: 30: 00: 00: 27 TYPE = EthernetBOOTPROTO = staticIPADDR = 192.168.100.147PREFIX = 24 GATEWAY = 192.168.100.1DNS1 = Signature = yesIPV6INIT = noNAME = eth0UUID = Signature = yesLAST_CONNECT = 1411901185

2) restart the network service to make the network settings take effect.

[hadoop@K-Master hadoop]$ sudo service network restartShutting down interface eth0:  Device state: 3 (disconnected)                                                    [  OK  ]Shutting down loopback interface:                   [  OK  ]Bringing up loopback interface:                     [  OK  ]Bringing up interface eth0:  Active connection state: activatedActive connection path: /org/freedesktop/NetworkManager/ActiveConnection/1                                                    [  OK  ]

3) Test the IP Address Configuration.

Run the ifconfig command to view the IP address of the network. The following information shows that the IP address of the eth0 wireless network adapter is 192.168.100.147, which is consistent with the IP address configured above, indicating that the IP address is successfully configured.

[hadoop@K-Master ~]$ ifconfigeth0  Link encap:Ethernet  HWaddr 06:8D:30:00:00:27  inet addr:192.168.100.147  Bcast:192.168.100.255  Mask:255.255.255.0  inet6 addr: fe80::48d:30ff:fe00:27/64 Scope:Link  UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1  RX packets:59099169 errors:0 dropped:0 overruns:0 frame:0  TX packets:30049168 errors:0 dropped:0 overruns:0 carrier:0  collisions:0 txqueuelen:1000  RX bytes:12477388443 (11.6 GiB)  TX bytes:8811418526 (8.2 GiB)loLink encap:Local Loopback  inet addr:127.0.0.1  Mask:255.0.0.0  inet6 addr: ::1/128 Scope:Host  UP LOOPBACK RUNNING  MTU:16436  Metric:1  RX packets:2266013 errors:0 dropped:0 overruns:0 frame:0  TX packets:2266013 errors:0 dropped:0 overruns:0 carrier:0  collisions:0 txqueuelen:0  RX bytes:666482169 (635.6 MiB)  TX bytes:666482169 (635.6 MiB)

4) modify the Host Name

[hadoop@K-Master hadoop]$ sudo vi /etc/sysconfig/networkNETWORKING=yesNETWORKING_IPV6=noHOSTNAME=Master[hadoop@K-Master hadoop]$ sudo vi /etc/hosts127.0.0.1               localhost.localdomain::1                     hdirect30 hdirect30192.168.100.201         K-Master

5) restart the host to make the host name take effect.

[hadoop@K-Master hadoop]$ sudo reboot

4. Disable the Firewall

Disable the firewall of all machines in the cluster before starting the cluster. Otherwise, the firewall is automatically disabled after datanode is enabled.

1) view the Firewall Status

[hadoop@K-Master ~]$ sudo service iptables statusiptables: Firewall is not running.

2) disable the Firewall

[hadoop@K-Master hadoop]$ sudo service iptables stopiptables: Setting chains to policy ACCEPT: filter   [  OK  ]iptables: Flushing firewall rules:                  [  OK  ]iptables: Unloading modules:                        [  OK  ]

3) permanently disable the Firewall

[hadoop@K-Master hadoop]$ sudo chkconfig iptables off

4) Disable SELINUX

[hadoop@K-Master hadoop]$ sudo vi /etc/selinux/configSELINUX=disabled

5. Install JDK

1) decompress

[hadoop@K-Master ~]$ scp hadoop@192.168.0.201:/home/hadoop/jdk-7u65-linux-x64.rpm .[hadoop@K-Master ~]$ sudo rpm -ivh jdk-7u65-linux-x64.rpm

2) edit the "/etc/profile" file and add the "JAVA_HOME", "CLASSPATH", and "PATH" content of Java to the file.

[hadoop@K-Master ~]$ sudo vim /etc/profile#JAVAexport JAVA_HOME=/usr/java/jdk1.7.0_65export JRE_HOME=$JAVA_HOME/jreexport CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib:$JRE_HOME/libexport PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin#HADOOPexport HADOOP_HOME=/usr/hadoop-1.2.1export PATH=$PATH:$HADOOP_HOME/binexport HADOOP_HOME_WARN_SUPPRESS=1

3) make the configuration file take effect

[hadoop@K-Master ~]$ source /etc/profile

For more details, please continue to read the highlights on the next page:

-------------------------------------- Split line --------------------------------------

Tutorial on standalone/pseudo-distributed installation and configuration of Hadoop2.4.1 under Ubuntu14.04

Install and configure Hadoop2.2.0 on CentOS

Build a Hadoop environment on Ubuntu 13.04

Cluster configuration for Ubuntu 12.10 + Hadoop 1.2.1

Build a Hadoop environment on Ubuntu (standalone mode + pseudo Distribution Mode)

Configuration of Hadoop environment in Ubuntu

Detailed tutorial on creating a Hadoop environment for standalone Edition

Build a Hadoop environment (using virtual machines to build two Ubuntu systems in a Winodws environment)

1 2 3 4 5 6 7 8 9 Next Page

[Content navigation]
Page 1: Hadoop server infrastructure	Page 1: Hadoop standalone Mode Construction
Page 1: Hadoop pseudo-Distributed Environment Construction	Page 1: Hadoop's fully distributed environment
Page 1: Word Count of Hadoop	Page 1: Hadoop single table join query
Page 1: One Hadoop-related query	Page 1: Multi-join query by one of Hadoop
Page 1: Hadoop inverted index