Centos-optimized Hadoop

Centos-optimized Hadoop
GuideHadoop is a distributed computing platform that allows users to easily deploy and use it. Users can easily develop and run applications that process massive data on Hadoop, this section describes how to install and optimize Supper Hadoop in centos 6.7.

1. Select the system installation type and customize additional packages

Go to the interface shown in 1-1. In the upper part, select different system installation types customized by the system. The default value is "Desktop". Here we select "Minimal" to minimize installation, in the lower part, after the system installation type is determined, select an additional software package group to be added. Select Customize now to Customize it immediately.

Figure 1-1 select the system installation package type and customize additional packages.
In the customized additional package group interface of 1-2, we need to select the big classification on the left, and then select the corresponding small package Group on the right. Note: The content in the packet Group on the right is displayed by default. Select Result (1) and figure (2.

Figure 1-2 custom installation package group selection page I

1.1 based on experience, the installation package should be minimized, that is, no installation is required or not sure whether or not required, so as to maximize system security.1.2 if some package groups or other partners are not selected during the installation process

After installation, you can add a package group that is not installed at the time of installation as follows:

yum groupinstall "Compatibility libraries" "Base" "Development tools"yum groupinstall "debugging Tools" "Dial-up Networking Support"

You can view the installed components in the yum groupinfo package group.

1.3 check the package selection status when logging on to the system after installing 6.7 below

yum grouplist:Installed Groups:BaseCompatibility librariesDebugging ToolsDevelopment toolsE-mail serverGraphical Administration ToolsHardware monitoring utilitiesLegacy UNIX compatibilityNetworking ToolsPerformance ToolsPerl SupportScientific supportSecurity Tools

In addition to the selected software packages, some necessary software packages are installed by default.

2. Network Settings

[root@centos ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0 DEVICE=eth0HWADDR=00:0C:29:6D:58:E4TYPE=EthernetUUID=5236fdbb-76ee-4f98-90cf-6bed4fb62661ONBOOT=yesNM_CONTROLLED=yesBOOTPROTO=staticIPADDR=192.168.2.200GATEWAY=192.168.2.1NETMASK=255.255.255.0

3. Linux system optimization Supper Hadoop3.1 Environment check

3.1.1 Operating System

# cat /etc/redhat-release CentOS release 6.6 (Final)

3.1.2 all hosts can ping each other

3.1.3 Linux kernel version

# uname -r2.6.32-504.el6.x86_64

3.2 add and install cluster users & close selinux & modify Linux Startup level to 3

useradd hadoopecho "hadoop123"|passwd --stdin hadoopyum grouplistsed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/configsetenforce 0grep SELINUX=disabled /etc/selinux/configgrep id:3:initdefault: /etc/inittab

3.3 optimize the Automatic startup service upon startup & increase file descriptors (Hadoop reads data files in parallel) & sudo permission management

Chkconfig -- list | grep 3: on | wc-lchkconfig -- list | grep 3: on | cut-d ""-f1 for name in 'chkconfig -- list | grep 3: on | cut-d ""-f1 '; do chkconfig $ name off; donechkconfig -- list | grep 3: on # If the System Based on lvm logical volume management needs to enable the lvm2-monitor blk-availability two services for name in crond ntpd lvm2-monitor blk-availability network rsyslog sshd; do chkconfig $ name on; doneiptables-L-n/etc/init. d/iptables stopiptables-L-nchkconfig -- list | grep iptablesecho 'hadoop ALL = (ALL) NOPASSWD: ALL '>/etc/sudoersecho' *-nofile 65535 '>/etc/security/limits. conftail-1/etc/security/limits. conf

3.4 configure the ntp server so that all servers have the same time. If the server is not an intranet server, you can directly install and start ntp.

Yum install htpd-y/etc/init. d/ntpd startchkconfig ntpd on $ ntpq-p # automatic public network time synchronization

If it is an intranet server, you need to configure an ntp server. All servers regularly synchronize time to the ntp server. The intranet ntp server is configured as follows:

Vim/etc/ntp. confrestrict default ignore // ntp cannot be modified or queried by default, and do not receive special packets restrict 127.0.0.1 // give the local machine all permissions restrict 192.168.2.0 mask 255.255.255.0 notrap nomodify // give the local machine the permission to synchronize time server 192.168.2.124 # local clockdriftfile/var/lib /ntp/driftfudge 127.127.1.0 stratum 10

Ntpq-p:

$ ntpq –p

All server servers must be consistent with the ntp service time:

$ Ntpdate 192.168.1.124 # The ntp server has just been configured. It takes about 5 minutes to provide the service normally. The error "no server suitable for synchronization found" is reported"

For all big data servers, set cron to synchronize ntp server time:

$ crontab -l00 1 * * * root /usr/sbin/ntpdate 192.168.1.124 >> /root/ntpdate.log 2>&1

3.5 disable Firewall

/etc/init.d/iptables status/etc/init.d/iptables stopchkconfig iptables off

3.6 disable Swap

swapoff -a

3.7 Linux overcommit-memory

echo 0 > /proc/sys/vm/overcommit_memory

3.8 All hosts perform hosts ip ing & repo files

# tail -6 /etc/hosts192.168.1.124 apache-server archive.cloudera.com archive-primary.cloudera.com192.168.1.201   server1192.168.1.202   server2192.168.1.203   server3192.168.1.204   server4

All hosts synchronize the same hosts file

# scp -r /etc/hosts hostname:/etc/

Repo file synchronization for All Hosts

# scp -r CentOS-httpd.repo cloudera-cdh5.repo hostname:/etc/yum.repos.d/

3.9 swappiness & transparent_hugepage

#sudo echo '0' > /proc/sys/vm/swappiness#sudo sysctl vm.swappiness=0#sudo echo '#by whoami 2015-10-30'>> /etc/sysctl.conf#sudo echo 'vm.swappiness=0'>> /etc/sysctl.conf# cat /sys/kernel/mm/redhat_transparent_hugepage/defrag [always] never# echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag# cat /sys/kernel/mm/redhat_transparent_hugepage/defragalways [never]# echo '#by whoami 2015-10-30'>> /etc/rc.local# echo 'echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag ' >> /etc/rc.local

4. clone a VM

The startup of the Virtual Machine eth0 cannot be solved. It is because the cloned Mac address is consistent, resulting in a conflict and cannot be started. The error message is as follows:

Bring up interface eth0: Device eth0 does not seem to be present,delaying initialization.

Delete the configuration of two ifcfg-eth0 and restart the system:

rm -f /etc/udev/rules.d/70-persistent-net.rules

Delete/etc/sysconfig/network-scripts/ifcfg-eth0, HWADDR, UUID two items, restart the network card to solve.

／etc/init.d/network restart

From: http://www.itweet.cn/2016/05/14/Install-and-optimization-linux-system/

Address: http://www.linuxprobe.com/centos-reduce-hadoop.html