Run hadoop with cloudera manager 5.2 on centos 6.5

For partition consideration, do not use LVM
Root --> 40 GB

VaR --> 100 GB

Swap -- 2x system memory

Ram --> 8 GB

Master node:
Raid 10, dual Ethernet cards, dual power supplies, etc.

Slave node:
1. Raid is not necessary

2. HDFS partition, not using LVM
/Etc/fstab -- ext3 defaults, noatime
Mount to/data/n/dfs/dn, for n = 0, 1, 2... (one partition per disk)

Cloudera CDH Repository:

Http://archive.cloudera.com/cdh5
Http://archive-primary.cloudera.com/cm5
Http://archive.cloudera.com/gplextras5

Cloudera parcel Repository:

Http://archive.cloudera.com/cdh5/parcels/latest
Http://archive.cloudera.com/gplextras5/parcels/latest
Http://archive.cloudera.com/impala/parcels/latest/
Http://archive.cloudera.com/sqoop-connectors/parcels/latest/
Http://archive.cloudera.com/accumulo-c5/parcels/latest/

On cloudera manager and all cluster nodes:

At least 3 servers for zookeeper, 3 datanodes for HDFS.

1. Disable SELinux and iptables
Service iptables stop
Chkconfig iptables off; chkconfig ip6tables off

Setenforce 0
VI/etc/sysconfig/SELinux
SELinux = disabled

2. Disable Ipv6
Echo "net. ipv6.conf. All. disable_ipv6 = 1">/etc/sysctl. conf
Sysctl-P

3. VI/etc/hosts to add all hosts FQDN, like below:
192.168.1.19 cm5.local cm5
192.168.1.20 master1.local master1 # HDFS namenode
192.168.1.21 master2.local master2 # yarn ResourceManager
192.168.1.22 slave1.local slave1
192.168.1.23 slave2.local slave2

VI/etc/sysconfig/network to set FQDN

Yum-y install NTP

Service ntpd start; chkconfig ntpd on

Cat <EOF>/etc/yum. Repos. d/ISO. Repo
[ISO]
Name = ISO
Base url = http://mirrors.aliyun.com/centos/6.5/ OS /x86_64
Enable = 1
Gpgcheck = 0
EOF

Yum-y install OpenSSH-clients

Echo "VM. swappiness = 0">/etc/sysctl. conf

Sysctl-P

VI/etc/security/limits. conf
* Soft nofile 32000
* Hard nofile 32000
* Soft nproc Unlimited
* Hard nproc Unlimited
* Soft memlock Unlimited
* Hard memlock Unlimited

Reboot to take effect

4.On cloudera manager, We will install MySQL 5.6 and Apache
Rpm-e -- nodeps mysql-libs
Yum-y install libaio Perl
Rpm-IVH MySQL-shared-compat-5.6.20-1.el6.x86_64.rpm
Rpm-IVH MySQL-shared-5.6.20-1.el6.x86_64.rpm
Rpm-IVH MySQL-server-5.6.20-1.el6.x86_64.rpm
Rpm-IVH MySQL-client-5.6.20-1.el6.x86_64.rpm

VI/etc/My. CNF
[Mysqld]
Transaction-isolation = read-committed
Symbolic-links = 0

Key_buffer = 16 m
Key_buffer_size = 32 m
Max_allowed_packet = 32 m
Thread_stack = 256 K
Thread_cache_size = 64
Query_cache_limit = 8 m
Query_cache_size = 64 m
Query_cache_type = 1
Max_connections = 550

Log-bin = mysql-bin
Binlog_format = mixed
Expire_logs_days = 10
Max_binlog_size = 100 m

Read_buffer_size = 2 m
Read_rnd_buffer_size = 16 m
Sort_buffer_size = 8 m
Join_buffer_size = 8 m

# InnoDB settings
Innodb_file_per_table = 1
Innodb_flush_log_at_trx_commit = 2
Innodb_log_buffer_size = 64 m
Innodb_buffer_pool_size = 4G
Innodb_thread_concurrency = 8
Innodb_flush_method = o_direct
Innodb_log_file_size = 512 m

Service MySQL start; chkconfig MySQL on
Cat ~ /. Mysql_secret
Mysqladmin-uroot-P 'oldpassword' password newpassword
Mysql_secure_installation

Install MySQL JDBC ctor:

Tar zxf mysql-connector-java-5.1.33.tar.gz
Mkdir/usr/share/Java
CP mysql-connector-java-5.1.33/mysql-connector-java-5.1.33-bin.jar/usr/share/Java/mysql-connector-java.jar

Mysql-u root-P

# For activity monitor
Create Database Amon default Character Set utf8;
Grant all on Amon. * To 'amon' @ '%' identified by 'amon ';
Grant all on Amon. * To 'amon' @ 'localhost' identified by 'amon ';

# For reports Manager
Create Database RMAN default Character Set utf8;
Grant all on RMAN. * To 'rman '@' % 'identified by 'rman ';
Grant all on RMAN. * To 'rman '@ 'localhost' identified by 'rman ';

# For Hive MetaStore Server
Create Database MetaStore default Character Set utf8;
Grant all on MetaStore. * To 'hive' @ '%' identified by 'hive ';
Grant all on MetaStore. * To 'hive' @ 'localhost' identified by 'hive ';

# For sentry Server
Create Database sentry default Character Set utf8;
Grant all on sentry. * To 'sentry '@' % 'identified by 'sentry ';
Grant all on sentry. * To 'sentry '@ 'localhost' identified by 'sentry ';

# For cloudera navigator audit server
Create Database nav default Character Set utf8;
Grant all on NAV. * To 'nav' @ '%' identified by 'nav ';
Grant all on NAV. * To 'nav' @ 'localhost' identified by 'nav ';

Flush privileges;

Yum-y install httpd

Service httpd start; chkconfig httpd on

Mkdir/var/www/html/cm520

Mkdir/var/www/html/cdh520

Mount-o loop cm520.iso/var/www/html/cm520

Mount-o loop cdh520.iso/var/www/html/cdh520

Cat <EOF>/etc/yum. Repos. d/cm520.repo
[Cm520]
Name = cm520
Baseurl = http: // 192.168.1.19/cm520
Enable = 1
Gpgcheck = 0
EOF

Yum-y install oracle-j2sdk1.7 cloudera-Manager-daemons cloudera-Manager-Server

Ln-S/usr/Java/jdk1.7.0 _ 67-cloudera/usr/Java/Default
Echo 'export java_home =/usr/Java/default'>/etc/profile
Echo 'export Path = $ java_home/bin: $ path'>/etc/profile
Source/etc/profile

/Usr/share/CMF/Schema/scm_prepare_database.sh mysql-uroot-ppassword cm5 cm5 cm5

Service cloudera-SCM-Server start

Wait several minutes, then open http: // 192.168.1.19: 7180

Username/password: admin/admin

Reboot to confirm it's OK, double check cloudera Manager Web Access

If it's OK

Yum-y install cloudera-Manager-agent
Service cloudera-SCM-Agent start

5.On all cluster nodes

Cat <EOF>/etc/yum. Repos. d/cm520.repo
[Cm520]
Name = cm520
Baseurl = http: // 192.168.1.19/cm520
Enable = 1
Gpgcheck = 0
EOF

Cat <EOF>/etc/yum. Repos. d/cdh51_repo
[Cdh520]
Name = cdh520
Baseurl = http: // 192.168.1.19/cdh520
Enable = 1
Gpgcheck = 0
EOF

The yum-y install oracle-j2sdk1.7 cloudera-Manager-Agent cloudera-Manager-daemons

Ln-S/usr/Java/jdk1.7.0 _ 67-cloudera/usr/Java/Default
Echo 'export java_home =/usr/Java/default'>/etc/profile
Echo 'export Path = $ java_home/bin: $ path'>/etc/profile
Source/etc/profile

VI/etc/cloudera-SCM-agent/config. ini
Server_host = cm5.local
Server_port = 7182

Service cloudera-SCM-Agent start

Yum-y install Avro-tools crunch flume-ng hadoop-HDFS-fuse hadoop-hdfs-nfs3 hadoop-httpfs hbase-SOLR hive-hbase hive-webhcat hue-Beeswax hue-hbase hue-Impala hue- pig hue-plugins hue-RDBMS hue-search hue-Spark hue-sqoop hue-Zookeeper Impala-shell kite llama mahout oozie pig-UDF-datafu search sentry SOLR-mapreduce spark- python sqoop sqoop2 whirr

650) This. width = 650; "src =" http://s3.51cto.com/wyfs02/M00/4C/95/wKioL1RAjmfzC8SjAAPg293yWsA741.jpg "Title =" cdh.jpg "alt =" wkiol1rajmfzc8sjaapg293ywsa741.jpg "/>

Install cloudera Management Service on cm5.local

650) This. width = 650; "src =" http://s3.51cto.com/wyfs02/M01/4C/94/wKiom1RAjkriXmyxAAH0LhUP9pU653.jpg "style =" float: none; "Title =" snap6.jpg "alt =" wkiom1rajkrixmyxaah0lhup9pu653.jpg "/>

Now add new cluster

650) This. width = 650; "src =" http://s3.51cto.com/wyfs02/M01/4C/95/wKiom1RAohazKRY1AAISreAsyps256.jpg "Title =" snap14.jpg "alt =" wkiom1raohazkry1aaisreasyps256.jpg "/>

OK

This article is from the ilovecat blog, be sure to keep this source http://hj192837.blog.51cto.com/655995/1565098

Run hadoop with cloudera manager 5.2 on centos 6.5