Cloudera Manager and CDH 5.14.0 Installation Process in CentOS 7

Cloudera Manager and CDH 5.14.0 Installation Process in CentOS 7

As we all know, the configuration of Apache Hadoop is cumbersome and fragmented. For this reason, Cloudera provides the Clouder Manager tool and encapsulates Apache Hadoop, flume, spark, hive, hbase and other big data products form CDH products with their own characteristics, and then use CM for installation. This facilitates cluster construction and provides cluster monitoring functions.

I. Environment:

1. Three vmwarevms (one as the master node and two as the slave node)

Hserver1n (master node)	Hserver2n (slave node)	Hserver3n (slave node)
CM Server
CM Agent	CM Agent	CM Agent
NameNode	DateNode	DateNode
Mysql

2. Operating System: CentOS 7

3. Cloudera Manager: 5.14.1

4. CDH: 5.14.0

5. JDK1.8

Ii. Software

1. Cloudera Manager

2. CDH

3. Search for JDK and MySQL by yourself

Iii. System Environment Settings:

Perform the following operations with the root user:

1. Set static IP addresses

[Root @ localhost ~] # Vim/etc/sysconfig/network-scripts/ifcfg-ens33
TYPE = "Ethernet"
BOOTPROTO = "static"
DEFROUTE = "yes"
PEERDNS = "yes"
PEERROUTES = "yes"
20174_failure_fatal = "no"
IPV6INIT = "yes"
IPV6_AUTOCONF = "yes"
20176_defroute = "yes"
IPV6_PEERDNS = "yes"
20176_peerroutes = "yes"
20176_failure_fatal = "no"
20176_addr_gen_mode = "stable-privacy"
NAME = "ens33"
UUID = "edcb54db-f59d-4893-bd8e-8ae60d0ba6f9"
DEVICE = "ens33"
ONBOOT = "yes"
GATEWAY = 192.168.124.2
IPADDR = 192.168.124.136
NETMASK = 255.255.255.0
DNS1 = 202.96.128.86
DNS2 = 223.5.5.5

You also need to set the virtual network:

(1) Click the virtual network editor.

(2) subnet, subnet mask, and NAT settings

 

(3) Select the VMnet8

(4) restart the network

[Root @ localhost ~] # Systemctl restart network

 

 

2. Disable firewall and SELINUX

[Root @ localhost ~] # Vim/etc/selinux/config. config/
Change selinux = enforcing to selinux = disabled.

Restart a VM

3. Install JDK

OpenJDK is installed on CentOS7 by default, so you need to uninstall and then install Oracle JDK.

4. Set the Host Name (CentOS7 and CentOS6 host name settings are different, please refer to the https://www.bkjia.com/Linux/2018-03/151464.htm)

（1）vim /etc/hostname

     hserver1n

 (2)hostname hserver1n

After a single virtual machine is set, copy the two virtual machines and modify the HostName, IP address, and UUID.

5. Configure the Host files of the three VMS

192.168.124.136 hserver1n

192.168.124.20.hserver2n

192.168.124.138 hserver3n

6. Access two slave nodes through SSH on the master node

7. Configure NTP time synchronization service for all nodes

Time when two slave nodes synchronize the master node

4. Install MySQL

• Mysql only needs to be installed on the master node
• Mysql5.6 a temporary password is generated for the root user when the mysql service starts. It is obtained through the grep 'Password'/var/log/mysqld. log command.
• Configure the password verification policy. I chose not to use the password verification policy to create a simple password.
• Configure the database character set. The default character set I configure is utf8.

5. Install Clouder Manager and CDH

All node operations:

1. Create a directory

$ Sudo mkdir/opt/cloudera-manager

2. decompress the downloaded Clouder Manager to this directory.

$ Sudo tar xzf cloudera-manager * .tar.gz-C/opt/cloudera-manager

3. Create cloudera-scm

Since Cloudera Manager and Managed Services use cloudera-scm by default, you need to create this user

$ Sudo useradd -- system -- home =/opt/cloudera-manager/cm-5.14.1/run/cloudera-scm-server -- no-create-home -- shell =/bin/false -- comment" cloudera SCM User "cloudera-scm

4. Create the Cloudera Manager Service local data storage directory on the master node

$ Sudo mkdir/var/lib/cloudera-scm-server
$ Sudo chown cloudera-scm: cloudera-scm/var/lib/cloudera-scm-server

5. Configure the Agent

Configure the Agent for all nodes, path:/opt/cloudera-manager/cm-5.14.1/etc/cloudera-scm-agent/config. ini: Change server_host to the Host Name of the master node. If the master node port is not customized, do not modify it.

6. Download The mysql-connector-java.jar and save it to the/usr/share/java directory for All Hosts

7. Configure mysql. For more information, see here.

7.1 install mysql on the master node and log on using the mysql Command

7.2 create an scm user and grant all permissions. The password is scm.

Mysql> grant all on *. * to 'scm '@ 'localhost' identified by 'temp' with grant option;
Query OK, 0 rows affected, 1 warning (0.00 sec)

7.3 create a database scm

7.4 initialize cloudera manager

[Root @ hserver1n ~] #/Opt/cloudera-manager/cm-5.14.1/share/cmf/schema/scm_prepare_database.sh mysql scm

JAVA_HOME =/usr/java/jdk1.7.0 _ 80
Verifying that we can write to/opt/cloudera-manager/cm-5.14.1/etc/cloudera-scm-server
Creating SCM configuration file in/opt/cloudera-manager/cm-5.14.1/etc/cloudera-scm-server
Executing:/usr/java/jdk1.7.0 _ 80/bin/java-cp/usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar: /opt/cloudera-manager/cm-5.14.1/share/cmf/schema /.. /lib/* com. cloudera. enterprise. dbutil. dbCommandExecutor/opt/cloudera-manager/cm-5.14.1/etc/cloudera-scm-server/db. properties com. cloudera. cmf. db.
Wed Mar 21 14:09:40 CST 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. according to MySQL 5.5.45 +, 5.6.26 + and 5.7.6 + requirements SSL connection must be established by default if explicit option isn't set. for compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false '. you need either to explicitly disable SSL by setting useSSL = false, or set useSSL = true and provide truststore for server certificate verification.
[Main] DbCommandExecutor INFO Successfully connected to database.
All done, your SCM database is configured correctly!

After reading the last sentence, the initialization is successful.

Note that the scm account cannot be deleted because cm still needs to be used later. To create a temporary user, you need to have CM create a user and a database. In this case, you can delete the created temporary user.

You can run the following command:

/Opt/cloudera-manager/cm-5.14.1/share/cmf/schema/scm_prepare_database.sh mysql-utemp-ptemp scm

8. Move the CDH installation package to the/opt/cloudera/parcel-repo directory of the master node.

Mobile files include:

Change CDH-5.14.0-1.cdh5.14.0.p0.24-el7.parcel.sha1 name to CDH-5.14.0-1.cdh5.14.0.p0.24-el7.parcel.sha

9. Start the CM Server on the master node and the Agent on all nodes.

Path:/opt/cloudera-manager/cm-5.14.1/etc/init. d

[Root @ hserver1n init. d] #./cloudera-scm-server start
Starting cloudera-scm-server: [OK]

[Root @ hserver1n init. d] #./cloudera-scm-agent start
Starting cloudera-scm-agent: [OK]

10. Open the CM Management page at http: // hserver1n: 7180,

The username and password are both admin

11. Select Cloudera Express

12. Select a host

13. Cluster Installation

 

14. Check whether the host is correct after installation.

There are two items to be modified. One is to modify the swapping threshold value from 30 to 10, and the maximum memory usage has been reached. The other is to disable the transparent large page compression function to improve performance.

15. Select the service to be installed. Here, select the core Hadoop and install other services as needed.

 

16. Assign roles to all hosts in a balanced manner to reduce the pressure on a host.

17. Create database users and related databases and configure services

Mysql> grant all on *. * to 'anggy '@' % 'identified by '000000' with grant option;
Query OK, 0 rows affected, 1 warning (0.03 sec)

Mysql> create database hive;
Query OK, 1 row affected (0.01 sec)

Mysql> create database hue;
Query OK, 1 row affected (0.01 sec)

Mysql> create database ooz;
Query OK, 1 row affected (0.00 sec)

 

18. cluster-related service configuration

The default HDFS block size is 128 MB. I only configured 16 MB for testing convenience. Other configurations include Data Directories, log directories, and ports.

19. The cluster is built and related services are started.

20. after the cluster is installed and merged on the initial interface, you can clearly see the services and running status installed in the cluster. The red exclamation mark indicates that some configurations and spaces in the cluster are different from those expected by CM, it can be adjusted according to the actual situation.

 

This article permanently updates link: https://www.bkjia.com/Linux/2018-03/151491.htm