Cloudera Manager and CDH 5.14.0 Installation Process in CentOS 7
As we all know, the configuration of Apache Hadoop is cumbersome and fragmented. For this reason, Cloudera provides the Clouder Manager tool and encapsulates Apache Hadoop, flume, spark, hive, hbase and other big data products form CDH products with their own characteristics, and then use CM for installation. This facilitates cluster construction and provides cluster monitoring functions.
I. Environment:
1. Three vmwarevms (one as the master node and two as the slave node)
Hserver1n (master node) |
Hserver2n (slave node) |
Hserver3n (slave node) |
CM Server |
|
|
CM Agent |
CM Agent |
CM Agent |
NameNode |
DateNode |
DateNode |
Mysql |
|
|
2. Operating System: CentOS 7
3. Cloudera Manager: 5.14.1
4. CDH: 5.14.0
5. JDK1.8
Ii. Software
1. Cloudera Manager
2. CDH
3. Search for JDK and MySQL by yourself
Iii. System Environment Settings:
Perform the following operations with the root user:
1. Set static IP addresses
[Root @ localhost ~] # Vim/etc/sysconfig/network-scripts/ifcfg-ens33
TYPE = "Ethernet"
BOOTPROTO = "static"
DEFROUTE = "yes"
PEERDNS = "yes"
PEERROUTES = "yes"
20174_failure_fatal = "no"
IPV6INIT = "yes"
IPV6_AUTOCONF = "yes"
20176_defroute = "yes"
IPV6_PEERDNS = "yes"
20176_peerroutes = "yes"
20176_failure_fatal = "no"
20176_addr_gen_mode = "stable-privacy"
NAME = "ens33"
UUID = "edcb54db-f59d-4893-bd8e-8ae60d0ba6f9"
DEVICE = "ens33"
ONBOOT = "yes"
GATEWAY = 192.168.124.2
IPADDR = 192.168.124.136
NETMASK = 255.255.255.0
DNS1 = 202.96.128.86
DNS2 = 223.5.5.5
You also need to set the virtual network:
(1) Click the virtual network editor.
(2) subnet, subnet mask, and NAT settings
(3) Select the VMnet8
(4) restart the network
[Root @ localhost ~] # Systemctl restart network
2. Disable firewall and SELINUX
[Root @ localhost ~] # Vim/etc/selinux/config. config/
Change selinux = enforcing to selinux = disabled.
Restart a VM
3. Install JDK
OpenJDK is installed on CentOS7 by default, so you need to uninstall and then install Oracle JDK.
4. Set the Host Name (CentOS7 and CentOS6 host name settings are different, please refer to the https://www.bkjia.com/Linux/2018-03/151464.htm)
(1)vim /etc/hostname
hserver1n
(2)hostname hserver1n
After a single virtual machine is set, copy the two virtual machines and modify the HostName, IP address, and UUID.
5. Configure the Host files of the three VMS
192.168.124.136 hserver1n
192.168.124.20.hserver2n
192.168.124.138 hserver3n
6. Access two slave nodes through SSH on the master node
7. Configure NTP time synchronization service for all nodes
Time when two slave nodes synchronize the master node
4. Install MySQL
- Mysql only needs to be installed on the master node
- Mysql5.6 a temporary password is generated for the root user when the mysql service starts. It is obtained through the grep 'Password'/var/log/mysqld. log command.
- Configure the password verification policy. I chose not to use the password verification policy to create a simple password.
- Configure the database character set. The default character set I configure is utf8.
5. Install Clouder Manager and CDH
All node operations:
1. Create a directory
$ Sudo mkdir/opt/cloudera-manager
2. decompress the downloaded Clouder Manager to this directory.
$ Sudo tar xzf cloudera-manager * .tar.gz-C/opt/cloudera-manager
3. Create cloudera-scm
Since Cloudera Manager and Managed Services use cloudera-scm by default, you need to create this user
$ Sudo useradd -- system -- home =/opt/cloudera-manager/cm-5.14.1/run/cloudera-scm-server -- no-create-home -- shell =/bin/false -- comment" cloudera SCM User "cloudera-scm
4. Create the Cloudera Manager Service local data storage directory on the master node
$ Sudo mkdir/var/lib/cloudera-scm-server
$ Sudo chown cloudera-scm: cloudera-scm/var/lib/cloudera-scm-server
5. Configure the Agent
Configure the Agent for all nodes, path:/opt/cloudera-manager/cm-5.14.1/etc/cloudera-scm-agent/config. ini: Change server_host to the Host Name of the master node. If the master node port is not customized, do not modify it.
6. Download The mysql-connector-java.jar and save it to the/usr/share/java directory for All Hosts
7. Configure mysql. For more information, see here.
7.1 install mysql on the master node and log on using the mysql Command
7.2 create an scm user and grant all permissions. The password is scm.
Mysql> grant all on *. * to 'scm '@ 'localhost' identified by 'temp' with grant option;
Query OK, 0 rows affected, 1 warning (0.00 sec)
7.3 create a database scm
7.4 initialize cloudera manager
[Root @ hserver1n ~] #/Opt/cloudera-manager/cm-5.14.1/share/cmf/schema/scm_prepare_database.sh mysql scm
JAVA_HOME =/usr/java/jdk1.7.0 _ 80
Verifying that we can write to/opt/cloudera-manager/cm-5.14.1/etc/cloudera-scm-server
Creating SCM configuration file in/opt/cloudera-manager/cm-5.14.1/etc/cloudera-scm-server
Executing:/usr/java/jdk1.7.0 _ 80/bin/java-cp/usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar: /opt/cloudera-manager/cm-5.14.1/share/cmf/schema /.. /lib/* com. cloudera. enterprise. dbutil. dbCommandExecutor/opt/cloudera-manager/cm-5.14.1/etc/cloudera-scm-server/db. properties com. cloudera. cmf. db.
Wed Mar 21 14:09:40 CST 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. according to MySQL 5.5.45 +, 5.6.26 + and 5.7.6 + requirements SSL connection must be established by default if explicit option isn't set. for compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false '. you need either to explicitly disable SSL by setting useSSL = false, or set useSSL = true and provide truststore for server certificate verification.
[Main] DbCommandExecutor INFO Successfully connected to database.
All done, your SCM database is configured correctly!
After reading the last sentence, the initialization is successful.
Note that the scm account cannot be deleted because cm still needs to be used later. To create a temporary user, you need to have CM create a user and a database. In this case, you can delete the created temporary user.
You can run the following command:
/Opt/cloudera-manager/cm-5.14.1/share/cmf/schema/scm_prepare_database.sh mysql-utemp-ptemp scm
8. Move the CDH installation package to the/opt/cloudera/parcel-repo directory of the master node.
Mobile files include:
Change CDH-5.14.0-1.cdh5.14.0.p0.24-el7.parcel.sha1 name to CDH-5.14.0-1.cdh5.14.0.p0.24-el7.parcel.sha
9. Start the CM Server on the master node and the Agent on all nodes.
Path:/opt/cloudera-manager/cm-5.14.1/etc/init. d
[Root @ hserver1n init. d] #./cloudera-scm-server start
Starting cloudera-scm-server: [OK]
[Root @ hserver1n init. d] #./cloudera-scm-agent start
Starting cloudera-scm-agent: [OK]
10. Open the CM Management page at http: // hserver1n: 7180,
The username and password are both admin
11. Select Cloudera Express
12. Select a host
13. Cluster Installation
14. Check whether the host is correct after installation.
There are two items to be modified. One is to modify the swapping threshold value from 30 to 10, and the maximum memory usage has been reached. The other is to disable the transparent large page compression function to improve performance.
15. Select the service to be installed. Here, select the core Hadoop and install other services as needed.
16. Assign roles to all hosts in a balanced manner to reduce the pressure on a host.
17. Create database users and related databases and configure services
Mysql> grant all on *. * to 'anggy '@' % 'identified by '000000' with grant option;
Query OK, 0 rows affected, 1 warning (0.03 sec)
Mysql> create database hive;
Query OK, 1 row affected (0.01 sec)
Mysql> create database hue;
Query OK, 1 row affected (0.01 sec)
Mysql> create database ooz;
Query OK, 1 row affected (0.00 sec)
18. cluster-related service configuration
The default HDFS block size is 128 MB. I only configured 16 MB for testing convenience. Other configurations include Data Directories, log directories, and ports.
19. The cluster is built and related services are started.
20. after the cluster is installed and merged on the initial interface, you can clearly see the services and running status installed in the cluster. The red exclamation mark indicates that some configurations and spaces in the cluster are different from those expected by CM, it can be adjusted according to the actual situation.
This article permanently updates link: https://www.bkjia.com/Linux/2018-03/151491.htm