CDH 2, Cloudera Manager installation

1. Cloudera Manager

Cloudera Manager is an end-to-end application that manages CDH.
Role
– Management
– Monitoring
– Diagnostics
– Integration

• architecture

　　　　　　

Server
– Management Console server and application logic
– Responsible for software installation, configuration, start-up and stop services
– Management Service runs the cluster
Agent
– Installed on each host
– Responsible for starting and stopping processes, configuring, monitoring hosts
Management Service
– A service consisting of a set of roles that performs various monitoring, alerting, and reporting functions
Database
Cloudera Repository
clients
Admin Console
Api

2, pre-installation environment construction

　　1. Network configuration (all nodes)
Vi/etc/sysconfig/network modifying hostname:

　　　　
Restart Network Service via service network restart effective

Vi/etc/hosts, modifying the correspondence between IP and host name

　　　　

　　2. SSH Password-Free login
Master Node Execution:
Ssh-keygen-t Dsa-p "-F ~/.SSH/ID_DSA
Generate no password key pair
Copy the public key to another node, execute
Cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
Test: Master node ssh other node ...; If not successful, then the other nodes in the other node to do their own password-free login: On the node to use the command ssh-keygen-t dsa-p "-F ~/.SSH/ID_DSA

And then repeat the above operation

　3. Turn off the firewall
Temporary shutdown:
Service Iptables Stop
Permanently closed (after reboot):
Chkconfig iptables off

　4. Turn off SELINUX

Temporary shutdown:
Setenforce 0
Modify configuration file/etc/selinux/config (restart effective):
Change Selinux=enforcing to Selinux=disabled
To view the SELINUX status:
1)/usr/sbin/sestatus–v
SELinux status:enabled (enabled: open; Disabled: OFF)
2) Use command: Getenforce

　5. Installing the JDK

This article uses RPM package installation .... Execution:
RPM-IVH jdk-7u80-linux-x64.rpm
To configure the environment variables, modify the/root/.bash_profile:
Export java_home=/usr/java/jdk1.7.0_80
Export path= $JAVA _home/bin: $PATH
Export classpath=.: $JAVA _homde/lib/dt.jar: $JAVA _home/lib/tools.jar

Effect:
Source/etc/profile
View version:
[Email protected] cdh]# java-version
Java Version "1.7.0_80"
Java (TM) SE Runtime Environment (build 1.7.0_80-b15)
Java HotSpot (TM) 64-bit Server VM (build 24.80-b11, Mixed mode)

　　6. Set up NTP

All nodes Install NTP:
Yum Install NTP
To configure boot boot:
Chkconfig ntpd on
Check whether the setting is successful:
Chkconfig--list ntpd (2-5 for on State success)
To set up synchronization:
Ntpdate-u ntp.sjtu.edu.cn (clock server based on the actual environment settings, this document uses the 210.72.145.44-National Time Service Center Server IP address)

　7. Install the configuration MySql

You can use Yum to bring your own MySQL command yum install mysql-server

　8. Download the dependency package

Yum-y install chkconfig python bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gs SAPI Fuse Fuse-libs redhat-lsb

3, CM installation

　　1. Install Cloudera Manager server&agent
Copy cloudera-manager-el6-cm5.4.3_x86_64.tar.gz to all Server, Agent nodes
To create a cm directory:
Mkdir/opt/cloudera-manager
Unzip the CM compression pack:
Tar xvzf cloudera-manager*.tar.gz-c/opt/cloudera-manager

　　2. Create user CLOUDERA-SCM (all nodes)
Perform:
　　　　

Useradd--system--home=/opt/cloudera-manager/cm-5.0/run/cloudera-scm-server  --no-create-home--shell=/bin/ false " Cloudera SCM User " CLOUDERA-SCM

　　3. Configure CM Agent
Modify Server_host and Server_port in file/opt/cloudera-manager/cm-5.4.3/etc/cloudera-scm-agent/config.ini (default 7180 is the same)

　　4. Configure the database for CM Server
Copy the driver package to the directory ( Note that the copy of the past driver package name must be the same as below, otherwise it will be an error ):
CP Mysql-connector-java-5.1.31/mysql-connector-java-5.1.31-bin.jar/usr/share/java/mysql-connector-java.jar

Login MySQL: $ mysql-p mysql-uroot (no password required for first login)

Perform:

mysql> use MySQL;

mysql> Delete from user where user= ';

mysql> Update user Set Password=password (' 123456 ') where user= ' root '; #为mysql的Root用户更改密码

--Open Connection permissions for other clients  if hive and MySQL are on the same server, no authorization is required   

Mysql>Grant all privileges on * * to ' root ' @ '% ' identified by ' 123456 ';

Mysql> Grant All on * * to ' temp ' @ ' percent ' identified by ' temp ' with GRANT option; #如果temp数据库存在, the execution of this sentence is deleted first

Mysql>flush privileges; Refresh Permissions

Quit MySQL and execute the following command

Cd/opt/cloudera-manager/cm-5.4.3/share/cmf/schema

./scm_prepare_database.sh mysql-h myhost1.sf.cloudera.com-utemp-ptemp--scm-host myhost2.sf.cloudera.com SCM SCM SCM

For example:

./scm_prepare_database.sh mysql-h node1-utemp-ptemp--scm-host node1 SCM SCM SCM
(Corresponds to: Database type, database server, user name, password, cmserver node ...)

mysql> drop user ' temp ' @ '% ';
　　　　If the previous step fails or the operation is interrupted, delete all libraries, start over/(ㄒoㄒ)/~~
　　　　

If you install components such as Oozie, you may need to manually create the required databases for the corresponding components, for example:
Create DATABASE ooziecm DEFAULT CHARACTER SET UTF8;
Grant all on ooziecm.* to ' ooziecm ' @ '% ' of ' identified by ' OOZIECM ';

　5. Create a Parcel directory
Manager node Create directory/opt/cloudera/parcel-repo, execute:
Mkdir-p/opt/cloudera/parcel-repo
Chown Cloudera-scm:cloudera-scm/opt/cloudera/parcel-repo
Copy the downloaded files (cdh-5.4.0-1.cdh5.4.0.p0.27-el6.parcel, Cdh-5.4.0-1.cdh5.4.0.p0.27-el6.parcel.sha, Manifest.json) to the directory.
Agent node to create directory/opt/cloudera/parcels, execute:
Mkdir-p/opt/cloudera/parcels
Chown Cloudera-scm:cloudera-scm/opt/cloudera/parcels

　6. Start CM manager&agent Service
Perform:
Manager:/opt/cloudera-manager/cm-5.4.3/etc/init.d/cloudera-scm-server start
Agents:/opt/cloudera-manager/cm-5.4.3/etc/init.d/cloudera-scm-agent start
Access: http://ManagerHost:7180, if you can access (user name, Password: admin), the installation is successful.
It takes a while for the Manager to start successfully, and it takes some time to create the corresponding table in the database.

4, CDH5 5 Installation

After the CM Manager && Agent successfully starts, log in to the front-end page for CDH installation configuration. The free version of CM5 has removed the limit of 50 nodes.

　　　　

　　　　　　　　　　

Here we point out the difference in functionality between different versions.

　　

Select the host that is currently installing CDH

　　　　

Choose the version of Install CDH, where we want to select the version we downloaded and upload to Linux

　　　　

The stage of downloading the installation configuration will be slow and you need to wait patiently

　　　　

Check the correctness, there will be a warning: XXX It is best to resolve the warning, and then restart and then access 7180 to install

　　　　

Choose which services you want to install here, and do not choose to install the services until the installation is successful.

　　　　

Configure the database link, if it is to follow the way down, then the database name, user name, password is temp: Temp Temp Temp

　　　　

This is a catalogue of various services, preferably recorded to facilitate later use

　　　　　　

Then there are a variety of installation, installed in the work will show the following interface

　　　　

Some warning messages and error messages are displayed on the left side of the installation, and it is recommended that all of these warnings be resolved if the enterprise environment is real.

　　　　

　　

CDH 2, Cloudera Manager installation