Offline installation of Cloudera Manager5.3.4 and CDH5.3.4

Offline installation of Cloudera Manager5.3.4 and CDH5.3.4

During this period of time, I have been installing CDH. The third installation was successful. There were many problems for the first time, but they were uninstalled later. The second Installation failed because they were not uninstalled. Later, the system was simply reinstalled, and the pure system was directly used for installation. A person ran to the school data center to install the system, which was so miserable.

Record the installation process.

System Environment
Operating System: CentOS 6.5x64. The configuration is as follows:

Cloudera Manager: 5.3.4 CDH: 5.3.4

Preparations (system environment setup)
Required Software cloudera-manager-el6-cm5.3.4_x86_64.tar.gz, mysql-connector-java-5.1.25-bin.jar
1. Set static IP addresses

Vi/etc/sysconfig/network-scripts/ifcfg-eth0

Because I have already set it when I reinstall the system, I will not set it here.

2. Modify the Host Name

Hostname Host Name (currently effective)
Vi/etc/sysconfig/network (Effective next time)

[Root @ Hadoop2 opt] # cat/etc/sysconfig/network
NETWORKING = yes
HOSTNAME = hadoop2
GATEWAY = 172.23.253.1
NTPSERVERARGS = iburst

Because I have already set it when I reinstall the system, I will not set it here.

3. Bind the IP address and HostName

Operate on hadoop1 Node

Vi/etc/hosts
172.23.253.20 hadoop1
172.23.253.22 hadoop2
172.23.253.23 hadoop3
172.23.253.24 hadoop4
172.23.253.25 hadoop5
172.23.253.26 hadoop6

Copy to another node (hadoop2 ~ Hadoop6)

Scp/etc/hosts hadoop2:/etc/
Scp/etc/hosts hadoop6:/etc/

4. Set SSH password-free login (all nodes)

1. regenerate the key (each node)

[Root @ hadoop1/] # cd/root/. ssh/
[Root @ hadoop1. ssh] # ssh-keygen-t rsa
[Root @ hadoop1. ssh] # cat id_rsa.pub> authorized_keys

Test whether ssh localhost is successful
Other nodes Generate Keys in the same way. 2. Set password-free logon between two nodes
Hadoop2 ~ Hadoop6 copies its public key to the target host

Ssh-copy-id-I hadoop1

Hadoop1: Copy your authorized_keys to another host, hadoop2 ~ Hadoop6, so that you can log on with each other without a password.

Scp/root/. ssh/authorized_keys hadoop2:/root/. ssh/
Scp/root/. ssh/authorized_keys hadoop6:/root/. ssh/

5. Install Oracle Java

CentOS comes with Open Jdk. However, to run CDH5, you must use the jdk of Oracle and support Java 7. When installing jdk, you must use rpm instead of tar. Due to the rpm installation method, two new soft connections are created under/usr/lib.
First, uninstall the built-in OpenJdk, use rpm-qa | grep java to query java-related packages, and use rpm-e -- nodeps package name to unmount them.
As the rpm package does not require us to configure environment variables, we only need to configure a global JAVA_HOME variable and execute the command:
Echo "JAVA_HOME =/usr/java/latest/">/etc/environment
Run the java-version and javac-version commands to check whether the Jdk is correctly installed. 1. Uninstall the system with OPEN-JDK (all nodes)

[Root @ hadoop1. ssh] # rpm-qa | grep java
Java-1.7.0-openjdk-1.7.0.45-2.4.3.3.el6.x86_64
Tzdata-java-2013g-1.el6.noarch
Java-1.6.0-openjdk-1.6.0.0-1.66.1.13.0.el6.x86_64
[Root @ hadoop1. ssh] # rpm-e -- nodeps java-1.7.0-openjdk-1.7.0.45-2.4.3.3.el6.x86_64
Tzdata-java-2013g-1.el6.noarch java-1.6.0-openjdk-1.6.0.0-1.66.1.13.0.el6.x86_64

2. Install the JDK of Oracle (all nodes)

[Root @ hadoop1 opt] # rpm-ivh jdk-7u80-linux-x64.rpm
Preparing... ######################################## ### [100%]
1: jdk ####################################### #### [100%]
Unpacking JAR files...
Rt. jar...
Jsse. jar...
Charsets. jar...
Tools. jar...
Localedata. jar...
Jfxrt. jar...
[Root @ hadoop1 opt] # echo "JAVA_HOME =/usr/java/latest/">/etc/environment
[Root @ hadoop1 opt] # java-version
Java version "1.7.0 _ 80"
Java (TM) SE Runtime Environment (build 1.7.0 _ 80-b15)
Java HotSpot (TM) 64-Bit Server VM (build 24.80-b11, mixed mode)
[Root @ hadoop1 opt] # javac-version
Javac 1.7.0 _ 80

6. install and configure MySql (master node)

Use yum install mysql-server to install the mysql server.
Set chkconfig mysqld on,
And start mysql service mysqld start,
Set the initial root password mysqladmin-u root password 'xxxx' as prompted '.

[Root @ hadoop1 opt] # yum install mysql-server
[Root @ hadoop1 opt] # chkconfig mysqld on
[Root @ hadoop1 opt] # service mysqld start
[Root @ hadoop1 opt] # service mysqld status

[Root @ hadoop1 opt] # mysqladmin-u root password 'liguodong'

Go to the mysql command line mysql-uroot-pxxxx and create the following database:

[Root @ hadoop1 opt] # mysql-uroot-pliguodong

Mysql> create database hive default charset utf8 COLLATE utf8_general_ci;
Query OK, 1 row affected (0.00 sec)

Mysql> create database amon default charset utf8 COLLATE utf8_general_ci;
Query OK, 1 row affected (0.00 sec)

Mysql> create database hue default charset utf8 COLLATE utf8_general_ci;
Query OK, 1 row affected (0.00 sec)

Set root authorization to access all the above databases.

Mysql> grant all privileges on *. * to 'root' @ 'hadoop1 'identified by 'liguodong' with grant option;
Query OK, 0 rows affected (0.00 sec)

Mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)

Official MySql configuration document of Cloudera:
Http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cm_ig_mysql.html#cmig_topic_5_5

7. Disable firewall and SELinux (all nodes)

Note: It must be executed on all nodes. because too many ports are involved, the firewall is temporarily closed to facilitate installation (the firewall will cause various exceptions in hadoop-related component communication .), After installation, you can set firewall policies as needed to ensure cluster security.
Disable Firewall:
Service iptables stop (temporarily disabled)
Chkconfig iptables off (effective after restart) Disable SELINUX:
(It is also possible to find that the installation process is not closed. I don't know if there will be any problems. Further verification is required)
Setenforce 0 (temporarily effective)
Modify selinux = disabled under/etc/SELINUX/config (it takes effect permanently after restart)

[Root @ hadoop1 opt] # service iptables stop
Iptables: Set the chain to policy ACCEPT: filter [OK]
Iptables: clear firewall rules: [OK]
Iptables: uninstalling module: [OK]
[Root @ hadoop1 opt] # chkconfig iptables off

[Root @ hadoop1 opt] # setenforce 0
[Root @ hadoop1 opt] # vi/etc/selinux/config
[Root @ hadoop1 opt] # cat/etc/selinux // config
SELINUXTYPE = targeted
SELINUX = disabled

8. install and configure the NTP service (all nodes)

All hosts in the cluster must maintain time synchronization. If the time difference is large, it may cause various problems.
The specific ideas are as follows:
The master node acts as the ntp server and the external synchronization time center, and then provides the time synchronization service for all datanode nodes. All datanode nodes are synchronized Based on the master node.
Install related components on all nodes: yum install ntp.
After the configuration is completed, start the startup: chkconfig ntpd on,
Check whether the setting is successful: chkconfig -- list ntpd. If 2-5 is on, the setting is successful.

[Root @ hadoop1 opt] # yum install ntp
[Root @ hadoop1 opt] # chkconfig ntpd on
[Root @ hadoop1 opt] # chkconfig -- list ntpd
Ntpd 0: off 1: off 2: Enable 3: Enable 4: Enable 5: Enable 6: Disable

Configure the Intranet NTP server (master node)

Before configuration, use ntpdate to manually synchronize the time, so that the time difference between the local machine and the time center is too large, so that ntpd cannot be synchronized normally. Use 202.112.10.36 as the time center. The command is ntpdate-u 202.112.10.36.

# Ntpdate-u 202.112.10.36
22 Dec 16:52:38 ntpdate [6400]: adjust time server 202.112.10.36 offset 0.012135 sec

The ntp service only has one configuration file. Only useful configurations are provided here, and unnecessary configurations are used # Note out, which is not provided here.

[Root @ hadoop1 opt] # vi/etc/ntp. conf
[Root @ hadoop1 opt] # cat/etc/ntp. conf
Driftfile/var/lib/ntp/drift
Restrict 127.0.0.1
Restrict-6: 1
Restrict default nomodify notrap
Server 202.112.10.36 prefer
Includefile/etc/ntp/crypto/pw
Keys/etc/ntp/keys

The configuration file is complete. Save and exit. Start the service and run the following command:

[Root @ hadoop1 opt] # service ntpd start

Check whether the synchronization is successful. Run the ntpstat command to check the synchronization status. The following status indicates that the synchronization is successfully started:

[Root @ hadoop1 ~] # Ntpstat
Synchronised to NTP server (202.112.10.36) at stratum 3
Time correct to within 133 MS
Polling server every 1024 s

If an exception occurs, wait for several minutes. Generally, the synchronization will take 5 to 10 minutes.

Configure the ntp client (hadoop2 ~ Hadoop6)

Driftfile/var/lib/ntp/drift
Restrict 127.0.0.1
Restrict-6: 1
Restrict default kod nomodify notrap nopeer noquery
Restrict-6 default kod nomodify notrap nopeer noquery
# The host name or ip address of the master node (intranet NTP server)
Server hadoop1

Includefile/etc/ntp/crypto/pw
Keys/etc/ntp/keys

Save and exit. before requesting the server, use ntpdate to synchronize the time: ntpdate-u hadoop1 (Master ntp server)

Synchronization may fail. Please do not worry. Generally, the local ntp server has not started properly. It usually takes 5 to 10 minutes to synchronize properly.
Start service: service ntpd start

[Root @ hadoop2 ~] # Ntpdate-u hadoop1
27 Jun 08:14:38 ntpdate [1899]: adjust time server 172.23.253.20 offset 0.000293 sec

[Root @ hadoop2 ~] # Service ntpd start
[Root @ hadoop2 ~] # Chkconfig ntpd on

Install and configure Cloudera Manager Server and AgentServer (master node)

: Http://archive-primary.cloudera.com/cm5/cm/5/ (choose one for you)
Select the desired region. The next installation is cloudera-manager-el6-cm5.3.4_x86_64.tar.gz. After the download is complete, upload the package to the master node.
Decompress the package to the/opt directory.
Because the cdh5 source will be searched in/opt/cloudera/parcel-repo by default, how to create the local source file of cdh5 will be introduced later.

[Root @ hadoop1 opt] # tar-zxvf cloudera-manager-el6-cm5.3.4_x86_64.tar.gz

Add a cloudera-scm user to all nodes:

[Root @ hadoop1 lib] # useradd -- system -- home =/opt/cm-5.3.4/run/cloudera-scm-server/-- no-create-home -- shell =/bin/false
-- Comment "Cloudera SCM User" cloudera-scm

Create a database for Cloudera Manager 5:
Because we use the Mysql database, we need to download the Mysql JDBC driver. The version downloaded from the official website is: mysql-connector-java-5.1.25-bin.jar put in the/opt/cm-5.3.4/share/cmf/lib/directory.

[Root @ hadoop1 opt] # mv mysql-connector-java-5.1.25-bin.jar/opt/cm-5.3.4/share/cmf/lib/

Then, create a database:

[Root @ hadoop1 opt] #/opt/cm-5.3.4/share/cmf/schema/scm_prepare_database.sh mysql
Cm-hlocalhost-uroot-pliguodong -- scm-host localhost scm

The format is: scm_prepare_database.sh Database Type Database Server username and password-scm-host Cloudera_Manager_Server machine where the following three do not know what to represent, directly copy the official website. Enable Cloudera Manager 5 Server:

[Root @ hadoop1 lib] #/opt/cm-5.3.4/etc/init. d/cloudera-scm-server start

Note:
Do not immediately shut down or restart the server when it is started for the first time, because the table and data will be automatically created when it is started for the first time,
If you exit midway through for special reasons, delete all tables and data before starting again. Otherwise, the startup may fail.

Agent configuration (all nodes)

Enable the Cloudera Manager 5 Agents client. Now the master node modifies the Host Name of the master node under serer_host under/opt/cm-5.3.4/etc/cloudera-scm-agent/config. ini.

[Root @ hadoop1 lib] # vi/opt/cm-5.3.4/etc/cloudera-scm-agent/config. ini
Server_host = hadoop1

First copy/opt/cm-5.3.4 to hadoop2 ~ On the hadoop6 node,

Scp-r/opt/cm-5.3.4 hadoop2:/opt/
···
Scp-r/opt/cm-5.3.4 hadoop6:/opt/

Wait until the copy is successful and start the Agent on all nodes: (Note that the Agent must be started with the Administrator permission)

[Root @ hadoop1 lib] #/opt/cm-5.3.4/etc/init. d/cloudera-scm-agent start

When the browser starts the Cloudera Manager 5 console (the default port number is 7180), the login page will be displayed after it is started successfully.

For more details, please continue to read the highlights on the next page:

1 2 Next Page