Data acquisition + Dispatch: Cdh5.8.0+mysql5.7.17+hadoop+sqoop+hbase+oozie+hue

First, pre-preparation

1. Operating system version

# Cat/etc/redhat-release
CentOS Release 6.5 (Final)

# Uname–a
Linux enc-bigdata05 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 03:15:09 UTC x86_64 x86_64 x86_64 gnu/linux

2. Pre-Software preparation

2-1. JDK Version: jdk-7u80-linux-x64.tar.gz

[jdk1.7.80] [1]

2-2. mysql version: mysql-5.7.17-linux-glibc2.5-x86_64.tar.gz

[Mysql-5.7.17-x86_64] [2]

[All versions of MySQL] [3]

2-3. Cloudera Manager Version: cloudera-manager-el6-cm5.8.0_x86_64.tar.gz

[Cloudera-manager-el6-cm5.8.0_x86_64.tar.gz] [4]

[All versions of CLOUDERA-MANAGER5] [5]

2-4. Parecel Package Version:

Cdh-5.8.0-1.cdh5.8.0.p0.42-el6.parcel
Cdh-5.8.0-1.cdh5.8.0.p0.42-el6.parcel.sha1
Manifest.json

[Cdh-5.8.0-1.cdh5.8.0.p0.42-el6.parcel] [6]

Note: Because the Linux operating version is centos6.5 x64, the version of the Cloudera Manager and Parecel package selected is the corresponding EL6 version.

If your Linux system version is Centos7, Ubuntu, and other versions, download the corresponding Cloudera manager and parecel version to install;

[Cloudera official website about CM5.8.0 installation instructions] [7]

----------

Ii. cm5.8.0 configuration of pre-installation environment

1. Modify the cluster hostname hostname

# vim/etc/sysconfig/network
: hostname=enc-bigdata05

Restart Server Shutdown-r Now, is the configuration file network is in effect

The rest of the host changes hostname, ENC-BIGDATA0[1-6]

----------

2. Modify the mapping of the system IP and hostname:/etc/hosts

2-1. Modify the mapping relationship

# vim/etc/hosts

2-2. Send the file hosts replication to each node in the cluster

# for a in {1..6}; Do scp/etc/hosts enc-bigdata0$a:/etc/hosts; Done

Note: Delete localhost's local mapping, or place it on the last side of the file

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
:: 1 localhost localhost.localdomain localhost6 localhost6.localdomain6

2-3. Check if the hosts for each node in the cluster have been modified.

# for a in {1..6}; do ssh enc-bigdata0$a cat/etc/hosts; Done

----------
3. Set SSH password-free login to get through each node of the cluster

3-1. Create a public and private key

# ssh-keygen-t Rsa-p '

Note: Run the above command on each node of the cluster to create the current user's public and private keys;

Each user can create their own SSH public key and private key, through each node, you can use other users (non-root users) to install CDH5.8.0;

3-2. Id_rsa.pub the public key of each node into the authentication file Authorized_keys;

# for a in {1..6}; Do ssh enc-bigdata0$a cat/root/.ssh/id_rsa.pub >>/root/.ssh/authorized_keys; Done

3-3. Send the Authorized_keys copy of the authentication file to each node:/root/.ssh/authorized_keys

# for a in {1..6}; Do Scp/root/.ssh/authorized_keys Enc-bigdata0$a:/root/.ssh/authorized_keys; Done

----------

4. Installing jdk1.7.0_80

4-1. Unzip the JDK to the specified path

# TAR-ZXVF Jdk-7u80-linux-x64.tar.gz-c/usr/local/

4-2. Send the extracted jdk1.7.0_80 directory to the specified directory of each node

# for a in {1..6}; Scp-r/usr/local/jdk1.7.0_80/enc-bigdata0$a:/usr/local/; Done

4-3. Modify the configuration file/etc/profile

# Vim/etc/profile

#install jdk-7u80-linux-x64
Export java_home=/usr/local/jdk1.7.0_80
Export java_bin= $JAVA _home/bin
Export java_lib= $JAVA _home/lib
Export classpath=.: $JAVA _lib/tools.jar: $JAVA _lib/dt.jar
Export path= $JAVA _bin: $PATH

4-4. Send configuration file replication to each node

# for a in {1..6}; Scp-r/etc/profile Enc-bigdata0$a:/etc/profile; Done

# Source/etc/profile

# for a in {1..6}; Do SSH enc-bigdata0$a/usr/local/jdk1.7.0_80/bin/java-version; Done

5. Modifying system parameters

5-1. Turn off the firewall
# service Iptables Status/stop

# for a in {1..6}; do SSH enc-bigdata0$a/etc/init.d/iptables status; Done

# for a in {1..6}; do ssh enc-bigdata0$a/etc/init.d/iptables stop; Done

# for a in {1..6}; Do ssh enc-bigdata0$a chkconfig iptables off; Done

# for a in {1..6}; Do ssh enc-bigdata0$a chkconfig iptables--list; Done

5-2. Turn off SELinux

# Setenforce 0 (temporary entry into force)

# Vim/etc/selinux/config (Permanent entry)

: selinux=disabled

# for a in {1..6}; Scp/etc/selinux/config enc-bigdata0$a:/etc/selinux/config; Done

5-3. Modify Swappiness

# echo "Vm.swappiness=0" >>/etc/sysctl.conf (permanent)

# echo 0 >/proc/sys/vm/swappiness (Temporary effect)

# for a in {1..6}; Do ssh enc-bigdata0$a echo "vm.swappiness=0" >>/etc/sysctl.conf; Done

5-4. Edit/etc/security/limits.conf

# vim/etc/security/limits.conf

* Soft Nofile 65535

* Hard Nofile 65535

# for a in {1..6}; scp/etc/security/limits.conf enc-bigdata0$a:/etc/security/limits.conf; Done

5-5. Modify the/etc/rc.local (command to run the boot)

#vim/etc/rc.local (Permanent entry)

: Echo Never >/sys/kernel/mm/transparent_hugepage/defrag

# for a in {1..6}; Do ssh enc-bigdata0$a echo never >/sys/kernel/mm/transparent_hugepage/defrag; Done (Temporary effect)

Third, installation mysql5.7.17

1. Check if MySQL has been installed before, remove the previously installed MySQL

# Rpm-qa | grep MySQL

Mysql-libs-5.1.71-1.el6.x86_64

# rpm-e mysql-libs-5.1.71-1.el6.x86_64--nodeps

# Find/-name MySQL

# RM-RF ...

Note: Force uninstall is not uninstalled,--nodeps--force

2. Add user, specify home directory, specify login level

# Usedel MySQL

# Groupdel MySQL

# Mkdir/usr/local/mysql

# useradd-d/usr/local/mysql/-s/sbin/nologin-m MySQL

# Chown-r Mysql:mysql/usr/local/mysql

# chmod 755/usr/local/mysql

3. Unzip to the development path:/usr/local/mysql

# TAR-ZXVF Mysql-5.7.17-linux-glibc2.5-x86_64.tar.gz-c/tmp/

# mv/tmp/mysql-5.7.17-linux-glibc2.5-x86_64/*/usr/local/mysql/

Note: Copy the files from the unpacked installation package to MySQL

# Mkdir/usr/local/mysql/data

Description: Create a data store path

# Chown-r mysql:mysql/usr/local/mysql/

4. Installation and initialization

#/usr/local/mysql/bin/mysql_install_db--user=mysql--basedir=/usr/local/mysql/--datadir=/usr/local/mysql/data/

# cp-a/USR/LOCAL/MYSQL/SUPPORT-FILES/MY-DEFAULT.CNF/ETC/MY.CNF

# cp-a/usr/local/mysql/support-files/mysql.server/etc/init.d/mysqld

#/usr/local/mysql/bin/mysqld_safe--user=mysql &

5. Start Mysqld

#/ETC/INIT.D/MYSQLD Status

#/etc/init.d/mysqld Restart

# Chkconfig Mysqld on

# chkconfig Mysqld--list

Add Shortcut: Ln-s/usr/local/mysql/bin/mysql/usr/bin/

6. Initial Password
mysql5.7 will generate an initial password in the root user's home directory.

# Cat/root/.mysql_secret

Password set for user ' [email protected] ' at 2016-06-01 15:23:25 fhcmc,k50x/y

# mysql-uroot-p (input initialization password)

Change Password
mysql> SET PASSWORD = PASSWORD (' 123qwe ');

mysql> flush Privileges;

7. Add Remote Access

First Method: Change table

mysql> use MySQL;

mysql> Update user Set host = '% ' where user = ' root ';

The second method: authorization

Mysql>grant all privileges on * * to ' root ' @ '% ';

Mysql>flush Privileges

8. Problem: Forget to initialize password

# Service Mysqld Stop

# VIM/ETC/MY.CNF Add skip-grant-tables at the end

# service Mysqld Start

# mysql-uroot-p (to avoid password login)

mysql> set password for ' root ' @ ' localhost ' =password (' new password ');

Mysql>flush privileges;

9. Configure/ETC/MY.CNF

---------------------------------------------------------------------

[MySQL]

# Set the MySQL client default character set

Default-character-set=utf8

[Mysqld]

#设置3306端口

Port = 3306

# set up the MySQL installation directory

Basedir=/usr/local/mysql

# Set up a storage directory for MySQL database data

Datadir=/usr/local/mysql/data

Tmpdir=/usr/local/mysql/tmp

# Maximum number of connections allowed

max_connections=2000

#服务端使用的字符集默认为8比特编码的latin1字符集

Character-set-server=utf8

# The default storage engine that will be used when creating a new table

Default-storage-engine=innodb

#设置不区分大小写 (0: Case sensitive; 1: Case insensitive)

Lower_case_table_names=1

#安装模式: Update mysql.user set Authentication_string=password (' 123qwe ') where user= ' root ';

#skip-grant-tables #忘记初始化密码

--------------------------------------------------------------------

10. Create a Database

Mysql>create database Amon default charset UTF8 collate utf8_general_ci; #activity Monitor

Mysql>create database hive default charset UTF8 collate utf8_general_ci; #hive

Mysql>create database Oozie default charset UTF8 collate utf8_general_ci; #oozie

Mysql>create database hue default charset UTF8 collate utf8_general_ci; #hue

11. Create a new user (you can also use the root user)

Mysql>create user ' admin ' @ '% ' identified by ' [email protected] ';

Mysql>grant all privileges on * * to ' admin ' @ '% ';

Mysql>flush privileges;

----------

Iv. installation of Cloudera Manager

1. Unzip the cloudera-manager-el6-cm5.8.0_x86_64.tar.gz to the specified path:/opt

# TAR-ZXVF Cloudera-manager-el6-cm5.8.0_x86_64.tar.gz-c/opt/

# for a in {1..6}; Do scp-r/opt/cm-5.8.0/enc-bigdata0$a:/opt/;d one

2. Modify the configuration file

# Vim/opt/cm-5.8.0/etc/cloudera-scm-agent/config.ini

Server_host=enc-bigdata05

# for a in {1..6}; Do scp-r/opt/cm-5.8.0/etc/cloudera-scm-agent/config.ini enc-bigdata0$a:/opt/cm-5.8.0/etc/cloudera-scm-agent/ Config.ini; Done

3. Create CDH built-in database SCM

# MV Mysql-connector-java-5.1.40-bin.jar/opt/cm-5.7.0/share/cmf/lib

#/opt/cm-5.7.1/share/cmf/schemascm_prepare_database.sh Mysql-hlocalhost-uroot–[email protected]--scm-host LocalHost SCM SCM SCM

scm_prepare_database.sh database Type server IP username password-scm-host [cloudera Manager Server server] SCM SCM;

4. Copy the Parecel package to the specified directory in Cdh-server and modify the Cdh-5.8.0-1.cdh5.8.0.p0.42-el6.parcel.sha1 file name

# CP cdh-5.8.0-1.cdh5.8.0.p0.42-el6.parcel/opt/cloudera/parcel-repo/

# CP cdh-5.8.0-1.cdh5.8.0.p0.42-el6.parcel.sha1/opt/cloudera/parcel-repo/

# CP manifest.json/opt/cloudera/parcel-repo/

# mv/opt/cloudera/parcel-repo/cdh-5.8.0-1.cdh5.8.0.p0.42-el6.parcel.sha1/opt/cloudera/parcel-repo/ Cdh-5.8.0-1.cdh5.8.0.p0.42-el6.parcel.sha

4. Start Cloudera-scm-server

#/opt/cm-5.8.0/etc/init.d/cloudera-scm-server Start/status/stop

5. Start Cloudera-scm-agent

# for a in {1..6}; Do ssh enc-bigdata0$a/opt/cm-5.8.0/etc/init.d/cloudera-scm-agent start; Done

6. Problem: Cloudera-scm-agent failed to start: Unable to create the Pidfile

Reason: Unable to create/opt/cm-5.8.0/run/cloudera-scm-agent

Workaround:

# mkdir/opt/cm-5.8.0/run/cloudera-scm-agent

# Chown-r Cloudera-scm:cloudera-scm/opt/cm-5.8.0/run/cloudera-scm-agent

7. Access URL: http://IP:7180/(configuration CDH5.8.0)

enc-bigdata0[1-6].enc.cn # #点击模式

Note: It is important to modify the JDK home directory to prevent the cluster host from detecting the JDK;

Note: To ensure that all checks are successful to prevent subsequent errors

Note: If the metabase is installed in MySQL, it is necessary to copy the corresponding database driver to Lib in Hive home directory;

# MV Mysql-connector-java-5.1.40-bin.jar/opt/cloudera/parcels/cdh-5.6.0-1.cdh5.6.0.p0.45/lib/hive/lib

Copy the corresponding database driver to the corresponding service home directory to prevent the database connection failure;

V. Questions

1. About Namenode memory problems, namenode memory if not enough, will cause the Namenode service process to die

Problem description: Can ' t open/opt/cm-5.7.0/run/cloudera-scm-agent/process/1033-hdfs-namenode/supervisord.conf, Permission deny

Workaround: Increase Memory

2. Unable to detect JDK presence

Workaround: Set the JDK home directory

3. Oozie Web page cannot be displayed

Workaround: Upload ext-2.2.zip to the specified path:/var/lib/oozie

Vi. Summary

* * This chapter is for reference only, the specific please take the official document as the prevail, thanks!! **

Data acquisition + Dispatch: Cdh5.8.0+mysql5.7.17+hadoop+sqoop+hbase+oozie+hue