CDH5.7.2 Offline Deployment Notes

1, the deployment of basic information Description: 1.1, host information operating system: Select CentOS6.7 x86-64 version mem:64gb,cpu:e5-2630 v3 @ 2.40GHz, disk:2tb*4 (data node storage, except the operating system) host information: Host name IPdcnn1 10.20.20.1dcnn2 10.20.20.2dcdn01 10.20.20.3dcdn02 10.20.20.4dcdn03 10 .20.20.51.2, CDH version information clouderamanager:cm5.7.2cdh:cdh5.7.2 version 2. Deployment Step 2.1 Download the installation package CM parcel Offline installation package: 5.7.2 http://archive.cloudera.com/cm5/cm/5/cloudera-manager-centos7-cm5.7.2_x86_64. tar.gz CDH Parcel Offline installation package: (5.7.2) Http://archive.cloudera.com/cdh5/parcels/5.7/CDH-5.7.2-1.cdh5.7.2.p0.18-el6.parce L HTTP://ARCHIVE.CLOUDERA.COM/CDH5/PARCELS/5.7/CDH-5.7.2-1.CDH5.7.2.P0.18-EL6.PARCEL.SHA1 Http://archi ve.cloudera.com/cdh5/parcels/5.7/manifest.json2.2 account Assignment Setup user bigdata, User Configuration sudo no password login rights, followed by Bigdata user operation, You can also directly use the root user operation to deploy the management platform of the host to the other host SSH-free login (deployment user, bigdata or root), here with the Bigdata 2.3 hostname settings: Each host/etc/hosts add all the host information, Change the host name with the hostname command 10.20.20.1 dcnn110.20.20.2 dcnn210.20.20.3 dcdn0110.20.20.4 dcdn0210.20.20.5 dcdn03 2.4 Network configuration (per host) Vim/etc/sysconfig/networknetworking=yeshostname=dcnn1 through the dcnn1-bit example service network restartRestart Network Service, effective 2.5 close iptables and SELinux (per host)sudo chkconfig iptables offsudo/etc/init.d/iptables StopTurn off SELinux (the actual installation process found that there is no shutdown is also possible, do not know if there will be problems, further verification): Setenforce 0 (Temporary effect) modified:/ETC/SELINUX/CONFIGX under the selinux= Disabled (permanent after reboot)  2.6 turn on NTP service here because it is isolated from the external network, configure their own NTPD server, NTP service is not installed, please install itself, the general Linux system will be installed by default NTPD service, but did not start, No can self-install configuration NTP server: 1) Open NTP service profile/ETC/NTP.CONF2) Add content      //ignore NTP request packet       Restrict default nomodify notrap     restrict 10.20.20.1 mask 255.255.255.0 nomodify     se RVer 10.20.20.1        //indicates that the NTP master server is synchronizing with its own system clock      server 127.127.1.0  & nbsp  //Sync Upper Server stratum size cannot be greater than or equal to 16     fudge 127.127.1.0 stratum 103) Start        Start service: sudo service ntpd start      restart effective: sudo chkconfig ntpd on  Configure NTP   CLIENT:1) Open NTP service profile/ETC/NTP.CONF2) Add content      server 10.20.20.13) Start       Start service: sudo service ntpd start      restart effective: sudo chkconfig ntpd on  Note: The configuration of NTP services,It takes a while to take effect, I myself test about 5 minutes, just restart the NTP service, check the status is not necessarily normal.  2.7 Installing the JDK

We use the proposed version 1.8.0_60 because CDH requires Oracle's Java (all nodes) CentOS OpenJDK, but running CDH5 requires Oracle's JDK, which requires more than 7 of Java support. First uninstall your own openjdk, use rpm -qa | grep javaQuery Java-related packages, using rpm -e --nodeps 包名Uninstall it. Installation: Rpm-ivh jdk-8u60-linux-x64.rpm since the RPM package does not require us to configure the environment variable, we only need to configure a global java_home variable to execute the command: echo "java_home=/usr/java/ latest/">>/etc/environment 2.8 install MySQL (install CentOS6.7 comes with version 5.1.7) MySQL database server: sudo rpm-hvi mysql-5.1.73-5.el6_6. X86_64.rpmsudo Rpm-hvi Perl-dbd-mysql-4.013-3.el6.x86_64.rpmsudo Rpm-hvi mysql-server-5.1.73-5.el6_6.x86_64.rpm chkconfig mysqld onSet the boot up and service mysqld startStart the MySQL service and follow the prompts to set the root password:       mysqladmin -u root password ‘bigdata‘ mysqladmin -u root -h dcnn1 password ‘bigdata‘MySQL database client: sudo rpm-hvi mysql-5.1.73-5.el6_6.x86_64.rpm CREATE DATABASE and set MySQL database access rights: #hiveCreate DATABASE hive DEFAULT CHARSET UTF8 COLLATE utf8_general_ci; UPDATE user SET Password=password (' hive ') WHERE user= ' hive '; #activity monitor create database amon DEFAULT CHARSET utf8 COLLATE utf8_general_ci;UPDATE user SET Password=password (' Amon ') WHERE user= ' Amon '; #oozie create database oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci;UPDATE user SET Password=password (' Oozie ') WHERE user= ' Oozie '; #report create database report DEFAULT CHARSET utf8 COLLATE utf8_general_ci;UPDATE user SET Password=password (' report ') WHERE user= ' report '; FLUSH privileges; #设置root授权访问以上所有的数据库: Grant all privileges on * * to ' root ' @ ' dcnn1 ' identified by ' Bigdata ' with Grant Opti On;flush privileges; #授权用户在服务端拥有数据库的访问权限Grant all privileges on * * to ' oozie ' @ ' dcnn1 ' identified by ' Oozie ' with Grant option;grant all privileges on * * to ' hi ve ' @ ' dcnn1 ' identified by ' hive ' with Grant option;grant-privileges on * * to ' amon ' @ ' dcnn1 ' identified by ' Amon ' wit H Grant option;grant all privileges on * * to ' report ' @ ' dcnn1 ' identified by ' report ' with grant Option;flush privileges; #授权root用户在客户端拥有数据库的访问权限GRANT all privileges on * * to ' hive ' @ '% ' identified by ' hive ' with GRANT OPTION; GRANT all privileges on * * to ' oozie ' @ ' percent ' identified by ' Oozie ' with GRANT OPTION; GRANT all privileges on * * to ' Amon ' @ ' percent ' identified by ' Amon ' with GRANT OPTION; GRANT all privileges on * * to ' report ' @ '% ' identified by ' report ' with GRANT Option;flush privileges; 2.9 Install CMCM and Namenode multiplexing, too few machines 1) upload the downloaded cloudera-manager-centos7-cm5.7.2_x86_64.tar.gz to the DCNN1 directory/opt (this directory is Cloudera Manager's default directory) 2) tar Xvfz cloudera-manager*.tar.gz, unzip to generate two directories Cloudera and cm-5.7.2     3) All nodes create user Cloudera-scmsudo useradd--system--home=/opt/cm-5.7.2/run/cloudera-scm-server/--no-create-home--shell= /bin/false--comment "Cloudera SCM User" cloudera-scm 4) set up a database for Cloudera Manager 5       First you need to go to MySQL's official website to download the JDBC driver mysql-connector-java-5.1.38.tar.gz (You can download the latest version): http://dev.mysql.com/downloads/connector/j /&NBSP, after decompression find Mysql-connector-java-5.1.38-bin.jar put into/opt/cm-5.7.2/share/cmf/lib/:  CP Mysql-connector-java-5.1.38-bin.jar/opt/cm-5.7.2/share/cmf/lib//opt/cm-5.7.2/share/cmf/schema/scm_prepare_ database.sh MySQL cm-h dcnn1-uroot-pbigdata--scm-host dcnn1 SCM SCM scm  encountered a problem: not resolved, but does not affect the subsequent deployment  ava_home=/usr/ja Va/jdk1.8.0_60verifying that we can write To/opt/cm-5.7.2/etc/cloudera-scm-serverlog4j:error Could not find value for key Log4j.appender.Alog4j:ERROR Could not instantiate appender named "A". Creating SCM configuration file in/opt/cm-5.7.2/etc/cloudera-scm-serverexecuting: /usr/java/jdk1.8.0_60/bin/ Java-cp/usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/opt/cm-5.7.2/share/cmf/schema/. /lib/* com.cloudera.enterprise.dbutil.dbcommandexecutor/opt/cm-5.7.2/etc/cloudera-scm-server/db.properties Com.cloudera.cmf.db.log4j:ERROR Could not find value for key Log4j.appender.Alog4j:ERROR Could not instantiate Appender NA Med "A" .   Agent Configuration1) Modify the Server_host in/opt/cm-5.7.2/etc/cloudera-scm-agent/config.ini, server_host the host name of the primary node 2) Synchronizing the agent to other nodessudo scp-r/opt/cm-5.7.2 [email protected]:/opt/;sudo scp-r/opt/cm-5.7.2 [email protected]:/opt/;sudo scp-r/opt/cm-5.7.2 [email protected]:/opt/;sudo scp-r/opt/cm-5.7.2 [email protected]:/opt/;2.10 Start service via Sudo/opt/cm-5.7.2/etc/init.d/cloudera-scm-server start service via sudo/opt/cm-5.7.2/etc/init.d/ Cloudera-scm-agent Start Agent side We start is actually a service script, need to stop services to change the above start parameter to stop on it, restart is restart. 2.11 via CM deployment CDH1) Log in http://10.20.20.1:71802) You can follow the wizard to make it easier. 3 encountered problem 1, Hive, and Oozie failed to boot because the JDBC driver is missing because my MySQL is installed at the far end, here to access it through the JDBC driver. So add Mysql-connector-java-5.1.22-bin.jar to hive and Oozie's Lib:sudo CP Mysql-connector-java-5.1.38-bin.jar/opt/cloudera /parcels/cdh-5.7.2-1.cdh5.7.2.p0.18/lib/hive/libsudo CP Mysql-connector-java-5.1.38-bin.jar/opt/cloudera/parcels /cdh-5.7.2-1.cdh5.7.2.p0.18/lib/oozie/libext 2, after installation, hive fails at query time, The reason is the permissions issue org.apache.hadoop.security.AccessControlException:Permission Denied:user=bigdata, Access=write, inode= "/ User ": Hdfs:supergroup:drwxr-xr-x the simplest way to do this: The service range is checked out.

CDH5.7.2 Offline Deployment Notes