CDH5.7.2 offline deployment notes

CDH5.7.2 offline deployment notes
1, deployment Basic Information Description: 1.1, host information operating system: Choose CentOS6.7 x86-64 version MEM: 64 GB, CPU: E5-2630 v3 @ 2.40 GHz, DISK: 2 TB * 4 (data node storage, excluding the operating system) host information: Host Name IPdcnn1 10.20.1dcnn2 10.20.00002dcdn01 10.20.000000003dcdn02 255.10.0000000051.2, CDH version information ClouderaManager: cm5.7.2CDH: cdh5.7.2 2. Deployment Step 2.1 Download the installation package CM parcel offline installation package: 5.7.2 http://archive.cloudera.com/cm5/cm/5/cloudera-manager-centos7-cm5.7.2_x86_64.tar.gz CDH parcel offline installation package: (5.7.2) http://archive.cloudera.com/cdh5/parcels/5.7/CDH-5.7.2-1.cdh5.7.2.p0.18-el6.parcel http://archive.cloudera.com/cdh5/parcels/5.7/CDH-5.7.2-1.cdh5.7.2.p0.18-el6.parcel . Sha1 http://archive.cloudera.com/cdh5/parcels/5.7/manifest.json2.2 Account allocation creates and installs the user bigdata. The user configures sudo without the Password Logon permission, which is followed by the bigdata user operation, you can also directly use the root user to operate the host deployed on the management platform to connect to the ssh Login-free (deployment user, bigdata or root) of other hosts. Here, you can set the host name using the bigdata 2.3 Host Name: add all host information to/etc/hosts for each host, and run the hostname command to change the host name 10.20.20.1 dcnn110.20.20.2 dcnn210.20.00003 dcdn0110.000000004 dcdn0210.000000005 dcdn03 2.4 network configuration (each host) use the dcnn1 bits as an example: vim/etc/sysconfig/networkNETWORKING = yesHOSTNAME = dcnn1service network restartRestart the network service and take effect. 2.5 disable IPtables and SELINUX (each host) sudo chkconfig iptables offsudo/etc/init. d/iptables stop to disable SELINUX (it is possible that SELINUX is not closed during the actual installation process. I don't know if there will be any problem. Further verification is required): setenforce 0 (temporarily effective) modification: selinux = disabled under/etc/SELINUX/configx (effective permanently after restart) 2.6 enable the NTP service because it is isolated from the Internet, configure your own ntpd server, ntp service is not installed, install ntpd by yourself. Generally, ntpd service is installed by default in linux but not started. If not, you can install and configure ntp server by yourself: 1) Open the ntp service configuration file/etc/ntp. conf2) add content // ignore ntp requests packet restrict default nomodify notrap restric T 10.20.20.1 mask 255.255.255.0 nomodify server 10.20.20.1 // indicates that the NTP master server synchronizes the system clock with its own server 127.127.1.0 // The stratum size of the Upper-layer server cannot exceed or equal to 16 fudge 127.127.1.0 stratum 103) start service: sudo service ntpd start restart effective: sudo chkconfig ntpd on Configure ntp client: 1) Open the ntp service configuration file/etc/ntp. conf2) add content server 10.20.000013) start service: sudo service ntpd start restart takes effect: sudo chkconfig ntpd on note: ntp service configuration takes effect for a period of time, I tested it for about five minutes. I just restarted the ntp service and the check status was not necessarily normal. 2.7 install JDK

 

We recommend that you use version 1.8.0 _ 60 because CDH requires Oracle's java (all nodes) CentOS to bring its own OpenJdk. However, to run CDH5, you need to use Oracle's JDK and support from Java 7 or later. First uninstall the built-in OpenJdk and use rpm -qa | grep javaQuery java-related packages and use Rpm-e -- nodeps package nameUninstall it. Installation: rpm-ivh jdk-8u60-linux-x64.rpm because it is an rpm package does not require us to configure the environment variables, we only need to configure a global JAVA_HOME variable, execute the command: echo "JAVA_HOME =/usr/java/latest/">/etc/environment 2.8 install MySQL (install CentOS6.7 with version 5.1.7) mysql Database Server: sudo rpm-hvi mysql-5.1.73-5.el6_6.x86_64.rpmsudo rpm-hvi perl-DBD-MySQL-4.013-3.el6.x86_64.rpmsudo rpm-hvi mysql-server-5.1.73-5.el6_6.x86_64.rpm chkconfig mysqld onSet boot start, and service mysqld startStart the mysql service and set the password for the root user as prompted:       mysqladmin -u root password 'bigdata' mysqladmin -u root -h dcnn1 password 'bigdata'Mysql database client: sudo rpm-hvi mysql-5.1.73-5.el6_6.x86_64.rpm create database and set mysql database access permissions: #hiveCreate database hive default charset utf8 COLLATE utf8_general_ci; UPDATE user SET password = PASSWORD ('hive') WHERE user = 'hive'; # activity monitor create database amon DEFAULT CHARSET utf8 COLLATE utf8_general_ci;UPDATE user SET password = PASSWORD ('amon') WHERE user = 'amon'; # oozie create database oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci;UPDATE user SET password = PASSWORD ('oozie ') WHERE user = 'oozie'; # report create database report DEFAULT CHARSET utf8 COLLATE utf8_general_ci;UPDATE user SET password = PASSWORD ('report') WHERE user = 'report'; flush privileges; # SET root authorization to access all the above databases: grant all privileges on *. * to 'root' @ 'dcnn1 'identified by 'bigdata' with grant option; flush privileges; # Authorize a user to have database access permissions on the serverGrant all privileges on *. * to 'oozie '@ 'dcnn1' identified by 'oozie 'with grant option; grant all privileges on *. * to 'hive' @ 'dcnn1 'identified by 'hive' with grant option; grant all privileges on *. * to 'amon' @ 'dcnn1 'identified by 'amon' with grant option; grant all privileges on *. * to 'report' @ 'dcnn1 'identified by 'report' with grant option; flush privileges; # Authorize the root user to have database access permissions on the clientGrant all privileges on *. * TO 'hive' @ '%' identified by 'hive' with grant option; grant all privileges on *. * TO 'oozie '@' % 'identified by 'oozie' with grant option; grant all privileges on *. * TO 'amon' @ '%' identified by 'amon' with grant option; grant all privileges on *. * TO 'report' @ '%' identified by 'report' with grant option; flush privileges; upload TO the dcnn1 directory/opt (the default directory of Cloudera manager) 2) tar xvfz cloudera-manager * .tar.gz, decompress and generate two directories cloudera and cm-5.7.2 3) all nodes create user cloudera-scmsudo useradd -- system -- home =/opt/cm-5.7.2/run/cloudera-scm-server/-- no-create-home -- shell =/bin/false -- comment "Cloudera SCM User" cloudera-scm 4) create a database for Cloudera Manager 5 first-mover to download the JDBC driver mysql-connector-java-5.1.38.tar.gz from the mysqlofficial website (you can download the latest version ),: http://dev.mysql.com/downloads/connector/j/ , Extract and find the mysql-connector-java-5.1.38-bin.jar put in/opt/cm-5.7.2/share/cmf/lib: cp mysql-connector-java-5.1.38-bin.jar/opt/cm-5.7.2/share/cmf/lib // opt/cm-5.7.2/share/cmf/schema/scm_prepare_database.sh mysql cm-h dcnn1-uroot-pbigdata -- scm-host dcnn1 scm problems with scm: not resolved, but does not affect the subsequent deployment AVA_HOME =/usr/java/jdk1.8.0 _ 60 Verifying that we can write to/opt/cm-5.7.2/etc/cloudera-scm-serverlog4j: ERROR cocould not find value for key log4j. appender. alog4j: ERROR cocould not instantiate appender named "". creating SCM configuration file in/opt/cm-5.7.2/etc/cloudera-scm-serverExecuting: /usr/java/jdk1.8.0 _ 60/bin/java-cp/usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar: /opt/cm-5.7.2/share/cmf/schema /.. /lib/* com. cloudera. enterprise. dbutil. dbCommandExecutor/opt/cm-5.7.2/etc/cloudera-scm-server/db. properties com. cloudera. cmf. db. log4j: ERROR cocould not find value for key log4j. appender. alog4j: ERROR cocould not instantiate appender named "". agent configuration 1) Modify/opt/cm-5.7.2/etc/cloudera-scm-agent/config. server_host in ini, server_host Host Name of the master node 2) synchronize Agent to other nodes sudo scp-r/opt/cm-5.7.2 root@dcnn2:/opt/;Sudo scp-r/opt/cm-5.7.2 root@dcdn01:/opt/;Sudo scp-r/opt/cm-5.7.2 root@dcdn02:/opt/;Sudo scp-r/opt/cm-5.7.2 root@dcdn03:/opt/;2.10 start the service through sudo/opt/cm-5.7.2/etc/init. d/cloudera-scm-server start the server through sudo/opt/cm-5.7.2/etc/init. d/cloudera-scm-agent start the agent end. We started a service script. You need to stop the service and change the preceding start parameter to stop. restart the service by restart. 2.11 deploy CDH1 through cm) log on to http: // 10.20.20.1: 71802). Follow the Wizard to proceed step by step, which is relatively simple. 3. Problems 1. hive and oozie startup failure lack the JDBC driver, because my mysql is installed in the remote end and the JDBC driver is used for access here. So add the mysql-connector-java-5.1.22-bin.jar to the lib of hive and oozie: sudo cp mysql-connector-java-5.1.38-bin.jar/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/hive/libsudo cp mysql-connector-java-5.1.38-bin.jar/opt/cloudera/parcels/CDH-5.7.2-1.cdh5.7.2.p0.18/lib/oozie/libext 2. Install hive in the query failed, the reason is the permission issue org. apache. hadoop. security. accessControlException: Permission denied: user = bigdata, access = WRITE, inode = "/u Ser ": hdfs: supergroup: drwxr-xr-x the simplest solution is to remove the check box for the service scope.