Turn Cloudera Manager and CDH 4 Ultimate Installation

System environment

Operating system: CentOS 6.5

Cloudera Manager Version: 4.8.1

CDH Version: 4.5.0

Each machine preparation work:

Yum-y Groupinstall "Development tools" Yum-y install wget

Cloudera-manager Zip package Address: http://archive.cloudera.com/cm4/cm/4/cloudera-manager-el6-cm4.8.1_x86_64.tar.gz

cdh:http://archive.cloudera.com/cdh4/parcels/

impala:http://archive.cloudera.com/impala/parcels/

Cloudera Search (SOLR): http://archive.cloudera.com/search/parcels/latest/

Official Installation Reference

http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM4Ent/latest/ Cloudera-manager-installation-guide/cmig_install_path_c.html

There are three ways to install this website:

　　

We chose the third method.

The reason is that the first method must require that all machines be connected, and that foreign sites are not very stable. Once failed, reloading is very painful.

The second method has not been studied carefully and it seems to be downloading many packages.

The third method has the least intrusive system, the biggest advantage can be fully offline installation, and re-install what is very convenient. The later cluster Unified package upgrade is also very good.

Preparatory work

All of the following operations are done as root.

0 modifying host Names

Modify the/etc/sysconfig/network file

Networking=yes HOSTNAME=YC01

Effective after reboot: Service network restart

Modify/etc/hosts file, similar to

127.0.0.1YC01 (Cloudera-manager-server)

       192.168.1.191 yc01 192.168.1.192 yc02 192.168.1.193 yc03 192.168.1.194 yc04 192.168.1.195 yc05

Note that the hosts on the Cloudera-manager-server host put 127.0.0.1 hadoop001 in front of the/opt/cloudera/parcels because the agent downloads the parcels package from the server and extracts it to Under the/opt/cloudera/parcels directory

1 Install salt (can be ignored)

Slightly.

2 Getting through SSH

All machines, use ssh-keygen-t RSA all the way to press ENTER on the line.

What have you done just now? The main is to generate SSH key and key storage path. Under the ~/.ssh.

Open ~/.ssh There are at least two files

Id_rsa, private key

Id_rsa.pub, Public key

On the main node:

Cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

SCP files to other machines:

SCP ~/.ssh/authorized_keys [Email protected]:~/.ssh/

Now log on to other machines without a password.

3 Installing Java

Because CDH4 support JAVA7, consider CDH5 only support JAVA7, decisive on. (later MySQL also used the latest 5.6.16, later found that the tragedy, do not know which reason, so the JDK has changed to the official recommendation version, or not, and the MySQL back to the 5.1.X version, the final can be.) Personal guess JDK can still use 7, MySQL can only use 5.5, and then go up on it. Reference http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/ cdh4-requirements-and-supported-versions/cdhrsv_topic_2.html)

Download and unzip the JDK from the Oracle website. Tar zxvf jdk-7u51-linux-x64.gz under/opt/java.

It seems to have seen before, CDH need a/usr/java/default of the soft chain, will be recognized. Just in case, build one, at least it's easier to manage:

Mkdir-p/usr/java; Cd/usr/java, Ln-s/opt/java/jdk1.7.0_51 latest; Ln-s Latest Default

Finally Java Add environment variable

Export Java_home=/usr/java/defaultexport path= $JAVA _home/bin: $PATHexport classpath=.: $JAVA _home/lib/dt.jar: $JAVA _ Home/lib/tools.jar

Make configuration effective

Source/etc/profile

4 Installing the MySQL database (The installation process has been skipped directly, directly yum-y install Mysql-server to build the required database)

Cloudera Manager's own database of needs, it now supports MySQL, PostgreSQL and Oracle, I have not used PostgreSQL, Oracle selectmen, decisive on MySQL.

Installing and configuring a MySQL Database

Official online said a lot, summed up three words:

1) Install a Mysql

http://dev.mysql.com/downloads/mysql/

Download the client and server

RPM-IVH mysql-server-5.6.16-1.el6.x86_64.rpm

RPM-IVH mysql-client-5.6.16-1.el6.x86_64.rpm

The installation server is a random code that will prompt for root generation.

　　　　　　　

SET PASSWORD = PASSWORD (' ycmedia_2014 ');

The default MySQL does not generate a/etc/my.cnf file, cp/usr/my.cnf/etc/my.cnf

2) Configure some parameters of MySQL itself, and CDH does not have any relationship. Emphasis is placed on the value of the max_connections. The recommendation is the number of databases *100+50. Large clusters (more than 50 units) are recommended to be divided into open.

Procedure: VI/ETC/MY.CNF, modify which parameters, refer to the above link. This step can be skipped directly for general purposes only.

3) Build database, authorize. What libraries are needed, free version: Amon, Hmon, Hive, Smon, SCM (Cloudera Manager itself, I named myself)

The names of these libraries are actually customizable. The other is the name of the official website example, that's it. Remain consistent. Only the last library was named after me.

Create DATABASE hive DEFAULT CHARSET UTF8 COLLATE utf8_general_ci;

Create database Amon DEFAULT CHARSET UTF8 COLLATE utf8_general_ci;

Create database Smon DEFAULT CHARSET UTF8 COLLATE utf8_general_ci;

Create database Hmon DEFAULT CHARSET UTF8 COLLATE utf8_general_ci;

Authorization to the user, similar to the following:

Grant all on * * to [e-mail protected] "%" identified by "123456";

5 Shutting down firewalls and SELinux

To turn off the firewall:

Service iptables Stop (temporary shutdown)

Chkconfig iptables off (effective after reboot)

To turn off SELinux:

Setenforce 0 (Temporary entry into force)

Modify selinux=disabled under/etc/selinux/config (permanent after reboot)

Officially started

Attached with Yum installation:

/etc/yum.repos.d/cloudera-manager.repo Configuring CDH4 sources

[Java]View Plaincopy

1. [Cloudera-manager]
2. Name = Cloudera Manager, Version 4.8. 3
3. BaseURL = http://archive.cloudera.com/cm4/redhat/6/x86_64/cm/4.8.3/
4. Gpgkey = http://archive.cloudera.com/redhat/cdh/rpm-gpg-key-cloudera
5. Gpgcheck = 1

Cloudera-manager-server use: Yum install cloudera-manager-daemons Cloudera-manager-server

Cloudera-manager-agent use: Yum install cloudera-manager-daemons cloudera-manager-agent

You can also download and install it.

1 Installing Cloudera Manager Server and Agents

Unpack the Tarball:

Tar xzf/opt/package/cloudera-manager*.tar.gz-c/opt/; mv/opt/cm-4.8.1/opt/cloudera-manager-4.8.1

Add CLOUDERA-SCM User

Useradd--system--home=/opt/cloudera-manager-4.8.1/run/cloudera-scm-server--no-create-home--shell=/bin/false-- Comment "Cloudera SCM User" CLOUDERA-SCM

For convenience, the following is called/opt/cloudera-manager-4.8.1 <tarball root>

Modify <tarball Root>/etc/cloudera-scm-agent/config.ini below the Server_port and server_host, put Server_port Change to Clouderamanager run hostname or IP, default server_port port is 7182

Cloudera Manager will save some stuff in the directory below.

　　

2 Setting up a database for Cloudera Manager

　　

Download the JDBC driver before you set up the database and put it under <tarball root>/share/cmf/lib/.

Build Database

<tarball root>/share/cmf/schema/scm_prepare_database.sh MySQL SCM  -hlocalhost-uroot-p123456  -- Scm-host localhost SCM SCM SCM

The format is: scm_prepare_database.sh database type database server user name password--scm-host cloudera_manager_server machine ....

The back of the three do not know what the representative, directly copied the official website. Detailed Description:

　　

3 Open Cloudera Manager Server side

On the machine you installed above, open the server side:

　　　　

<tarball root>/etc/init.d/cloudera-scm-server Start

Set to boot:

CP <tarball Root>/etc/init.d/cloudera-scm-server/etc/init.d/cloudera-scm-server Chkconfig Cloudera-scm-server on

4 Open Cloudera Manager Agents side

First SCP <tarball root> to all machines, and then on each machine turn on the agents side:

sudo <tarball root>/etc/init.d/cloudera-scm-agent start

Set boot up:

CP <tarball root>/etc/init.d/cloudera-scm-agent/etc/init.d/cloudera-scm-agent chkconfig cloudera-scm-agent on

5 Starting the Cloudera Manager management Console

The address is similar to http://example.com:7180/, the default port is 7180, if the above is not changed.

User name password is admin

At this point: the Cloudera Manager has finished installing.

------------------------------------------------------------------------------------Split line is not gorgeous------------------------------ ----------------------------------------------------

----------------------------------------------------------------------------------need a gorgeous split line here------------------------------ --------------------------------------------------

6 Prepare parcels for installation of CDH

Put the previously downloaded CDH and IMPALA into/opt/cloudera/parcel-repo, and then generate the Sha file,

The Sha file references the Http://archive.cloudera.com/cdh4/parcels/latest/manifest.json inside, and each file is corresponding. The hash value can be taken out, the simplest of course is directly calculated on the OK.

Put the Manifest.json in there, too. Content Reference website

　　

　　

For more information see: Using Parcels

7 Start installing CDH4 from the interface

After landing from the interface, select version

　　

It can be noted that the previous version of CDH has a maximum of 50 node limits and is now unlimited. Go on.

　　

Go on.

Special attention is needed here, we have n servers, now only one, why? Because I didn't modify server_host=localhost when I started Cloudera Manager Agents.

Vi./etc/cloudera-scm-agent/config.ini changed to the machine name of Cloudera Manager server. Restart the agent (under <tarball root>./etc/init.d/cloudera-scm-agent restart)

Now all the machines are there. Click Continue.

We choose Parcels Way, continue. Will install CDH Impala SOLR. It should be done in a few minutes. Depends on your network transfer speed.

Go on.

Select the service to install, pay special attention to the red box "Check role Assignment", too inconspicuous!!! Once did not notice, a bit depressed. Click Assign Roles. After the allocation is good, continue, of course, later can be changed.

Test the connection must be successful to continue, this place, it is possible to appear in the local root + password can be logged on the MySQL, the test will fail here. Mainly because the user of root is more complicated.

Because your root account may be essentially like this, this place I do not understand MySQL deeply, the effect should be local use although root, that is essentially not the same user.

Regardless of 3,721, all authorized.

Grant all on * * to [e-mail protected] "YC01" identified by "123456"; Grant all on * * to [e-mail protected] "localhost" identified by "123456"; Grant all on * * to [e-mail protected] "127.0.0.1" identified by "123456";

Also here is a note, a considerable note. For example, for a monitor, when we assigned a role in front of a certain machine, then the database is preferably on the same machine.

Click Continue.

Modify it to fit your own configuration. Do the following two actions before continuing. Because the next hive installation will fail, place the JDBC package in the Lib directory of hive in advance. Such as

SCP Mysql-connector-java-5.1.28.jar [Email protected]:/opt/cloudera/parcels/cdh-4.5.0-1.cdh4.5.0.p0.30/lib/hive/ lib/

Hue may also fail with the following error:

Do not know what is the package, so the first direct this:

Yum-y Install libxslt* LIBXML2

Wait patiently for all startup successes.

Congratulations, the installation is successful, the next step is to start your Hadoop journey!

Reprint Please specify source: http://www.cnblogs.com/thinkCoding/p/3567408.html

Turn Cloudera Manager and CDH 4 Ultimate Installation