System environment
Operating system: CentOS 6.5
Cloudera Manager Version: 4.8.1
CDH Version: 4.5.0
Each machine preparation work:
Yum-y Groupinstall "Development tools" Yum-y install wget
Cloudera-manager Zip package Address: http://archive.cloudera.com/cm4/cm/4/cloudera-manager-el6-cm4.8.1_x86_64.tar.gz
cdh:http://archive.cloudera.com/cdh4/parcels/
impala:http://archive.cloudera.com/impala/parcels/
Cloudera Search (SOLR): http://archive.cloudera.com/search/parcels/latest/
Official Installation Reference
http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM4Ent/latest/ Cloudera-manager-installation-guide/cmig_install_path_c.html
There are three ways to install this website:
We chose the third method.
The reason is that the first method must require that all machines be connected, and that foreign sites are not very stable. Once failed, reloading is very painful.
The second method has not been studied carefully and it seems to be downloading many packages.
The third method has the least intrusive system, the biggest advantage can be fully offline installation, and re-install what is very convenient. The later cluster Unified package upgrade is also very good.
Preparatory work
All of the following operations are done as root.
0 modifying host Names
Modify the/etc/sysconfig/network file
Networking=yes HOSTNAME=YC01
Effective after reboot: Service network restart
Modify/etc/hosts file, similar to
127.0.0.1YC01 (Cloudera-manager-server)
192.168.1.191 yc01 192.168.1.192 yc02 192.168.1.193 yc03 192.168.1.194 yc04 192.168.1.195 yc05
Note that the hosts on the Cloudera-manager-server host put 127.0.0.1 hadoop001 in front of the/opt/cloudera/parcels because the agent downloads the parcels package from the server and extracts it to Under the/opt/cloudera/parcels directory
1 Install salt (can be ignored)
Slightly.
2 Getting through SSH
All machines, use ssh-keygen-t RSA all the way to press ENTER on the line.
What have you done just now? The main is to generate SSH key and key storage path. Under the ~/.ssh.
Open ~/.ssh There are at least two files
Id_rsa, private key
Id_rsa.pub, Public key
On the main node:
Cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
SCP files to other machines:
SCP ~/.ssh/authorized_keys [Email protected]:~/.ssh/
Now log on to other machines without a password.
3 Installing Java
Because CDH4 support JAVA7, consider CDH5 only support JAVA7, decisive on. (later MySQL also used the latest 5.6.16, later found that the tragedy, do not know which reason, so the JDK has changed to the official recommendation version, or not, and the MySQL back to the 5.1.X version, the final can be.) Personal guess JDK can still use 7, MySQL can only use 5.5, and then go up on it. Reference http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/ cdh4-requirements-and-supported-versions/cdhrsv_topic_2.html)
Download and unzip the JDK from the Oracle website. Tar zxvf jdk-7u51-linux-x64.gz under/opt/java.
It seems to have seen before, CDH need a/usr/java/default of the soft chain, will be recognized. Just in case, build one, at least it's easier to manage:
Mkdir-p/usr/java; Cd/usr/java, Ln-s/opt/java/jdk1.7.0_51 latest; Ln-s Latest Default
Finally Java Add environment variable
Export Java_home=/usr/java/defaultexport path= $JAVA _home/bin: $PATHexport classpath=.: $JAVA _home/lib/dt.jar: $JAVA _ Home/lib/tools.jar
Make configuration effective
Source/etc/profile
4 Installing the MySQL database (The installation process has been skipped directly, directly yum-y install Mysql-server to build the required database)
Cloudera Manager's own database of needs, it now supports MySQL, PostgreSQL and Oracle, I have not used PostgreSQL, Oracle selectmen, decisive on MySQL.
Installing and configuring a MySQL Database
Official online said a lot, summed up three words:
1) Install a Mysql
http://dev.mysql.com/downloads/mysql/
Download the client and server
RPM-IVH mysql-server-5.6.16-1.el6.x86_64.rpm
RPM-IVH mysql-client-5.6.16-1.el6.x86_64.rpm
The installation server is a random code that will prompt for root generation.
SET PASSWORD = PASSWORD (' ycmedia_2014 ');
The default MySQL does not generate a/etc/my.cnf file, cp/usr/my.cnf/etc/my.cnf
2) Configure some parameters of MySQL itself, and CDH does not have any relationship. Emphasis is placed on the value of the max_connections. The recommendation is the number of databases *100+50. Large clusters (more than 50 units) are recommended to be divided into open.
Procedure: VI/ETC/MY.CNF, modify which parameters, refer to the above link. This step can be skipped directly for general purposes only.
3) Build database, authorize. What libraries are needed, free version: Amon, Hmon, Hive, Smon, SCM (Cloudera Manager itself, I named myself)
The names of these libraries are actually customizable. The other is the name of the official website example, that's it. Remain consistent. Only the last library was named after me.
Create DATABASE hive DEFAULT CHARSET UTF8 COLLATE utf8_general_ci;
Create database Amon DEFAULT CHARSET UTF8 COLLATE utf8_general_ci;
Create database Smon DEFAULT CHARSET UTF8 COLLATE utf8_general_ci;
Create database Hmon DEFAULT CHARSET UTF8 COLLATE utf8_general_ci;
Authorization to the user, similar to the following:
Grant all on * * to [e-mail protected] "%" identified by "123456";
5 Shutting down firewalls and SELinux
To turn off the firewall:
Service iptables Stop (temporary shutdown)
Chkconfig iptables off (effective after reboot)
To turn off SELinux:
Setenforce 0 (Temporary entry into force)
Modify selinux=disabled under/etc/selinux/config (permanent after reboot)
Officially started
Attached with Yum installation:
/etc/yum.repos.d/cloudera-manager.repo Configuring CDH4 sources
[Java]View Plaincopy
- [Cloudera-manager]
- Name = Cloudera Manager, Version 4.8. 3
- BaseURL = http://archive.cloudera.com/cm4/redhat/6/x86_64/cm/4.8.3/
- Gpgkey = http://archive.cloudera.com/redhat/cdh/rpm-gpg-key-cloudera
- Gpgcheck = 1
Cloudera-manager-server use: Yum install cloudera-manager-daemons Cloudera-manager-server
Cloudera-manager-agent use: Yum install cloudera-manager-daemons cloudera-manager-agent
You can also download and install it.
1 Installing Cloudera Manager Server and Agents
Unpack the Tarball:
Tar xzf/opt/package/cloudera-manager*.tar.gz-c/opt/; mv/opt/cm-4.8.1/opt/cloudera-manager-4.8.1
Add CLOUDERA-SCM User
Useradd--system--home=/opt/cloudera-manager-4.8.1/run/cloudera-scm-server--no-create-home--shell=/bin/false-- Comment "Cloudera SCM User" CLOUDERA-SCM
For convenience, the following is called/opt/cloudera-manager-4.8.1 <tarball root>
Modify <tarball Root>/etc/cloudera-scm-agent/config.ini below the Server_port and server_host, put Server_port Change to Clouderamanager run hostname or IP, default server_port port is 7182
Cloudera Manager will save some stuff in the directory below.
2 Setting up a database for Cloudera Manager
Download the JDBC driver before you set up the database and put it under <tarball root>/share/cmf/lib/.
Build Database
<tarball root>/share/cmf/schema/scm_prepare_database.sh MySQL SCM -hlocalhost-uroot-p123456 -- Scm-host localhost SCM SCM SCM
The format is: scm_prepare_database.sh database type database server user name password--scm-host cloudera_manager_server machine ....
The back of the three do not know what the representative, directly copied the official website. Detailed Description:
3 Open Cloudera Manager Server side
On the machine you installed above, open the server side:
<tarball root>/etc/init.d/cloudera-scm-server Start
Set to boot:
CP <tarball Root>/etc/init.d/cloudera-scm-server/etc/init.d/cloudera-scm-server Chkconfig Cloudera-scm-server on
4 Open Cloudera Manager Agents side
First SCP <tarball root> to all machines, and then on each machine turn on the agents side:
sudo <tarball root>/etc/init.d/cloudera-scm-agent start
Set boot up:
CP <tarball root>/etc/init.d/cloudera-scm-agent/etc/init.d/cloudera-scm-agent chkconfig cloudera-scm-agent on
5 Starting the Cloudera Manager management Console
The address is similar to http://example.com:7180/, the default port is 7180, if the above is not changed.
User name password is admin
At this point: the Cloudera Manager has finished installing.
------------------------------------------------------------------------------------Split line is not gorgeous------------------------------ ----------------------------------------------------
----------------------------------------------------------------------------------need a gorgeous split line here------------------------------ --------------------------------------------------
6 Prepare parcels for installation of CDH
Put the previously downloaded CDH and IMPALA into/opt/cloudera/parcel-repo, and then generate the Sha file,
The Sha file references the Http://archive.cloudera.com/cdh4/parcels/latest/manifest.json inside, and each file is corresponding. The hash value can be taken out, the simplest of course is directly calculated on the OK.
Put the Manifest.json in there, too. Content Reference website
For more information see: Using Parcels
7 Start installing CDH4 from the interface
After landing from the interface, select version
It can be noted that the previous version of CDH has a maximum of 50 node limits and is now unlimited. Go on.
Go on.
Special attention is needed here, we have n servers, now only one, why? Because I didn't modify server_host=localhost when I started Cloudera Manager Agents.
Vi./etc/cloudera-scm-agent/config.ini changed to the machine name of Cloudera Manager server. Restart the agent (under <tarball root>./etc/init.d/cloudera-scm-agent restart)
Now all the machines are there. Click Continue.
We choose Parcels Way, continue. Will install CDH Impala SOLR. It should be done in a few minutes. Depends on your network transfer speed.
Go on.
Select the service to install, pay special attention to the red box "Check role Assignment", too inconspicuous!!! Once did not notice, a bit depressed. Click Assign Roles. After the allocation is good, continue, of course, later can be changed.
Test the connection must be successful to continue, this place, it is possible to appear in the local root + password can be logged on the MySQL, the test will fail here. Mainly because the user of root is more complicated.
Because your root account may be essentially like this, this place I do not understand MySQL deeply, the effect should be local use although root, that is essentially not the same user.
Regardless of 3,721, all authorized.
Grant all on * * to [e-mail protected] "YC01" identified by "123456"; Grant all on * * to [e-mail protected] "localhost" identified by "123456"; Grant all on * * to [e-mail protected] "127.0.0.1" identified by "123456";
Also here is a note, a considerable note. For example, for a monitor, when we assigned a role in front of a certain machine, then the database is preferably on the same machine.
Click Continue.
Modify it to fit your own configuration. Do the following two actions before continuing. Because the next hive installation will fail, place the JDBC package in the Lib directory of hive in advance. Such as
SCP Mysql-connector-java-5.1.28.jar [Email protected]:/opt/cloudera/parcels/cdh-4.5.0-1.cdh4.5.0.p0.30/lib/hive/ lib/
Hue may also fail with the following error:
Do not know what is the package, so the first direct this:
Yum-y Install libxslt* LIBXML2
Wait patiently for all startup successes.
Congratulations, the installation is successful, the next step is to start your Hadoop journey!
Reprint Please specify source: http://www.cnblogs.com/thinkCoding/p/3567408.html
Turn Cloudera Manager and CDH 4 Ultimate Installation