Use Cloudera Manager to install Hadoop

Use Cloudera Manager to install Hadoop

Hadoop is composed of many different services (such as HDFS, Hive, HBase, Spark, and so on). These services also have some dependencies. If you directly download the original Apache package, it is troublesome to download multiple times and configure multiple times. As a result, some companies have customized Hadoop, such as Cloudera, Hortonworks, and MapR. These companies all have their own Hadoop releases, and the Cloudera releases are called CDH. Because the company's system uses CDH, CDH5 has been installed for many times in this period of time, and some practical operations have been done, and some Cloudera documents have been read.

Cloudera uses Cloudera Manager, a Hadoop management system on the Web interface, to install, configure, and monitor Hadoop-related services.

Install Cloudera Manager

The official guide is provided here in several ways. I am only talking about one of them. It is not the simplest, but it can give you a better understanding of Cloudera Manager.

The following figure shows how Cloudera Manager manages Hadoop services.

Configure the YUM source of Cloudera Manager

Add a file CM. repo under/etc/yum. repos. d/. The content is as follows.

 

[CM]name=CMbaseurl=http://archive-primary.cloudera.com/cm5/RedHat/6/x86_64/cm/5.1.3/gpgcheck=0

[CM] Name = CM Baseurl = http: // archive-primary.cloudera.com/cm5/redhat/6/x86_64/cm/5.1.3/ Gpgcheck = 0

Run the following command to check whether the Yum repository configuration is successful. If the yum list can list the cloudera-manager-*** packages, it indicates that the configuration is successful. If you have installed it in a lab environment and may not be able to connect to the Internet, you can use the reposync command on a machine that can connect to the Internet to clone the Internet yum source to your local machine, then a local http service is set up to serve as a yum server for the lab machine. You can also configure a proxy in the CM. repo file of the lab to connect to the Internet. Just Google the steps.

 

[root@bogon yum.repos.d]# yum clean allLoaded plugins: fastestmirror, refresh-packagekit, securityCleaning repos: CM base extras updatesCleaning up EverythingCleaning up list of fastest mirrors[root@bogon yum.repos.d]# yum list | grep clouderacloudera-manager-daemons.x86_64        5.1.3-1.cm513.p0.155.el6          @CM    cloudera-manager-server.x86_64         5.1.3-1.cm513.p0.155.el6          @CM    cloudera-manager-agent.x86_64          5.1.3-1.cm513.p0.155.el6          CM     cloudera-manager-server-db-2.x86_64    5.1.3-1.cm513.p0.155.el6          CM

[Root @ bogon yum. repos. d] # yum clean all Loaded plugins: fastestmirror, refresh-packagekit, security Cleaning repos: CM base extras updates Cleaning up Everything Cleaning up list of fastest mirrors [Root @ bogon yum. repos. d] # yum list | grep cloudera Cloudera-manager-daemons. x86_64 5.1.3-1. cm513.20.155. el6 @ CM Cloudera-manager-server. x86_64 5.1.3-1. cm513.20.155. el6 @ CM Cloudera-manager-agent. x86_64 5.1.3-1. cm513.20.155. el 6 CM Cloudera-manager-server-db-2. x86_64 5.1.3-1. cm513.20.155. el 6 CM

Install the Cloudera Manager package. When this package is installed, the cloudera-manager-daemons package on which it depends is automatically installed.

 

yum install cloudera-manager-server

Yum install cloudera-manager-server

Check what these two packages are installed. You can see that the server basically copies some configuration files. the real website management is installed through daemons. The specific files are under the/usr/share/cmf directory, and the logs are under the/var/log/cloudera-scm-server Directory, the runtime configuration file is in/var/run/cloudera-scm-server

[root@bogon ~]# rpm -ql cloudera-manager-server/etc/cloudera-scm-server/etc/cloudera-scm-server/db.properties/etc/cloudera-scm-server/log4j.properties/etc/default/cloudera-scm-server/etc/rc.d/init.d/cloudera-scm-server/etc/security/limits.d/cloudera-scm.conf/opt/cloudera/csd/opt/cloudera/parcel-repo/usr/sbin/cmf-server/var/lib/cloudera-scm-server[root@bogon ~]# rpm -ql cloudera-manager-daemons.../usr/share/cmf/yarn-fixtures/details_1.json/usr/share/cmf/yarn-fixtures/details_2.json/usr/share/cmf/yarn-fixtures/details_3.json/usr/share/cmf/yarn-fixtures/details_4.json/usr/share/cmf/yarn-fixtures/details_5.json/usr/share/cmf/yarn-fixtures/details_6.json/usr/share/cmf/yarn-fixtures/details_7.json/usr/share/cmf/yarn-fixtures/details_8.json/usr/share/cmf/yarn-fixtures/details_9.json/var/log/cloudera-scm-server/var/run/cloudera-scm-server

[Root @ bogon ~ ] # Rpm-ql cloudera-manager-server /Etc/cloudera-scm-server /Etc/cloudera-scm-server/db. properties /Etc/cloudera-scm-server/log4j. properties /Etc/default/cloudera-scm-server /Etc/rc. d/init. d/cloudera-scm-server /Etc/security/limits. d/cloudera-scm. conf /Opt/cloudera/csd /Opt/cloudera/parcel-repo /Usr/sbin/cmf-server /Var/lib/cloudera-scm-server [Root @ bogon ~ ] # Rpm-ql cloudera-manager-daemons ... /Usr/share/cmf/yarn-fixtures/details_1. json /Usr/share/cmf/yarn-fixtures/details_2. json /Usr/share/cmf/yarn-fixtures/details_3. json /Usr/share/cmf/yarn-fixtures/details_4. json /Usr/share/cmf/yarn-fixtures/details_5. json /Usr/share/cmf/yarn-fixtures/details_6. json /Usr/share/cmf/yarn-fixtures/details_7. json /Usr/share/cmf/yarn-fixtures/details_8. json /Usr/share/cmf/yarn-fixtures/details_9. json /Var/log/cloudera-scm-server /Var/run/cloudera-scm-server

Install the database and the corresponding driver. This is because Cloudera Manager needs to save the configuration to the database. In addition, when Hive is installed later, the database needs to save the metadata.

 

yum install mysql-serveryum install mysql-connector-java.noarch

Yum install mysql-server Yum install mysql-connector-java. noarch

Then configure some database-related parameters, add the database cmf, the corresponding user cmf, And the password to the database.

 

mysql> create database cmf DEFAULT CHARACTER SET utf8;Query OK, 1 row affected (0.00 sec)mysql> grant all on cmf.* TO 'cmf'@'localhost' IDENTIFIED BY '123456';Query OK, 0 rows affected (0.00 sec)

Mysql> create database cmf default character set utf8; Query OK, 1 row affected (0.00 sec) Mysql> grant all on cmf. * TO 'cmf' @ 'localhost' identified by '123 '; Query OK, 0 rows affected (0.00 sec)

Modify the/etc/cloudera-scm-server/db. properties file and configure the database to which the Cloudera manager accesses, the corresponding user name and password.

 

# The database type# Currently 'mysql', 'postgresql' and 'Oracle' are valid databases.com.cloudera.cmf.db.type=mysql# The database host# If a non standard port is needed, use 'hostname:port'com.cloudera.cmf.db.host=localhost# The database namecom.cloudera.cmf.db.name=cmf# The database user<a href="http://linmingren.me/blog/wp-content/uploads/2014/09/cloudera-welcome.png"></a>com.cloudera.cmf.db.user=cmf# The database user's passwordcom.cloudera.cmf.db.password=123456

# The database type # Currently 'mysql', 'postgresql ', and 'oracle' are valid databases. Com. cloudera. cmf. db. type = mysql # The database host # If a non standard port is needed, use 'hostname: port' Com. cloudera. cmf. db. host = localhost # The database name Com. cloudera. cmf. db. name = cmf # The database user <a href = "http://linmingren.me/blog/wp-content/uploads/2014/09/cloudera-welcome.png"> </a> Com. cloudera. cmf. db. user = cmf # The database user's password Com. cloudera. cmf. db. password = 123456

Now you can start the cloudera-scm-server Service. Run the netstat command to check whether port 7180 has been started (from service start to port enabling on my machine for about half a minute ), if yes, you can enter localhost: 7180 in the browser to access the Cloudera Manger web Interface (the default user password is admin/admin ), if not, see what is an exception in/var/log/cloudera-scm-server/cloudera-scm-server.log.

 

service cloudera-scm-server restart

Service cloudera-scm-server restart

This is the first time you log on to the homepage.