Hadoop cluster installation based on Cloudera Manager 5 and CDH5 (version 5.3.3)

First, Cloudera Manager/cdh5

1, about Cloudera Manager and CDH is what, here do not do detailed introduction. There are official website and encyclopedia introduction.

Attached official website address: Cloudera Manager

2, the official website Installation Guide

Official documentation provides three ways to install: Online automatic installation/manual installation package installation/manual use of Cloudera Manager management installation

The third way to install a Hadoop cluster is here.

Second, environmental planning

1. System: CentOS 6.4_x86

MASTER:4G memory, HDD as large as possible

SLAVE1:2G memory, HDD as large as possible

SLAVE2:2G memory, HDD as large as possible

2. Cloudera Manager 5.3.3

3, CDH 5.3.3

Install package Download:

• Cloudera Manager 5.3.3:http://archive-primary.cloudera.com/cm5/cm/5/

• CDH 5.5.5:http://archive-primary.cloudera.com/cdh5/parcels/5.3.3/

Download the package:

1. Cdh-5.3.3-1.cdh5.3.3.p0.5-el6.parcel

2. Cdh-5.3.3-1.cdh5.3.3.p0.5-el6.parcel.sha1

3. Manifest.json

Third, the system environment preparation work

• This execution permission is the root user

• All machines must have SSH trust mode.

• Modify host name using Hosts file or DNS server

• Close Iptables and SELinux

• Uninstalling the system's own OPENJDK, installing Oracle's JDK

• Master node install MySQL

• Time for all nodes must be synchronized (NTP server or other method)

• Modify kernel parameters for all nodes

  I. Echo 0 >/proc/sys/vm/swappiness

  Ii.echo never >/sys/kernel/mm/redhat_transparent_hugepage/defrag

Iv. official start of installation

1, Cloudera Manager default directory is under/opt, extracted into the/opt directory.

# tar XF cloudera-manager-el6-cm5.3.3_x86_64.tar.gz-c/opt/# ls/opt/cloudera cm-5.3.3## Install mysql-connector-java# yum-y Install Mysql-connector-java

2. Initialize the database

#/opt/cm-5.3.3/share/cmf/schema/scm_prepare_database.sh MySQL cm-hlocalhost-uroot-p123456--scm-host localhost SCM    SCM scm# MYSQL-UROOT-P123456-E "show databases;" To see if the next CM database was created successfully

3. Copying data to other nodes

# Modify the configuration file Server_host host name for Master node # grep Server_host/opt/cm-5.3.3/etc/cloudera-scm-agent/config.ini server_host= master# scp-rp/opt/cm-5.3.3 slave1:/opt/# scp-rp/opt/cm-5.3.3 slave2:/opt/## Create system users per node # useradd--system--home=/opt/ Cm-5.3.3/run/cloudera-scm-server--no-create-home--shell=/bin/false--comment "cloudera SCM User" CLOUDERA-SCM

4, the configuration of the database

# hive# CREATE DATABASE hive default CHARSET UTF8 COLLATE utf8_general_ci;# active monitor# CREATE database Amon default C Harset UTF8 COLLATE utf8_general_ci;# Authorized Master Host # Grant all on * * to ' root ' @ ' master ' identified by ' PASSWD ';

5. Prepare parcels installation package to/opt/cloudera/parcel-repo/

# Ls/opt/cloudera/parcel-repo/cdh-5.3.3-1.cdh5.3.3.p0.5-el6.parcel Cdh-5.3.3-1.cdh5.3.3.p0.5-el6.parcel.sha manifest.json# Note: Cdh-5.3.3-1.cdh5.3.3.p0.5-el6.parcel.sha downloaded file renamed # # # Start the server and Agent script for master node #/OPT/CM-5.3.3/ Etc/init.d/cloudera-scm-agent start#/opt/cm-5.3.3/etc/init.d/cloudera-scm-server start# Server port start slower # # Start agent script for all other nodes #/opt/cm-5.3.3/etc/init.d/cloudera-scm-agent start

Five, CDH 5 installation configuration

1. Access the http://master-ip:7180 port to start the installation

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M02/6C/20/wKiom1VAYYDjBCXpAAC5d7N0yYc844.jpg "title=" c1.png "alt=" Wkiom1vayydjbcxpaac5d7n0yyc844.jpg "/>

Login Name: admin Password: admin

2. Select a version to install, and select the first free version here. Then next--Next

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M01/6C/1E/wKiom1VASp3jAAQLAAIqbasUW1o224.jpg "title=" c2.png "alt=" Wkiom1vasp3jaaqlaaiqbasuw1o224.jpg "/>

3. Select All hosts:

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M00/6C/1A/wKioL1VATHGz8yDaAAEhFEDSrCA045.jpg "title=" c3.png "alt=" Wkiol1vathgz8ydaaaehfedsrca045.jpg "/>

4, if the configuration is correct, the following interface will appear

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M02/6C/1A/wKioL1VATKfiFVprAAFo7cUuH7U250.jpg "title=" c4.png "alt=" Wkiol1vatkfifvpraafo7cuuh7u250.jpg "/>

5. Then proceed to the next step, as we have downloaded the offline package, so the download will be completed soon

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M01/6C/1F/wKiom1VAS3rgM30HAADIuBznOYc167.jpg "title=" c5.png "alt=" Wkiom1vas3rgm30haadiubznoyc167.jpg "/>

6, the host detection, check whether the host to meet the installation requirements, meet the requirements will be all passed, otherwise please set according to the requirements

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M01/6C/1F/wKiom1VATRjBPaitAAFK1VsDb8c464.jpg "title=" C11.png "alt=" Wkiom1vatrjbpaitaafk1vsdb8c464.jpg "/>

7. Then select the package you want to install, you can select it all or customize it, or you can choose a package that integrates the functionality of a component to install

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M00/6C/1F/wKiom1VATXrSyOT8AAIUKmDmoqo471.jpg "title=" C12.png "alt=" Wkiom1vatxrsyot8aaiukmdmoqo471.jpg "/>

8. Role assignment, default here, can be adjusted on demand

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M01/6C/1F/wKiom1VATdqAar6sAAFen2iofhk264.jpg "title=" Cc1.png "alt=" Wkiom1vatdqaar6saafen2iofhk264.jpg "/>

9. Database testing:

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M02/6C/1B/wKioL1VAUJHRbcADAADudcyMESk690.jpg "title=" C123.png "alt=" Wkiol1vaujhrbcadaadudcymesk690.jpg "/>

10, Audit change parameters, you can use the default, or you can change as needed

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M02/6C/1F/wKiom1VAT5jDp-POAAC8bWDI2d4249.jpg "title=" C1123.png "alt=" Wkiom1vat5jdp-poaac8bwdi2d4249.jpg "/>

11. Start the normal installation configuration, wait for the installation to complete and then visit the homepage again.

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M01/6C/1F/wKiom1VAT9HRD-ozAAEJegf_kTc998.jpg "title=" Caaa.png "alt=" Wkiom1vat9hrd-ozaaejegf_ktc998.jpg "/>

12. Landing screen after completion

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M00/6C/1F/wKiom1VAUx2Sp9m-AAJcSSMDXTQ450.jpg "title=" m1.png "alt=" Wkiom1vaux2sp9m-aajcssmdxtq450.jpg "/>

Queries are often not displayed because of the low performance of the host and the large latency of the data. And because there is not enough disk space here, there are a lot of warning messages. To this installation has all been completed.

Vi. Other issues explained

The Oozie Web interface opens:

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M01/6C/20/wKiom1VAWXWTbxzxAADxt9g7Px0785.jpg "title=" m3.png "alt=" Wkiom1vawxwtbxzxaadxt9g7px0785.jpg "/>

The Cloudera documentation has instructions for configuring Oozie:

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M00/6C/1C/wKioL1VAXRCzYHe7AAED9KDDkY8680.jpg "title=" Oozie.png "alt=" Wkiol1vaxrczyhe7aaed9kddky8680.jpg "/>

Install this action:

# mv ext-2.2.zip/var/lib/oozie/# cd/var/lib/oozie# Unzip Ext-2.2.zip

Refresh the interface:

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M00/6C/1C/wKioL1VAXc2iUocbAAEpwnZLhek178.jpg "title=" o1.png "alt=" Wkiol1vaxc2iuocbaaepwnzlhek178.jpg "/>

The hadoop2.x version has updated some new features to support high availability of HDFS and more. Cloudera Manager management interface can be directly operated by this series of functions, or is very convenient.

There is an action options bar in the upper right corner of the HDFs interface in the cluster:

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M00/6C/20/wKiom1VAXWjgE5O4AAGBrGqzUSk844.jpg "title=" M21.png "alt=" Wkiom1vaxwjge5o4aagbrgqzusk844.jpg "/>

After clicking on the prompt will be able to configure the normal. Management interface to add the removal of the host in the cluster is also very convenient, the specific operation will not be demonstrated.

Vii. Hadoop test Procedures

# calculate PI Value # sudo-u hdfs hadoop jar/opt/cloudera/parcels/cdh/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar Pi 10 100...Job finished in 126.439 secondsestimated value of Pi is 3.14800000000000000000# to see execution results

Detailed job information can be found in Yarn's web interface

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M00/6C/1C/wKioL1VAYG3SQCFlAAF-p-mgzsc557.jpg "title=" r1.png "alt=" Wkiol1vayg3sqcflaaf-p-mgzsc557.jpg "/>

There are also many test programs that can run tests. This is not a demonstration.

In addition, this installation is quick and easy, but it is not conducive to the overall understanding, it is recommended to use the way to install the installation package, configuration files, etc. are to be written manually, for the understanding of the more profound.

This article is from the "Soul" blog, make sure to keep this source http://chenpipi.blog.51cto.com/8563610/1640255

Hadoop cluster installation based on Cloudera Manager 5 and CDH5 (version 5.3.3)