Use Ambari on RHEL6.5 to quickly build a Hadoop Cluster

Use Ambari on RHEL6.5 to quickly build a Hadoop Cluster

Use Ambari on RHEL6.5 to quickly build a Hadoop Cluster

Ip hostname
10.7.1.1 master. coke
10.7.1.2 dn1.coke
10.7.1.3 dn2.coke
10.7.1.4 dn3.coke
10.7.1.5 dn4.coke

1. ssh

Master. coke generates pub key as the central control machine, and adds it to each machine including/root/. ssh/authorized_keys of master. coke

2, hostname

Change the hostname and/etc/sysconfig/network HOSTNAME = XXXX for each machine in the hadoop cluster against the table

3, ntp

Enable the ntp service to ensure clock synchronization between different machines

4. repo

Https://cwiki.apache.org/confluence/display/AMBARI/Install+Ambari+1.7.0+from+Public+Repositories

Cd/etc/yum. repos. d/

Wget http://public-repo-1.hortonworks.com/ambari/CentOS6/1.x/updates/1.7.0/ambari.repo

Yum install ambari-server

Yum install mysql-server mysql

Create a database and grant permissions in advance

Mysql> grant all on ambari. * to 'ambari' @ 'localhost' identified by "175an.com ";

Mysql> flush privileges;
 

[@ Master. coke yum. repos. d] # yum install ambari-server
Loaded plugins: product-id, security, sub‑manager
This system is not registered to Red Hat sub‑management. You can use sub‑manager to register.
6ASU5-updates | 1.1 kB: 00
HighAvailability | 3.9 kB: 00
LoadBalancer | 3.9 kB: 00
ResilientStorage | 3.9 kB: 00
ScalableFileSystem | 3.9 kB: 00
Server | 3.9 kB
Updates-ambari-1.7.0 | 2.9 kB 00: 00
Ambari-1.x | 1.3 kB 00: 00
Epel | 1.1 kB: 00
Setting up Install Process
Resolving Dependencies
-> Running transaction check
-> Package ambari-server.noarch. 7.0-169 will be installed
-> Processing Dependency: postgresql-server> = 8.1 for package: ambari-server-1.7.0-169.noarch
-> Running transaction check
-> Package postgresql-server.x86_64. 4.20-1. el6_5 will be installed
-> Processing Dependency: postgresql (x86-64) = 8.4.20-1. el6_5 for package: postgresql-server-8.4.20-1.el6_5.x86_64
-> Processing Dependency: postgresql-libs (x86-64) = 8.4.20-1. el6_5 for package: postgresql-server-8.4.20-1.el6_5.x86_64
-> Processing Dependency: libpq. so.5 () (64bit) for package: postgresql-server-8.4.20-1.el6_5.x86_64
-> Running transaction check
-> Package postgresql. x86_64. 4.20-1. el6_5 will be installed
-> Package postgresql-libs.x86_64. 4.20-1. el6_5 will be installed
-> Finished Dependency ResolutionDependencies Resolved

========================================================== ========================================================== ========================================================== ========================================================== ========================
Package Arch Version Repository Size
========================================================== ========================================================== ========================================================== ========================================================== ========================
Installing:
Ambari-server noarch 1.7.0-169 Updates-ambari-1.7.0 96 M
Installing for dependencies:
Postgresql x86_64 8.4.20-1. el6_5 6ASU5-updates 2.6 M
Postgresql-libs x86_64 8.4.20-1. el6_5 6asu5-updateles 201 k
Postgresql-server x86_64 8.4.20-1. el6_5 6ASU5-updates 3.4 M

Transaction Summary
========================================================== ========================================================== ========================================================== ========================================================== ========================
Install 4 Package (s)

Total download size: 102 M
Installed size: 152 M
Is this OK [y/N]: y
Downloading Packages:
(1/4): ambari-server-1.7.0-169.noarch.rpm | 96 MB
(2/4): postgresql-8.4.20-1.el6_5.x86_64.rpm | 2.6 MB
(3/4): postgresql-libs-8.4.20-1.el6_5.x86_64.rpm | 201 kB
(4/4): postgresql-server-8.4.20-1.el6_5.x86_64.rpm | 3.4 MB
-----------------------------------------------------------
Total 6.7 MB/s | 102 MB
Warning: rpmts_HdrFromFdno: Header V4 RSA/SHA1 Signature, key ID 07513cad: NOKEY
Retrieving key from http://public-repo-1.hortonworks.com/ambari/centos6/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
Importing GPG key 0x07513CAD:
Userid: "Jenkins (HDP Builds) <jenkin@hortonworks.com>"
From: http://public-repo-1.hortonworks.com/ambari/centos6/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
Is this OK [y/N]: y
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Installing: postgresql-libs-8.4.20-1.el6_5.x86_64 1/4
Installing: postgresql-8.4.20-1.el6_5.x86_64 2/4
Installing: postgresql-server-8.4.20-1.el6_5.x86_64 3/4
Installing: ambari-server-1.7.0-169.noarch 4/4
Verifying: postgresql-server-8.4.20-1.el6_5.x86_64 1/4
Verifying: postgresql-libs-8.4.20-1.el6_5.x86_64 2/4
Verifying: postgresql-8.4.20-1.el6_5.x86_64 3/4
Verifying: ambari-server-1.7.0-169.noarch 4/4

Installed:
Ambari-server.noarch. 7.0-169

Dependency Installed:
Postgresql. x86_64. 4.20-1. el6_5 postgresql-libs.x86_64. 4.20-1. el6_5 postgresql-server.x86_64. 4.20-1. el6_5

Complete!
 

Ambari-server setup

 

Using python/usr/bin/python2.6
Setup ambari-server
Checking SELinux...
SELinux status is 'Disabled'
Customize user account for ambari-server daemon [y/n] (n )? Y
Enter user account for ambari-server daemon (root ):
Adjusting ambari-server permissions and ownership...
Checking firewall...
Checking JDK...
[1]-Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7
[2]-Oracle JDK 1.6 + Java Cryptography Extension (JCE) Policy Files 6
[3]-Custom JDK
========================================================== ==============================================
Enter choice (1 ):
To download the Oracle JDK and the Java Cryptography Extension (JCE) policy Files you must accept the license terms found at http://www.oracle.com/technetwork/java/javase/terms/license/index.htmland not accepting will cancel the Ambari Server setup and you must install the JDK and JCE files manually.
Do you accept the Oracle Binary Code License Agreement [y/n] (y )? Y
Downloading JDK from http://public-repo-1.hortonworks.com/ARTIFACTS/jdk-7u67-linux-x64.tar.gz to/var/lib/ambari-server/resources/jdk-7u67-linux-x64.tar.gz
JDK distribution size is 142376665 bytes
Jdk-7u67-linux-x64.tar.gz... 100% (135.8 Mbit/s of 135.8 MB)
Successfully downloaded JDK distribution to/var/lib/ambari-server/resources/jdk-7u67-linux-x64.tar.gz
Installing JDK to/usr/jdk64
Successfully installed JDK to/usr/jdk64/jdk1.7.0 _ 67
Downloading JCE Policy archive from http://public-repo-1.hortonworks.com/ARTIFACTS/UnlimitedJCEPolicyJDK7.zip to/var/lib/ambari-server/resources/UnlimitedJCEPolicyJDK7.zip
Successfully downloaded JCE Policy archive to/var/lib/ambari-server/resources/UnlimitedJCEPolicyJDK7.zip
Completing setup...
Using ing database...
Enter advanced database configuration [y/n] (n )? Y
========================================================== ==============================================
Choose one of the following options:
[1]-PostgreSQL (Embedded)
[2]-Oracle
[3]-MySQL
[4]-PostgreSQL
========================================================== ==============================================
Enter choice (1): 3
Hostname (localhost ):
Port (1, 3306 ):
Database Name (ambari ):
Username (ambari ):
Enter Database Password (bigdata ):
WARNING: Before starting Ambari Server, you must copy the MySQL JDBC driver JAR file to/usr/share/java.
Press <enter> to continue.
ERROR: Before starting Ambari Server, you must copy the MySQL JDBC driver JAR file to/usr/share/java.
ERROR: Exiting with exit code-1.
REASON: Before starting Ambari Server, you must copy the MySQL JDBC driver JAR file to/usr/share/java.
 

Yum install-y mysql-connector-java # resolve the error

Come back

 

[@ Master. coke 6 Server] # ambari-server setup

Using python/usr/bin/python2.6

Setup ambari-server

Checking SELinux...

SELinux status is 'Disabled'

Ambari-server daemon is configured to run under user 'root'. Change this setting [y/n] (n )?

Adjusting ambari-server permissions and ownership...

Checking firewall...

Checking JDK...

Do you want to change Oracle JDK [y/n] (n )?

Completing setup...

Using ing database...

Enter advanced database configuration [y/n] (n )? Y

========================================================== ==============================================

Choose one of the following options:

[1]-PostgreSQL (Embedded)

[2]-Oracle

[3]-MySQL

[4]-PostgreSQL

========================================================== ==============================================

Enter choice (1): 3

Hostname (localhost ):

Port (1, 3306 ):

Database Name (ambari ):

Username (ambari): amb

Enter Database Password (bigdata ):

Re-enter password:

Copying JDBC drivers to server resources...

Refreshing ing remote database connection properties...

WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema:/var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE. SQL

Proceed with processing ing remote database connection properties [y/n] (y )?

Copying JDBC drivers to server resources...

Extracting system views...

...

Adjusting ambari-server permissions and ownership...

Ambari Server 'setup' completed successfully.
 

Import Database

 

Mysql ambari-u175an-p </var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE. SQL
 

Grant all on ambari. * to "amb @ localhost" identified by "amb123 ″;

 

 

Admin Name: admin

Cluster Name: coke

Total Hosts: 6 (6 new)

Repositories:

RedHat5 (HDP-2.2 ):
Http://public-repo-1.hortonworks.com/HDP/centos5/2.x/GA/2.2.0.0

Redhat5 (HDP-UTILS-1.1.0.20 ):
Http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/repos/centos5

Redhat6 (HDP-2.2 ):
Http://public-repo-1.hortonworks.com/HDP/centos6/2.x/GA/2.2.0.0

Redhat6 (HDP-UTILS-1.1.0.20 ):
Http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/repos/centos6

SUSE11 (HDP-2.2 ):
Http://public-repo-1.hortonworks.com/HDP/suse11sp3/2.x/GA/2.2.0.0

Suse11 (HDP-UTILS-1.1.0.20 ):
Http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/repos/suse11sp3

Ubuntu12 (HDP-2.2 ):
Http://public-repo-1.hortonworks.com/HDP/ubuntu12/2.x/GA/2.2.0.0

Ubuntu12 (HDP-UTILS-1.1.0.20 ):
Http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/repos/ubuntu12

 

Services:
HDFS DataNode: 6 hosts
NameNode: master. coke
SNameNode: dn1.coke

YARN + MapReduce2 App Timeline Server: master. coke
NodeManager: 6 hosts
ResourceManager: master. coke

Tez Clients: 6 hosts

Nagios Server: master. coke
Administrator: nagiosadmin/(qilinli@sohu-inc.com)

Ganglia Server: master. coke

Hive Metastore: master. coke
HiveServer2: master. coke
WebHCat Server: master. coke
Database: MySQL (Existing MySQL Database)

HBase Master: dn1.coke
RegionServer: 6 hosts

Pig Clients: 6 hosts

Sqoop Clients: 6 hosts

Oozie Server: dn1.coke
Database: MySQL (Existing MySQL Database)

ZooKeeper Server: 3 hosts

Falcon Server: dn3.coke

Storm DRPC Server: dn1.coke
Nimbus: dn1.coke
UI Server: dn1.coke
Supervisor: 6 hosts

Flume: 2 hosts

Slider Clients: 6 hosts

Knox Gateway: dn1.coke

Kafka Broker: 3 hosts

Install and configure Hadoop2.2.0 on CentOS

Build a Hadoop environment on Ubuntu 13.04

Cluster configuration for Ubuntu 12.10 + Hadoop 1.2.1

Build a Hadoop environment on Ubuntu (standalone mode + pseudo Distribution Mode)

Configuration of Hadoop environment in Ubuntu

Detailed tutorial on creating a Hadoop environment for standalone Edition

Build a Hadoop environment (using virtual machines to build two Ubuntu systems in a Winodws environment)