The big data cluster environment ambari supports cluster management and monitoring, and provides hadoop + hbase + zookeepe

Apache Ambari is a Web-based tool that supports the supply, management, and monitoring of Apache Hadoop clusters. Ambari currently supports most Hadoop components, including HDFS, MapReduce, Hive, Pig, Hbase, Zookeper, Sqoop, and Hcatalog.
 
Apache Ambari supports centralized management of HDFS, MapReduce, Hive, Pig, Hbase, Zookeper, Sqoop, and Hcatalog. It is also one of the five top-level hadoop management tools.
 
Ambari has made the following achievements:
 
The step-by-step installation wizard simplifies cluster supply.
You can configure key O & M metrics (metrics) in advance to check whether Hadoop Core (HDFS and MapReduce) and related projects (such as HBase, Hive, and HCatalog) are healthy.
Supports visualization and analysis of jobs and task execution to better view dependencies and performance.
A complete RESTful API is used to expose monitoring information and integrate existing O & M tools.
The user interface is very intuitive. You can easily and effectively view information and control clusters.
Ambari uses Ganglia to collect metrics, and uses Nagios to support system alarms. When an administrator needs to pay attention (such as node downtime or insufficient disk space), the system will send an email to it.
 
In addition, Ambari can install a secure (Kerberos-based) Hadoop cluster to support Hadoop security and provide role-based user authentication, authorization, and audit functions, LDAP and Active Directory are integrated for user management.
 
1. Install centos 6.5
 
Each operating system is configured with three NICs, one net, one host only, and one bridge. Except for net forwarding, IP addresses are dynamically obtained. Other nic ip addresses are statically configured.
 
System version:
CentOS release 6.5 (Final)
CPU:
Core: 4 cores
Cpu MHz: 3192.620
Cache size: 6144 KB
Clflush size: 64
Cache_alignment: 64
Address sizes: 42 bits physical, 48 bits virtual
Memory
MemTotal: 4G
Hard disk
Total capacity 80 GB
 
Node name
IP1
Service
Remarks
Master
192.168.5.12
Ntp, mysql
 
Epele iptables & selinux stop
OpenSSL (v1.01, build 16 or later );
 
Python v2.6;
 
Slave1
192.168.5.13
Ntp,
 
Epele iptables & selinux stop
OpenSSL (v1.01, build 16 or later );
 
Python v2.6;
 
Slave2
192.168.5.14
Ntp,
 
Epele iptables & selinux stop
OpenSSL (v1.01, build 16 or later );
 
Python v2.6;
 
2. ssh key-free login
 
Create User hadoop
 
[Root @ master ~] # Useradd hadoop
[Root @ master ~] # Passwd hadoop
Change password
All machines use the same password.
* During key configuration, you can select the root user to generate the key or the su hadoop account to generate the key. The key must be generated after the host name is configured, the key copy process should be synchronized to avoid errors.
 
The command is as follows:
Ssh-keygen-t rsa: generate an rsa encryption key
 
Cp id_rsa.pub authorized_keys copy file
 
Cat id_rsa.pub> authorized_keys: append other keys to this file and distribute them to all machines.
 
[Hadoop @ master. ssh] $ cat id_rsa.pub
Ssh-rsa Secure/GIgphhLdKFA + secure + rM/secure + yw5vmow.bz9zbxqxck/secure + QUrS08ks + 8BaSi7Kw + 4 NmfDNNd/secure = hadoop@master.com
Ssh-rsa keys/7cV9cJjZ2/keys + keys/042OsQ/FXLvtqVRl4 + keys + poMpl + yVcZt0SD1Yic8Er5/i61Pg1ehCFKQ = hadoop@slave1.com
Ssh-rsa Secure/secure/xOUlCzuwSaS5YMG5eKrn1TDCK + FxcBxk6QpNAWO665XeKN/secure/+ gE2skI9RpJapVDm + TiZU/wOVqQek/secure + Tw = hadoop@slave2.com
Scp: Copy to another host
 
Hadoop@192.168.5.13:. ssh/scp authorized_keys/
 
3. Configure all the hosts files.
 
[Root @ master. ssh] # vi/etc/hosts
127.0.0.1 localhost. localdomain localhost4 localhost4.localdomain4
: 1 localhost. localdomain localhost6 localhost6.localdomain6
192.168.5.12 master
192.168.5.13 slave1
192.168.5.14 slave2
Wq is saved.
 
Distribution:
[Root @ master. ssh] # scp/etc/hosts root @ slave1:/etc/hosts
[Root @ master. ssh] # scp/etc/hosts root @ slave2:/etc/hosts
 
4. Check whether you can log on without a key After configuration is complete.
 
[Hadoop @ master ~] $ Ssh slave1
The authenticity of host 'slave1 (192.168.5.13) 'can't be established.
RSA key fingerprint is 16: c3: fc: d7: fc: db: 98: ba: ef: aa: 54: e4: d9: f5: 51: be.
Are you sure you want to continue connecting (yes/no )? Yes
Warning: Permanently added 'slave1 '(RSA) to the list of known hosts.
[Hadoop @ slave1 ~] $
Servers can log on to each other through ssh.
5. install mysql yum install mysql-server and directly start mysql to create ambari database authorization
 
The read and write permissions of the ambari account to the database must be guaranteed during the authorization process.
[Root @ masterysql-u root-p
 
Mysql> create database ambari;
Query OK, 1 row affected (0.00 sec)
 
Mysql> use ambari;
Database changed
 
Mysql>
Create user 'ambari' @ '%' identified by 'ambari ';
Grant all privileges on *. * TO 'ambari' @ '% ';
Create user 'ambari' @ 'localhost' identified by 'ambari ';
Grant all privileges on *. * TO 'ambari' @ 'localhost ';
Create user 'ambari' @ 'cluster-02 'identified BY 'ambari ';
Grant all privileges on *. * TO 'ambari' @ 'master ';
Flush privileges;
 
6. Install ambari
 
[Root @ master ~] # Yum install wget
Install wget
 
First, you must obtain the public repository file of Ambari ). Log on to the Linux host and run the following command (you can also manually download it yourself ):
Wget requests
Copy the downloaded ambari. repo file to the Linux system directory/etc/yum. repos. d /. After copying, we need to obtain a list of all the source files of the public library. Execute the following commands in sequence.
Yum clean all
Yum list | grep ambari
 
Cleaning up list of fastest mirrors
[Root @ master yum. repos. d] # yum list | grep ambari
Ambari-agent.x86_64 2.0.1-45 Updates-ambari-2.0.1
Ambari-log4j.noarch 2.0.1.45-1 Updates-ambari-2.0.1
Ambari-metrics-collector.x86_64 2.0.1-45 Updates-ambari-2.0.1
Ambari-metrics-common.noarch 2.0.1-45 Updates-ambari-2.0.1
Ambari-metrics-hadoop-sink.x86_64 2.0.1-45 Updates-ambari-2.0.1
Ambari-metrics-monitor.x86_64 2.0.1-45 Updates-ambari-2.0.1
Ambari-server.noarch 2.0.1-45 Updates-ambari-2.0.1
Hdp_mon_ganglia_addons.noarch 2.0.1.45-1. el6 Updates-ambari-2.0.1
Hdp_mon_nagios_addons.noarch 2.0.1.45-1. el6 Updates-ambari-2.0.1
7. Configure ambari
 
After the installation is complete, you need to make a simple configuration for the Ambari Server. Run the following command.
Amari-server setup
[Root @ master ~] # Ambari-server setup
Using python/usr/bin/python2.6
Setup ambari-server
Checking SELinux...
SELinux status is 'enabled'
SELinux mode is 'enable'
Temporarily disabling SELinux disable selinux
WARNING: SELinux is set to 'permission' mode and temporarily disabled.
OK to continue [y/n] (y )? Y
Customize user account for ambari-server daemon [y/n] (n )?
Adjusting ambari-server permissions and ownership...
Checking iptables...
WARNING: iptables is running. Confirm the necessary Ambari ports are accessible. Refer to the Ambari documentation for more details on ports.
OK to continue [y/n] (y )?
Checking JDK...
[1] Oracle JDK 1.7
[2] Oracle JDK 1.6
[3]-Custom JDK: you can add local java environment variables. 3. You can directly add the local java installation path/usr/java/jdk1.7.0 _ 67.
========================================================== ==============================================
Enter choice (1): 1
To download the Oracle JDK and the Java Cryptography Extension (JCE) policy Files you must accept the license terms found at http://www.oracle.com/technetwork/java/javase/terms/license/index.html and not accepting will cancel the Ambari Server setup and you must install the JDK and JCE files manually.
Do you accept the Oracle Binary Code License Agreement [y/n] (y )?
Downloading JDK from http://public-repo-1.hortonworks.com/ARTIFACTS/jdk-7u67-linux-x64.tar.gz to/var/lib/ambari-server/resources/jdk-7u67-linux-x64.tar.gz
Choose one of the following options:
[1]-PostgreSQL (Embedded)
[2]-Oracle
[3]-MySQL
[4]-PostgreSQL
========================================================== ==============================================
Enter choice (1 ):
Input: 3
 
Hostname (localhost ):
Port (1, 3306 ):
Database Name (ambari): ambari
Username (ambari): ambari
Enter Database Password (cluster ):
Re-enter password:
Input: ambari
 
WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema:/var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE. SQL
Proceed with processing ing remote database connection properties [y/n] (y )?
Input: y
After this step is installed, check whether the tables in the ambari database in the mysql database have been created and not manually imported to the ambari database.
To log on to the mysql database, execute the following script Ambari-DDL-MySQL-CREATE. SQL
[Root @ cluster-02 ~] # Mysql-u root-p
 
Mysql> use ambari;
Database changed
 
Mysql> source/var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sq
The script is successfully executed.
 
Check whether the script initialization is successful. The following table list is displayed.
Mysql> show tables;
+ ------------------------------- +
| Tables_in_ambari |
+ ------------------------------- +
| ClusterHostMapping |
| QRTZ_BLOB_TRIGGERS |
| QRTZ_CALENDARS |
| QRTZ_CRON_TRIGGERS |
| QRTZ_FIRED_TRIGGERS |
| QRTZ_JOB_DETAILS |
| QRTZ_LOCKS |
8ambari startup
 
After the simple setup configuration is complete. Then you can start Ambari. Run the following command.
Ambari-server start
[Root @ hadoop1 ~] # Ambari-server start
Using python/usr/bin/python2.6
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at/var/lib/ambari-server/resources...
Server PID at:/var/run/ambarriers-server/ambari-server.pid
Server out at:/var/log/ambarriers-server/ambari-server.out
Server log at:/var/log/ambari-server/ambari-server.log
Waiting for server start ....................
Ambari Server 'start' completed successfully.
Run tail-f/var/log/ambari-server/ambari-server.log to view server logs
[Root @ master ~] # Tail-f/var/logs/ambari-server/ambari-server.log
29 Dec 2015 13:25:44, 472 INFO [main] Configuration: 514-Generation of file with password
29 Dec 2015 13:25:46, 115 INFO [main] Configuration: 527-Reading password from existing file
29 Dec 2015 13:25:47, 753 INFO [main] Configuration: 527-Reading password from existing file
29 Dec 2015 13:25:48, 195 INFO [main] ViewExtractor: 82-Creating archive folder/var/lib/ambari-server/resources/views/work/ADMIN_VIEW {2.0.0 }.
29 Dec 2015 13:25:48, 196 INFO [main] ViewExtractor: 89-Extracting files from ambari-admin-2.0.1.45.jar.
29 Dec 2015 13:25:49, 386 INFO [main] Configuration: 527-Reading password from existing file
29 Dec 2015 16:32:07, 171 INFO [main] Configuration: 527-Reading password from existing file
29 Dec 2015 16:32:07, 213 INFO [main] Configuration: 747-Hosts Mapping File null
29 Dec 2015 16:32:07, 213 INFO [main] HostsMap: 60-Using hostsmap file null
29 Dec 2015 16:32:08, 470 INFO [main] ControllerModule: 173-Detected versions s as the database type from the JDBC URL
29 Dec 2015 16:32:18, 870 INFO [main] AmbariServer: 655-Getting the controller
29 Dec 2015 16:32:22, 555 INFO [main] StackDirectory: 408-stack'/var/lib/ambari-server/resources/stacks/HDP/2.0.6.GlusterFS 'doesn' t contain an upgrade directory
29 Dec 2015 16:32:22, 613 INFO [main] StackDirectory: 429-Role command order info was loaded from file: /var/lib/ambari-server/resources/stacks/HDP/2.0.6.GlusterFS/role_command_order.json
29 Dec 2015 16:32:22, 703 INFO [main] StackDirectory: 408-stack'/var/lib/ambari-server/resources/stacks/HDP/2.1.GlusterFS 'doesn' t contain an upgrade directory
9. Complete
 
Open Address
Http: // 192.168.1.12: 8080/complete