Hortonworks-based large data cluster environment deployment pipelining

First, Ambari and HDP installation files:

1. System

Operating system: CentOS7

2. Software

This installation uses the latest version:

ambari-2.7.0.0

hdp-3.0.0.0

Detailed information is as follows:

Ambari 2.7.0 Repositories

Base url:http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.0.0 Repo File:http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.0.0/ambari.repo Tarball MD5 | Asc:http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.0.0/ambari-2.7.0.0-centos7.tar.gz

HDP 3.0 Repositories

Base url:http://public-repo-1.hortonworks.com/hdp/centos7/3.x/updates/3.0.0.0 Repo File:http://public-repo-1.hortonworks.com/hdp/centos7/3.x/updates/3.0.0.0/hdp.repo Tarball MD5 | Asc:http://public-repo-1.hortonworks.com/hdp/centos7/3.x/updates/3.0.0.0/hdp-3.0.0.0-centos7-rpm.tar.gz Hdp-utils Base Url:http://public-repo-1.hortonworks.com/hdp-utils-1.1.0.22/repos/centos7 Tarball MD5 | Asc:http://public-repo-1.hortonworks.com/hdp-utils-1.1.0.22/repos/centos7/hdp-utils-1.1.0.22-centos7.tar.gz HDP-GPL Url:http://public-repo-1.hortonworks.com/hdp-gpl/centos7/3.x/updates/3.0.0.0/hdp.gpl.repo Tarball MD5 | Asc:http://public-repo-1.hortonworks.com/hdp-gpl/centos7/3.x/updates/3.0.0.0/hdp-gpl-3.0.0.0-centos7-gpl.tar.gz

Second, configure the server

1, install the necessary dependence:

Yum-y install net-tools vim NTP bzip2 gcc readline-devel zlib-devel httpd wget

2. Configure the/etc/hosts file for each server: Add the following 3 lines to it

192.168.0.105 chavin.king Chavin 192.168.0.107 chavin02.king Chavin02 192.168.0.108 chavin03.king chavin03 192.168.0.109 chavin01.king Chavin01

3. All servers are created installation users: Ambari

Groupadd Ambari Useradd-g Ambari Ambari echo "Ambari" | passwd--stdin Ambari

4. All servers configure sudo permissions for Cloudera users

chmod u+w/etc/sudoers echo "Ambari all= (Root) nopasswd:all" >>/etc/sudoers chmod u-w/etc/sudoers

5. All servers shut down the firewall, disable SELinux

Sed-i '/selinux=enforcing/d '/etc/selinux/config Sed-i '/selinux=disabled/d '/etc/selinux/config echo "selinux=disabled" >>/etc/selinux/config

Systemctl Stop Firewalld Systemctl Disable FIREWALLD

6. The number of open files and the maximum number of processes for all server settings

Cp/etc/security/limits.conf/etc/security/limits.conf.bak echo "* Soft Nproc 32000" >>/etc/security/limits.conf echo "* Hard Nproc 32000" >>/etc/security/limits.conf echo "* Soft nofile 65535" >>/etc/security/limits.conf echo "* Hard nofile 65535" >>/etc/security/limits.conf

7. Configure SSH Keyless entry: At least Namenode and ResourceManager node configuration

SSH-KEYGEN-T RSA Ssh-copy-id chavin.king Ssh-copy-id chavin01.king Ssh-copy-id chavin02.king Ssh-copy-id chavin03.king

8. Configure the cluster time Synchronization service: Configure with NTP service

NTP Master node configuration:

Cp/etc/ntp.conf/etc/ntp.conf.bak Cp/etc/sysconfig/ntpd/etc/sysconfig/ntpd.bak echo "Restrict 192.168.0.0 mask 255.255.255.0 nomodify notrap" >>/etc/ntp.conf echo "Sync_hwclock=yes" >>/etc/sysconfig/ntpd Systemctl Restart NTPD

NTP Client configuration:

#crontab-E Enter the following line to save the exit: 0-59/1 * * * */opt/scripts/sync_time.sh The above script reads as follows: # cat/opt/scripts/sync_time.sh /sbin/service ntpd Stop /usr/sbin/ntpdate Chavin /sbin/service ntpd Start

9. Install Java software on all nodes and configure environment variables

Export java_home=/usr/java/jdk1.8.0_181 Export path= $JAVA _home/bin: $JAVA _home/jre/bin: $PATH Export classpath=.: $JAVA _home/lib: $JAVA _home/jre/lib: $CLASSPATH

10. Installing PostgreSQL Database

Reference Document: Https://www.cnblogs.com/wcwen1990/p/6655438.html

Official website Download: postgresql-9.5.5.tar.bz2

Installation:

TAR-JXVF postgresql-9.5.5.tar.bz2 CD postgresql-9.5.5 ./configure--prefix=/pg955 Make World Make Install-world Userdel Postgres Groupadd Postgres Useradd-g Postgres Postgres Chown-r postgres:postgres/pg955/ Su-postgres

To configure environment variables:

Vim. Bash_profile Export pghome=/pg955 Export Pgdata=/pg955/data Export ld_library_path= $PGHOME/lib: $LD _library_path Path= $PGHOME/bin: $PATH

Initialize the database:

Pg_ctl Init

To modify a configuration file:

Vim postgresql.conf

listen_addresses = ' * '

To modify an access control file:

Vim pg_hba.conf

Host All 192.168.0.0/24 MD5

Start the database:

Pg_ctl start-l $PGHOME/logfile 2>&1 >/dev/null

To modify the Postgres database password:

Psql

postgres=# \password

To create a Ambari database:

Create user Ambari with password ' Ambari ';

Create DATABASE Ambari Owner=ambari encoding ' UTF8 ' template template0;

Grant all privileges on the database Ambari to Ambari;

Create schema Ambari;

11. Install Apache server:

Service httpd Start

Cd/var/www/htmp

mkdir Ambari

mkdir HDP three, installation Ambari

1, upload files and unzip the corresponding files:

Upload the Ambari and HDP installation files to the master server, unzip the Ambari files into the/var/www/htmp/ambari/directory, unzip the HDP and Hdp-untils files into the/var/www/htmp/hdp/directory.

TAR-ZXVF ambari-2.7.0.0-centos7.tar.gz-c/var/www/html/ambari/ TAR-ZXVF hdp-3.0.0.0-centos7-rpm.tar.gz-c/var/www/html/hdp/ TAR-ZXVF hdp-utils-1.1.0.22-centos7.tar.gz-c/var/www/html/hdp/

2. Configuring the Yum source file

wget http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/ 2.7.0.0/ambari.repo. Vim Ambari.repo #VERSION_NUMBER =2.7.0.0-897 [ambari-2.7.0.0] #json. url = Http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json Name=ambari version-ambari-2.7.0.0 baseurl=http://chavin.king/ambari/ambari/centos7/2.7.0.0-897 Gpgcheck=1 Gpgkey=http://chavin.king/ambari/ambari/centos7/2.7.0.0-897/rpm-gpg-key/rpm-gpg-key-jenkins Enabled=1 Priority=1

Wget Http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.0.0.0/hdp.repo. Vim Hdp.repo #VERSION_NUMBER =3.0.0.0-1634 [HDP-3.0.0.0] NAME=HDP version-hdp-3.0.0.0 baseurl=http://chavin.king/hdp/hdp/centos7/3.0.0.0-1634 Gpgcheck=1 Gpgkey=http://chavin.king/hdp/hdp/centos7/3.0.0.0-1634/rpm-gpg-key/rpm-gpg-key-jenkins Enabled=1 Priority=1 [hdp-utils-1.1.0.22] Name=hdp-utils version-hdp-utils-1.1.0.22 baseurl=http://chavin.king/hdp/hdp-utils/centos7/1.1.0.22 Gpgcheck=1 Gpgkey=http://chavin.king/hdp/hdp-utils/centos7/1.1.0.22/rpm-gpg-key/rpm-gpg-key-jenkins Enabled=1 Priority=1

3, Installation Ambari-server

Yum-y Install Ambari-server

4, Configuration Ambari-server

[[email protected] ~]# ambari-server Setup Using Python/usr/bin/python Setup Ambari-server Checking SELinux ... SELinux status is ' disabled ' Customize user account for Ambari-server daemon [y/n] (n)? Y Enter user account for Ambari-server daemon (root): Ambari Adjusting ambari-server permissions and ownership ... Checking Firewall status ... Checking JDK ... [1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8 [2] Custom JDK ==================================================================== Enter Choice (1): 2 WARNING:JDK must is installed on all hosts and Java_home must is valid on all hosts. Warning:jce Policy files is required for configuring Kerberos security. If you plan to use kerberos,please make sure JCE Unlimited strength Jurisdiction Policy Files is valid on the all hosts. Path to Java_home:/usr/java/jdk1.8.0_181 Validating JDK on Ambari server...done. Check JDK version for Ambari Server ... JDK version Found:8 Minimum JDK version is 8 for Ambari. Skipping to setup different JDK for Ambari Server. Checking GPL Software Agreement ... GPL License for lzo:https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)? Y Completing Setup ... Configuring Database ... Enter Advanced Database configuration [y/n] (n)? Y Configuring Database ... ==================================================================== Choose one of the following options: [1]-PostgreSQL (Embedded) [2]-Oracle [3]-mysql/mariadb [4]-PostgreSQL [5]-Microsoft SQL Server (Tech Preview) [6]-SQL Anywhere [7]-BDB ==================================================================== Enter Choice (1): 4 Hostname (localhost): chavin.king Port (5432): Database name (Ambari): Postgres schema (Ambari): Username (Ambari): Enter Database Password (bigdata): Re-enter password: Configuring Ambari Database ... Configuring remote database Connection Properties ... Warning:before starting Ambari Server, you must run the following DDL against the database to create the schema:/var/lib /ambari-server/resources/ambari-ddl-postgres-create.sql Proceed with configuring remote database connection properties [y/n] (y)? Extracting system views ... Ambari-admin-2.7.0.0.897.jar .... Ambari repo file contains latest JSON URL Http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json, updating stacks Repoinfos with it ... Adjusting ambari-server permissions and ownership ... Ambari Server ' Setup ' completed successfully.

5. Execute database Script

Psql--host=chavin.king--port=5432--username=ambari-f/var/lib/ambari-server/resources/ Ambari-ddl-postgres-create.sql

6. Start Ambari-server

[[email protected] ~]# ambari-server start Using Python/usr/bin/python Starting Ambari-server Ambari Server running with administrator privileges. Organizing resource Files At/var/lib/ambari-server/resources ... Ambari database consistency Check started ... Server PID at:/var/run/ambari-server/ambari-server.pid Server out at:/var/log/ambari-server/ambari-server.out Server Log at:/var/log/ambari-server/ambari-server.log Waiting for server start ...... ..... ..... ................... Server started listening on 8080 DB configs consistency check found warnings. See/var/log/ambari-server/ambari-server-check-database.log for more details. Ambari Server ' start ' completed successfully.

Through the http://chavin.king:8080 address login, you can see the following login interface, the initial user name and password are admin.

Iv. Creating a HDP cluster

1, landing interface, empty:

2. Enter the cluster name Chavin_cluster, then click Next:

3. Configure the local Yum Source:

4, specify the cluster machine, enter the ID_RSA key:

5. Wait

Wait

This uses NTP for time synchronization, ignoring.

6. Next Step:

7, according to the planning configuration

8, according to the planning configuration

9. Set the password for each account

10, configuration data, first to configure the PostgreSQL driver: Ambari-server setup--jdbc-db=postgres--jdbc-driver=/mnt/ Postgresql-42.2.5.jar, and then depending on the database connection address configuration, you can also select the MySQL database

11, according to the planning configuration:

12, according to the planning configuration:

13, the rules of planning configuration:

14. Installation Summary

15, wait for the installation to complete, the time is longer:

16, installation completed, due to insufficient resources, some of the formation of warning, ignored.

17, to this point, HDP installation completed.

Hortonworks-based large data cluster environment deployment pipelining