Hadoop2.7.3 and Spark2.1.0 cluster detailed setup Tutorials

1. This article describes

This document includes the following Ubuntu virtual machine installation process Ubuntu configuration static IP configuration ssh password-free login ubuntu domestic apt software source configuration jdk,scala,mysql Software Installation Hadoop cluster installation spark cluster installation Anaconda installation

Version of the software used and download address VMware Workstation 12.5.2 ubuntu-14.04.1-server-amd64 Xshell 5 xftp 5 JDK 1.8 Scala 2.12.1 Hadoop 2.7.3 Spa rk-2.1.0-bin-hadoop2.7 anaconda2-4.3.1-linux-x86_64 2. Preparing the virtual machine

The cluster consists of 2 machines, one master, one slaver, so 2 virtual machines need to be installed, the steps are as follows: 2.1 new virtual machine

Open VMware Workstation, click File-New virtual machine

Click Next in succession, choose to Install the operating system later

Next, select the guest operating system , where I chose Ubuntu 64-bit

Next, name the virtual machine

Next, CPU and memory configuration, depending on your machine performance, I chose dual core and 2G memory

Next, network type, using NAT by default

Next, IO controller type, default LSI Logic (L)

Next, disk type, default SCSI (S)

Next, select disk and create a new virtual disk by default

Next, specify the disk capacity, allocate 20GBby default, split the virtual disk into multiple files

Next, specify the disk file, by default

Next, click Finish, New end

2.1 Installing a virtual machine

Next, click Edit Virtual machine settings

Next, choose Download good virtual machine ISO image

Next, you can turn on the virtual machine.

Select system language, default 中文版

Select Install Ubuntu Server

Select language again, default 中文版

Select location,Other->asia->china

Configure the keyboard, not detected by default, select Chinese (US) to

Now it will automatically perform some operations, about half a minute

Configure hostname, where I am configured as Master

Create a new user, the user name I use here or the master

Set the password and confirm the password that I set here is 123456

Because the password is too simple, you will be asked whether to use a weak password, select YES to

Whether to encrypt the home directory, by default select NO to

Wait about half a minute, the system automatically configures some operations

Confirm Time Zone

Select the disk partitioning method, by default, select the unique default partition

Write partition

Next click on continue and confirm to start the real installation system

Default is empty, no proxy is configured

Turn off Automatic Updates

Select the installed software, OpenSSH Server is selected here

Install boot boot

Installation complete, select continue to restart

See the following interface, the installation starts, enter a user name and password to log in. Follow the same steps, then install a virtual machine, Hostnaem set to Slaver, as the slave node, where the installation process is no longer repeated.

After logging into the system, configure the root user password, enter the sudo passwd command, and then enter the master user password before you can set the root user password.

2.3 ubuntu configuration static IP

After the system is installed, the DHCP assigned dynamic IP address is used by default, and a static IP address is configured for the machine for ease of use.

Open VMware Workstation, click Edit--Virtual network editor , click Change settings

Select VMnet8, view the subnet IP and subnet mask, and if necessary, click NAT Settings

In NAT settings, you can also see information such as the subnet IP and subnet mask, and you can also configure the gateway IP

Enter the command vi/etc/network/interfaces, edit the NIC configuration file

Set the static IP address, based on the subnet information found above, the subnet mask, and the gateway to set. Here the master IP is set to 192.168.10.51

After configuration,: Wq Save,reboot Restart the virtual machine, log in with the root user.

Next, configure the SSH connection, enter the command vi/etc/ssh/sshd_config/, edit the SSH configuration file, and set permitrootlogin to Yes

Enter the command service SSH Restart to restart the SSH service

Here I use a more useful SSH client software Xshell, to connect the virtual machine, create a new connection

Successfully connected Xshell

Set up the salver virtual machine in the same way.

The host's IP can be set through the Control Panel, network, and internet-> network connection->vmware Networks Adapter VMnet8

I set it up as a static ip:192.168.10.10.

2.4 Configuring SSH password-free login

First configure the hosts for 2 virtual machines to open the Hosts file

root@master:~/.ssh# vi/etc/hosts

Add the following configuration

192.168.10.51   Master
192.168.10.52   slaver

Log in to the master machine, create the key, and press Enter all the time, and you will generate 2 files in the /root/.ssh directory Id_rsa and Id_rsa.pub

root@master:~# ssh-keygen-t RSA

Copy the public key id_rsa.pub and rename it to Authorized_keys

root@master:~/.ssh# CP id_rsa.pub Authorized_keys

Create the . SSH folder in the root directory of slaver

root@slaver:~# cd/root
root@slaver:~# mkdir. SSH

Copy the Authorized_keys to the /root/.ssh directory of the slaver virtual machine

The first time to confirm the connection and enter the password for the root user of the slaver virtual machine

When you are done, try SSH to slaver without losing the password to succeed

root@master:~/.ssh# SSH Root@slaver

4. Ubuntu domestic APT Source configuration

First set up the DNS server

root@master:~/.ssh# echo "NameServer 221.12.1.227" >/etc/resolv.conf

Back up the original file

root@slaver:~# Cp/etc/apt/sources.list/etc/apt/sources.list.old

Modify the contents of the file /etc/apt/sources.list as follows

Deb Http://mirrors.163.com/ubuntu/trusty main restricted universe Multiverse
Deb http://mirrors.163.com/ubuntu/ Trusty-security main restricted universe Multiverse
Deb Http://mirrors.163.com/ubuntu/trusty-updates main Restricted universe Multiverse
Deb http://mirrors.163.com/ubuntu/trusty-proposed main restricted universe Multiverse
Deb Http://mirrors.163.com/ubuntu/trusty-backports main restricted universe multiverse
DEB-SRC Http://mirrors.163.com/ubuntu/trusty main restricted universe multiverse
deb-src http://mirrors.163.com/ubuntu/ Trusty-security main restricted universe Multiverse
deb-src http://mirrors.163.com/ubuntu/trusty-updates Main Restricted universe Multiverse
deb-src http://mirrors.163.com/ubuntu/trusty-proposed main restricted universe Multiverse
deb-src http://mirrors.163.com/ubuntu/trusty-backports main restricted universe multiverse

Update Software

Apt-get Update
Apt-get upgrade

5. Installing Jdk,scala,mysql

Upload all the software you use first, and use the XFTP software to upload it to the/opt directory.

5.1 Installing the JDK

Go to/opt directory, unzip jdk-8u121-linux-x64.tar.gz

root@master:/opt# cd/opt/
root@master:/opt# tar-zxvf jdk-8u121-linux-x64.tar.gz

Configuring Environment variables

root@master:/opt# Vi/etc/profile

At the end, add the following lines of configuration

Export java_home=/opt/jdk1.8.0_121
export jre_home= $JAVA _home/jre
export classpath= $JAVA _home/lib: $JRE _ Home/lib: $CLASSPATH
export path= $JAVA _home/bin: $JRE _home/bin: $PATH

Refresh the/etc/profile file

root@master:/opt# Source/etc/profile

Determine if the environment variable is in effect

root@master:/opt# java-version
java Version "1.8.0_121"
Java (TM) SE Runtime Environment (build 1.8.0_121-b13) C2/>java HotSpot (TM) 64-bit Server VM (build 25.121-b13, Mixed mode)

Install from node ibid . 5.2 Installing Scala

Go to/opt directory, unzip scala-2.12.1.tgz

root@master:/opt# cd/opt/
root@master:/opt# tar-zxvf scala-2.12.1.tgz

Configuring Environment variables

root@master:/opt# Vi/etc/profile

Change the bottom few lines of configuration

Export java_home=/opt/jdk1.8.0_121
export jre_home= $JAVA _home/jre
export classpath= $JAVA _home/lib: $JRE _ Home/lib: $CLASSPATH
export scala_home=/opt/scala-2.12.1
export path= $JAVA _home/bin: $JRE _home/bin: $SCALA _ Home/bin: $PATH

Refresh the/etc/profile file

root@master:/opt# Source/etc/profile

Determine if the environment variable is in effect

root@master:/opt# scala-version
Scala Code runner version 2.12.1-Copyright 2002-2016, LAMP/EPFL and Lightbend, Inc. .

Install from node ibid . 5.3 Installing MySQL

root@master:/opt# Apt-get Install Mysql-server

Enter the root user password and repeat

Setting the MySQL character set, first look at the current character set, some latin1 format

root@slaver:/opt# mysql-uroot-p123456 Welcome to the MySQL monitor. Commands End With;
or \g. Your MySQL Connection ID is version:5.5.54-0ubuntu0.14.04.1 (Ubuntu) Copyright (c), Oracle and/or I TS affiliates.

All rights reserved. Oracle is a registered trademark of the Oracle Corporation and/or its affiliates.

Other names trademarks of their respective owners. Type ' help ', ' or ' \h ' for help.

Type ' \c ' to clear the current input statement.
Mysql> Show variables like '%char% '; +--------------------------+----------------------------+
| variable_name |
Value | +--------------------------+----------------------------+
| character_set_client | UTF8 | | character_set_connection | UTF8 | | Character_set_database | Latin1 | | Character_set_filesystem | binary | | Character_set_results | UTF8 | | Character_set_server     | Latin1 | | Character_set_system | UTF8 | | Character_sets_dir |
/usr/share/mysql/charsets/| +--------------------------+----------------------------+ 8 rows in Set (0.00 sec)

Editing a configuration file

root@master:/opt# vi/etc/mysql/my.cnf

Add a configuration under the [MySQL] and [client] tabs

Default-character-set=utf8

Add a configuration under the [mysqld] tab

Character-set-server=utf8

Add # to bind-address = 127.0.0.1 to comment out this line, making it easy for clients to connect

Restart MySQL Service

root@master:/opt# Service MySQL Restart

Connect the MySQL command line terminal to view the current character set condition

root@master:/opt# mysql-uroot-p123456 Welcome to the MySQL monitor. Commands End With;
or \g. Your MySQL Connection ID is version:5.5.54-0ubuntu0.14.04.1 (Ubuntu) Copyright (c), Oracle and/or I TS affiliates.

All rights reserved. Oracle is a registered trademark of the Oracle Corporation and/or its affiliates.

Other names trademarks of their respective owners. Type ' help ', ' or ' \h ' for help.

Type ' \c ' to clear the current input statement.
Mysql> Show variables like '%char% '; +--------------------------+----------------------------+
| variable_name |
Value | +--------------------------+----------------------------+
| character_set_client | UTF8 | | character_set_connection | UTF8 | | Character_set_database | UTF8 | | Character_set_filesystem | binary | | Character_set_results | UTF8 | | Character_set_server     | UTF8 | | Character_set_system | UTF8 | | Character_sets_dir |
/usr/share/mysql/charsets/| +--------------------------+----------------------------+ 8 rows in Set (0.00 sec)

Configuring remote connections

mysql> use MySQL
Reading table information for completion of table and column names you
can turn off this Featu Re to get a quicker startup with-a

Database changed
mysql> Select Host, user from user; 
+-----------+------------------+
| host      | user             |
+-----------+------------------+
| 127.0.0.1 | root             |
|:: 1       | root             |
| localhost | Debian-sys-maint |
| localhost | Root             |
| Master    | root             |
+-----------+------------------+
5 rows in