Install and configure Spark in CentOS 7.0

Install and configure Spark in CentOS 7.0

Installation environment:
Virtual Machine: VMware®Workstation 8.0.1 (network bridging)
OS: CentOS 7
JDK: jdk-7u79-linux-x64.tar
Scala: scala-2.11.7
Spark version: spark-1.4.0-bin-Hadoop2.4
User: created when Hadoop is installed on Centos and belongs to the Administrator group.

Step 1: Configure SSH

Log on to the system using hadoop and run it on the terminal:

Yum install openssh-server

If the prompt is:

The reason is that the yum service is in use and needs to be forcibly unlocked:

Rm-rf/var/run/yum. pid

The terminal will download the installation package online and install it on its own. After the installation is complete, enter the following command to verify whether port 22 is Enabled:

Netstat-nat

Make sure port 22 is enabled, and then check whether the SSH installation is correct. Enter

Ssh localhost

Enter the current user name and password and press enter to confirm that the installation is successful and the password is required for ssh login.

Here we will focus on the following:

The Ssh configuration is actually to configure password-free access and use identity creden。 instead of password verification. You only need to provide one identity creden。 for access without entering the password. That is to say, every user has a unique credential. If you want to access the credential, you need to give it to the user (that is, copy it to another user's directory)

Next, enter a command on the terminal to enter the Hadoop account directory:

Cd/home/hadoop

Enter again:

Ssh-keygen-t rsa

Then press ENTER

Next, go to the. ssh folder again and append id_rsa.pub to the authorized_keys file. The command is as follows:

Cd. ssh

By the way, check what files are in the current directory. Id_rsa is the private key of the account, and id_rsa.pub is the public key of the account.

What if a master server has multiple accounts and requires password-free access?

There should be a file named authorized_keys under the master server. If you need to configure password-free access, append your public key to this file.

Cp id_rsa.pub authorized_keys

Test logon without a password again

Ssh localhost

It is best to open the terminal several times, test ssh logon, or restart the service test:

Service sshd restart service

Service sshd start service

Service sshd stop service

Netstat-antp | grep sshd check whether port 22 is enabled

If the permission is denied at any time, add sudo to the command, and the following will be denied:

Chkconfig sshd on

Chkconfig sshd off disable SSH startup

-------------------------------------- Split line --------------------------------------

Spark1.0.0 Deployment Guide

Install Spark0.8.0 in CentOS 6.2 (64-bit)

Introduction to Spark and its installation and use in Ubuntu

Install the Spark cluster (on CentOS)

Hadoop vs Spark Performance Comparison

Spark installation and learning

Spark Parallel Computing Model

-------------------------------------- Split line --------------------------------------

For more details, please continue to read the highlights on the next page:

1 2 Next Page