Installation and configuration of Hadoop pseudo-distributed-----spark

The Hadoop environment has been set up in the previous chapters, this section focuses on building the spark platform on Hadoop 1 Download the required installation package

1) Download the spark installation package 2) Download the Scala installation package and unzip the installation package This example takes the following version as an example

2 Configuring environment variables

Use the command sudo gedit/etc/profile to configure the following environment variables

3 Viewing the Scala version

4 Configuring Spark

Copy Spark-env.sh.template to spark-env.sh configure the following variables

5 starting Hadoop and spark clusters 5.1 Starting a Hadoop cluster

1) SSH localhost

2) Hadoop Namenode-format (multiple formatting may cause Datanode to start, which is required to view the log NameID and dataid configuration of the same number)

3) start-all.sh input JPS appears as shown below to start correctly

5.2 Launching the spark cluster

1) Enter Spark's sbin directory input./start-all.sh appears as shown in the following image to start correctly

6 testing Spark

Enter the Spark-shell console of the Spark's Bin directory

7 monitoring cluster status using Web pages