WordCount Interactive analysis in the Spark shell based on the HDFs file system

Spark is a distributed memory computing framework that can be deployed in yarn or Mesos managed distributed Systems (Fully distributed) or in a pseudo distributed way on a single machine. It can also be deployed on a single machine in a standalone manner. There are interactive and submit ways to run spark. All of the actions in this article are interactive operations that are deployed in standalone mode by Spark. Refer to Hadoop Ecosystem for specific deployment options.

HDFS is a distributed file management system that is installed by default as Hadoop is installed. The deployment method has local mode and cluster mode, which is used in this article when local mode. Refer to Hadoop Ecosystem for specific deployment options.

Goal:

The ability to perform wordcount operations in Spark-shell via the HDFs file system.

Premise:

There is a file that can be viewed by the following command.

Hadoop Fs-ls/

If it does not exist, add one (the license file needs to exist in the local directory). For more Hadoop commands, refer to Hadoop commands.

Hadoop fs-put License/license.txt

See if Hadoop is already running through the Web browser.

http://localhost:50070

Steps:

Step 1: Enter the Spark-shell interactive command line.

Spark-shell

Step 2: Read the License.txt file and check whether the read was successful. If it does not exist, the following error is indicated.

val s = sc.textfile ("Hdfs://localhost:9000/license.txt")

S.count

Step 3: Set the number of files to output and perform statistical logic

Val numoutputfiles = 128

Val counts = S.flatmap (line = Line.split ("")). Map (Word = = (Word, 1)). Reducebykey (_ + _, Numoutputfiles)

Step 4: Save the calculation results in HDFs

Counts.saveastextfile ("Hdfs://localhost:9000/license_hdfs.txt")

Step 5: View the results in the shell

Hadoop fs-cat/license_hdfs.txt/*

Conclusion:

With HDFs, we can easily perform interactive analysis (word count statistics) in Spark-shell.

Resources:

Http://hadoop.apache.org/docs/r1.0.4/cn/commands_manual.html

Http://spark.apache.org/docs/latest/programming-guide.html

Http://coe4bd.github.io/HadoopHowTo/sparkScala/sparkScala.html

Http://coe4bd.github.io/HadoopHowTo/sparkJava/sparkJava.html

WordCount Interactive analysis in the Spark shell based on the HDFs file system