Spark Intellij IDEA Development environment setup

(1) Create a Scala project

File->new->project, such as

Choose Scala

Then next

Where the Project SDK specifies that the installed Jdk,scala SDK specifies the installation of Scala (using the idea's own Scala SDK), here is the item name command Sparkwordcount, then the finish

When developing an application in idea, it is often necessary to write source code through certain files directory organization, such as source file directory, test source file directory, and the following shows the creation of Main/scala source file directory under the SRC directory of IntelliJ idea.
Press F4 or right-click the project file directly

Then choose Open Module setting, open the project configuration, click on the src directory, then right-click Create Main/scala folder, then tap the Scala folder to sources, as shown in

# # # (2) Import Spark 1.5.0 dependency Package
Directly F4 Open Project Structure, and then select libraries

Click + Add external Dependency package, select "Java", then select Spark-assembly-1.5.0-hadoop2.4.0.jar

After success as

This completes the Spark development environment configuration

2. Spark application Development (1) Local execution of the spark WordCount program

Create an Sparkwordcount Application object in the Src/main/scala source file directory, and edit the following:

import org. Apache. Spark. Sparkcontext. _import org. Apache. Spark. {sparkconf, Sparkcontext}object sparkwordcount{def main (args:array[string]) {//input files can be either local Linux system files or other source files, such as HD FS if (args. Length==0) {System. Err. println("Usage:sparkwordcount <inputfile>") System. Exit(1)}//Run as a local thread, you can specify the number of threads,//AS. Setmaster("local[2]"), two threads executed//given below is a single-threaded execution val conf = new sparkconf (). Setappname("Sparkwordcount"). Setmaster("Local"Val sc = new Sparkcontext (conf)//wordcount operation, calculates the number of lines in the file that contain spark Val COUNT=SC. Textfile(Args (0)). Filter(line = line. Contains("Spark")). Count()//print result println ("Count="+count) SC. Stop()  }}

Compile the code directly build->make Project

Then programmed to run the parameters, Run->edit configurations

Main class Input: Sparkwordcount
Program arguments input:/hadooplearning/spark-1.5.0-bin-hadoop2.4/readme.md
Such as:

After the completion of the direct Run->run or ALT+SHIFT+F10 run the program, the execution results such as:

(2) Execute Spark wordcount program on spark Cluster to package sparkwordcount as a jar file

Modify the program contents as follows:

import org. Apache. Spark. Sparkcontext. _import org. Apache. Spark. {sparkconf, Sparkcontext}object sparkwordcount{def main (args:array[string]) {//input files can be either local Linux system files or other source files, such as HD FS if (args. Length==0) {System. Err. println("Usage:sparkwordcount <inputfile> <outputfile>") System. Exit(1)}//When submitting a cluster, the local thread does not function val conf = new sparkconf (). Setappname("Sparkwordcount"). Setmaster("Local"Val sc = new Sparkcontext (conf)//rdd2 for all lines that contain spark Val RDD2=SC. Textfile(Args (0)). Filter(line = line. Contains("Spark")//Save the content, in the example is saved on HDFs RDD2. Saveastextfile(Args (1)) SC. Stop()  }}

Click Project Sparkwordcount, and then press F4 to make a project structure and select artifacts, as

Select Jar->form modules with dependencies, as

Go to the following interface
In main class, select Sparkwordcount, as

Click OK to get the following interface

Because the late commit to run on the cluster, so the relevant jar packages exist, in order to reduce the volume of the jar package, Spark-assembly-1.5.0-hadoop2.4.0.jar and other jar packages can be deleted, such as

When you are sure, then click Build->build Artifacts

The generated jar file is saved in the [email protected]:~/ideaprojects/sparkwordcount/out/artifacts/sparkwordcount_jar# directory, such as:

Submit Cluster Run

Spark: / /sparkmaster:7077--class Sparkwordcount

--executor-memory 1g

/root /ideaprojects/ Sparkwordcount/out/artifacts/sparkwordcount_jar/sparkwordcount.jar

HDFs: / /ns1/ README . MD

HDFs: / /ns1/ Sparkwordcountresult

Execution Result:

HDFs file has been generated by Sparkwordcountresult

Use

root @sparkmaster :/hadooplearning/ spark-1.5 0 -bin-hadoop2. 4 /bin # Hadoop DFS- ls 

/sparkwordcountresult

root @sparkmaster :/ hadooplearning/spark-1.5 0 -bin-hadoop2. 4 /bin # Hadoop DFS- cat 

/sparkwordcountresult/part-00000

To view the contents of the catalog, the results are as follows:

From for notes (Wiz)

Spark Intellij Idea development environment build