IntelliJ idea Development spark, hit Jar pack

Qingming Holiday toss for two days, summed up two ways to use the IDE for the Spark program, record:

The first method is simpler, both of which are compiled with SBT.

Note: There is no need to install the Scala program locally, otherwise there is a version compatibility issue when compiling the program.

First, based on the NON-SBT way

Create a Scala idea project

We use the NON-SBT method, click "Next"

Name the project, others follow the default

Click "Finish" to complete the project creation

Modify the properties of an item

First modify the Modules option

Create two folders under SRC and change their properties to source

Modify libraries Below

Bring in the jar packages needed for spark development

After joining:

After the import package is complete, create a packages below the project Scala

Create an Object

Building Spark Driver Code

This program is the processing code of the Sogou log

Next package, use project structure's artifacts

Using the From modules with dependencies

Select Main Class

Click "OK"

Change the name to Firstsparkappjar

Because Scala and spark are installed on each machine, you can erase both Scala and spark-related jar files.

The next Build

Select "Build Artifacts"

Select Build for the first time, then select rebuild for the same project, then wait for the compilation to complete

Run the program using Spark-submit below

Final task Run complete

Two, based on the SBT method

Development tools Download

Spark development requires the following development compilation tools:

1. Scala IDE: This article takes IntelliJ idea as an example to develop, download the address:

https://www.jetbrains.com/idea/download/

2. The SBT (Simple Build tool) compilation Tools Download:

Http://www.scala-sbt.org/download.html

After downloading the installation, execute the SBT command under DOS to download the jar package that it requires:

The default jar package (. Idea-build,. Ivy2,. SBT) is downloaded to the C-drive user directory

(Note: Use the command SBT to make sure the Internet speed, preferably using a proxy download)

Development tool Configuration