spark執行例子eclipse maven打包jar

標籤：https   環境   hadoop   enc   develop   ace   win32   代碼   eve   

首先在eclipse Java EE中建立一個Maven project具體選項如下

 

 

點擊Finish建立成功，接下來把預設的jdk1.5改成jdk1.8

 

 

 

然後編輯pom.xml加入spark-core依賴

<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.2.1</version>
</dependency>

然後拷貝書中的源碼範例程式，由於書中spark版本為1.2我的環境spark是2.2.1所以需要修改代碼適應新版本spark API

JavaRDD<String> words = input.flatMap(
new FlatMapFunction<String, String>() {
public Iterator<String> call(String x) {
return Arrays.asList(x.split(" ")).iterator();
}});

 

 

 

 

 然後執行Maven install接下來可以進入目錄E:\developtools\eclipse-jee-neon-3-win32\workspace\learning-spark-mini-example\target找到learning-spark-mini-example-0.0.1-SNAPSHOT.jar並上傳到spark2.2.1環境的linux目錄

 

 

 然後在linux中執行如下命令，如

[[email protected] ~]# spark-submit \
> --class com.oreilly.learningsparkexamples.mini.java.WordCount \
> learning-spark-mini-example-0.0.1-SNAPSHOT.jar \
> /opt/spark-2.2.1-bin-hadoop2.7/README.md wordcounts

 

 

spark執行例子eclipse maven打包jar