SPARK2 file processing and jar package execution

Uploading Data files

Mkdir-p data/ml//home/wangxiao/data/ml/affairs.txt

Hadoop fs-mkdir-p/datafile/wangxiao/

Hadoop Fs-ls/hadoop fs-put/home/wangxiao/data/ml/affairs.txt/datafile/wangxiao/

Hadoop fs-ls-r/datafile

Hdfs://ns1/datafile/wangxiao/affairs.txt

Execute JAR Package

Mkdir-p jar/

$SPARK _home/bin/spark-submit \

--class "ML2" \

--master local[2] \

/home/huai/jar/test.jar

Download data files to local

Hadoop fs-get/datafile/wangxiao/affairs.txt/home/wangxiao/data/ml/

Delete a file on HDFs

Hadoop fs-rm-r-f/datafile/wangxiao/data*

SPARK2 file processing and jar package execution