python spark環境配置

標籤：png   添加   技術   右鍵   RoCE   實驗   color   split   sha   

1、下載如下

放在D盤
添加 SPARK_HOME = D:\spark-2.3.0-bin-hadoop2.7。

• 並將 %SPARK_HOME%/bin 添加至環境變數PATH。
• 然後進入命令列，輸入pyspark命令。若成功執行。則成功設定環境變數

找到pycharm sitepackage目錄

右鍵點擊即可進入目錄，將上面D:\spark-2.3.0-bin-hadoop2.7裡面有個/python/pyspark目錄拷貝到上面的 sitepackage目錄

安裝 py4j

實驗如下代碼：

from __future__ import print_functionimport sysfrom operator import addimport os# Path for spark source folderos.environ[‘SPARK_HOME‘] = "D:\spark-2.3.0-bin-hadoop2.7"# Append pyspark to Python Pathsys.path.append("D:\spark-2.3.0-bin-hadoop2.7\python")sys.path.append("D:\spark-2.3.0-bin-hadoop2.7\python\lib\py4j-0.9-src.zip")from pyspark import SparkContextfrom pyspark import SparkConfif __name__ == ‘__main__‘:    inputFile = "D:\Harry.txt"    outputFile = "D:\Harry1.txt"    sc = SparkContext()    text_file = sc.textFile(inputFile)    counts = text_file.flatMap(lambda line: line.split(‘ ‘)).map(lambda word: (word, 1)).reduceByKey(lambda a, b: a + b)    counts.saveAsTextFile(outputFile)

計算成功即可

python spark環境配置