Spark development environment-locally installed spark2.x and boot

Developing a spark project with Python requires that spark be installed locally

A Local installation

1. Download http://spark.apache.org/downloads.html

Select the Hadoop version for this machine, click the link to download

2. Click the link to complete the download

3. Extracting files

4. Configure Environment variables

New spark_home=d:\spark\spark-2.2.0-bin-hadoop2.6

Append%spark_home%/bin to the system variable path;

Two to start spark locally

1. Enter D:\spark\hadoop-2.6.5\bin\winutils.exe chmod 777 D:\tmp\hive in cmd (you need to create a new folder first D:\tmp\hive)

2. Run Spark-shell. Be careful not to drive in C, because the Spark-shell run will produce some temporary files, create a new Spark-shell file in the D-Drive spark, and then run the Spark-shell after you enter this folder from CMD.

3. Enter Spark.conf.get ("Spark.sql.catalogImplementation") after entering Spark-shell, and then test the code in an interactive environment.

The Sparkshell folder will appear