[Hadoop+spark+python] Big Data actual combat handy notes

1. Submit a Task

Directive (I have configured the SPARK-SUBMIT environment variable)

An article by article explains as follows:

(1) –class Org.apache.spark.examples.SparkPi

Set up a running environment, Java or yarn

(2) –master spark://192.168.12.233:7077

Set the address of the Spark service in the format of spark://+

(3) –executor-memory 10G

Allocates the memory, this attribute each worker will assign 10G, according to the actual situation assigns

(4) –total-executor-cores 10

Allocate running CPU cores, no more than the total number of cores

(5) Fielname

To run the file, relative path or absolute path can be, if it is a Python file must be able to run in the command line environment, requirment and package environment to be satisfied with the line, I generally is the project package to write a good setup.py first compiled in the Run the main program