Python Pyspark Introductory article

Python Pyspark Introductory article

I. Introduction to the Environment:

1. Install JDK 7 or more

2.python 2.7.11

3.IDE Pycharm

4.package:spark-1.6.0-bin-hadoop2.6.tar.gz

Two. Setup

1. Unzip spark-1.6.0-bin-hadoop2.6.tar.gz to directory D:\spark-1.6.0-bin-hadoop2.6

2. Configure the environment variable path, add D:\spark-1.6.0-bin-hadoop2.6\bin, after which you can enter Pyspark on the CMD side and return to the following installation:

3. Copy the Pyspark file under D:\spark-1.6.0-bin-hadoop2.6\python to the C:\Python27\Lib\site-packages

4. Install py4j, pip install py4j-i https://pypi.douban.com/simple

5. Configure the PYCHAR environment variable:

Three. Example

1.make a new Python file:wordCount.py

#!/usr/bin/env python#-*-coding:utf-8-*-import sysfrom pyspark import sparkcontextfrom operator import AddImport  R Edef Main ():    sc = sparkcontext (appname= "Wordscount")    lines = sc.textfile (' words.txt ')    counts = Lines.flatmap (Lambda  x:x.split ("))                . Map (Lambda  x: (x, 1))                . Reducebykey (add)    output = Counts.collect ()    print output for    (Word, count) in output:        print "%s:%i"% (word, count)    Sc.stop () if __name__ = = "__main__":    Main ()

2. The words.txt in the code are as follows:

The dynamic lifestylepeople leads nowadayscauses Many reactions in we bodies and the one that's the most frequent of all is the headache

3. Configure the SPARK environment variable for the current running program:

3.1 Toolbar run---Edit configuration--> Click three points behind enviroment variables

3.2 Then click +, enter Key:spark_home, value:d:\spark-1.6.0-bin-hadoop2.6

4. Output results such as:

Four. In-depth exercises:

1. Documentation: http://spark.apache.org/docs/latest/api/python/pyspark.html

2. Under the extracted spark document, there are many instances under example that can be practiced. D:\spark-1.6.0-bin-hadoop2.6\examples\src\main\python

Author: Little Lightning

Source: http://www.cnblogs.com/yueyanyu/

The copyright of this article is owned by the author and the blog Park, Welcome to reprint, Exchange, but without the consent of the author must retain this statement, and in the article page obvious location to give the original link. If you feel this article is good for you, welcome to like, welcome to explore. This blog from the Internet resources, if the infringement of your rights, please contact the blogger to delete.

Translated from: https://www.cnblogs.com/yueyanyu/p/6497956.html

Python Pyspark Introductory article