The construction of Python environment under spark distributed platform

Actually, I've been doing this for two weeks. Two weeks ago, I received this project to complete a python-written music recommendation system based on the spark distributed platform. I've heard of Python before, and this time I accidentally saw a word "life is short,you need Python". I make up my mind to learn python well.

First of all, project because of the need for Python development environment must be his version >2.7.3, and I use the CentOS operating system, its own Python version is 2.6.0, no way to meet the requirements. So the first thing I did was upgrade the Python version.

Upgrade for version 1.python:

1) Download Python-2.7.3

wget http://python.org/ftp/python/2.7.3/Python-2.7.3.tar.bz2

2) Unzip

TAR-JXVF python-2.7.3.tar.bz2

3) You need to install SSL in advance.

Yum Install Openssl-devel-y

4) Turn on SSL

Vim Python-2.7.3/modules/setup.dist

Find the SSL-related section and remove the comment.

#修改变动如下:

# Socket Module Helper for Socket (2)

_socket socketmodule.c timemodule.c

# Socket Module helper for SSL support; You must comment out of the other

# socket line above, and possibly edit the SSL variable: #SSL =/usr/local/ssl

_ssl _ssl.c \-duse_ssl-i$ (SSL)/include-i$ (SSL)/INCLUDE/OPENSSL \

-l$ (SSL)/lib-lssl-lcrypto

5) Install Python:

./configure

Make all

Make install

Make clean

Make Distclean

6. View the Python version you just installed:

/usr/local/bin/python2.7-v

7) Point the system's default Python to version 2.7

mv/usr/bin/python/usr/bin/python2.6.0

Ln-s/usr/local/bin/python2.7/usr/bin/python

The Python-v view will then be displayed as 2.7.3.

2. Because the project needs to use three libraries in Python, numpy,scipy and Recsys.

Here is a brief introduction to these three libraries:

NumPy: A scientific computing package implemented in Python. Includes: 1, a powerful n-dimensional array object array;2, a relatively mature (broadcast) function library, 3, for the integration of C/D and Fortran Code Toolkit, 4, Practical linear algebra, Fourier transform and random number generation function. NumPy and sparse matrix Operations pack scipy are more convenient to use. NumPy (Numeric Python) offers a number of advanced numerical programming tools such as matrix data types, vector processing, and sophisticated operations libraries. Designed for rigorous digital processing.

SciPy: is a convenient, easy-to-use, scientific and Engineered Python Toolkit. It includes statistics, optimization, integration, linear algebra modules, Fourier transforms, signal and image processing, ordinary differential equation solvers, and more.

Recsys: It provides matrix operations for some algorithms (see http://ocelma.net/software/python-recsys/build/html/index.html for more information)

Installation process:

sudo apt-get install python-scipy python-numpysudo apt-get install python-pipsudo pip install csc-pysparse networkx divisi If you don't have pip installed# (i.e. the last command, sudo pip install, fails) # then do:# sudo easy_install csc-pysp arse# sudo easy_install networkx# sudo easy_install divisi2

Then download the Python--recsys (GitHub Link: https://github.com/ocelma/python-recsys)

Tar xvfz python-recsys.tar.gzcd python-recsyssudo python setup.py install

At this point, my spark-based Python development environment has been completed.

The construction of Python environment under spark distributed platform