Spark Cluster Replacement python installation environment

由于需要用到很多第三方的包，比如numpy，pandas等，而生产环境又无法上网单个包安装太麻烦，因此考虑把之前安装的python的独立环境换成Anaconda的集成开发环境。以下是具体步骤： 1安装Anaconda3-5.0.1-Linux-x86_64.sh脚本 2给/opt/modules/anaconda3/bin/python3.6建立软链接

Ln-s/opt/modules/anaconda3/bin/python/usr/bin/python3
3 Modifying the root environment variable
/root/.bashrc
/root/.bash_profile
Modify the HDFS user environment variable because most of the cluster operations are done by HDFS users.
Su-hdfs
~/.bashrc
Export path= "/usr/bin: $PATH"
4 Modifying the PYSPARK2 command
Vi/usr/bin/pyspark2
Modify the Spark2-submit command
Vi/usr/bin/spark2-submit
Modify the Pyspark_python variable
Several nodes have to be modified, and then spark is replaced with a new python, and the usual packages are available.

Spark Cluster Replacement python installation environment