IPython Notebook running Python spark program

1. Install Pip

Because the centos7.0 python system is 2.7.5, and Pip is not installed, you need to install PIP first

$ wget https://bootstrap.pypa.io/get-pip.py$ python get-pip.py$ pip install numpy pandas scipy Jupyter

　　

2. Configure Startup Items

$ vim./.bashrcexport pyspark_driver_python=/usr/bin/ipythonexport pyspark_python=/usr/bin/python$ source./.BASHRC

　　

3.jupyter cannot be accessed remotely

$ jupyter Notebook--allow-root

In fact, at this time, local if there are browser words, you can enter access, but not, so need remote access: http://ip:8888, found not access

(i) Configuring remote access Jupyter

1) First enter Ipython to generate the secret key

$ ipythonfrom Notebook.auth Import passwdpasswd ()

　　

Setting a password generates a SHA1 secret key, such as:

2) Generate the config file for Jupyter

$ jupyter Notebook--generate-config

　　

The configuration file is generated at the ~/.jupyter/jupyter_notebook_config.py

3) Modify configuration file: ~/.jupyter/jupyter_notebook_config.py

$vim ~/.jupyter/jupyter_notebook_config.py

　　

Add the following, where SHA1 is the string that was generated above

c.notebookapp.ip= ' * ' C.notebookapp.password = U ' sha1:f9030dd55bce:75fd7bbaba41be6ff5ac2e811b62354ab55b1f63 ' C. Notebookapp.open_browser = Falsec.NotebookApp.port =8888

Save exit.

4) Start Jupyter

$jupyter Notebook--allow-root

　　

On the remote computer, open the browser and enter:

http://your-server-ip:8888

Need to enter a password, that is the password set above, input can

4. Start

$ Pyspark_driver_python=ipython pyspark_driver_python_opts= "Notebook--allow-root" Pyspark # Where--allow-root is due to root login master

　　

IPython Notebook running Python spark program