Pyspark Add Redis module _spark

Installing the Redis module
and pack the Redis module

Pip install Redis
mkdir redis
mv .../site-packages/redis redis import shutil dir_name
= "Redis"
output_filename = "./redis"
shutil.make_archive (output_filename, ' zip ', dir_name)

Redis.zip folder structure, must have Redis folder as root folder

redis/
redis/lock.pyc
redis/connection.py
redis/exceptions.py
redis/utils.pyc
redis/_ Compat.pyc
redis/_compat.py
redis/connection.pyc
redis/__init__.py
redis/client.py
redis/utils.py
redis/client.pyc
redis/sentinel.py
redis/sentinel.pyc
redis/lock.py
REDIS/__INIT__.PYC
REDIS/EXCEPTIONS.PYC

Here's the wrong

Lock.pyc
connection.py
exceptions.py
utils.pyc
_compat.pyc
_compat.py Connection.pyc
__init__.py
client.py
utils.py
client.pyc
sentinel.py Sentinel.pyc
lock.py
__init__.pyc
exceptions.pyc

Spark.sparkContext.addPyFile ("Redis.zip")
import Redis

This way, the Redis module is loaded into spark nodes.

Or

./bin/spark-submit--py-files Wesam.zip mycode.py