Redis與Scrapy

標籤：

Redis與Scrapy Redis與Scrapy

Redis is an open source, BSD licensed, advanced key-value cache and store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets, sorted sets, bitmaps and hyperloglogs. ——Redis Home Page

1. 安裝scrapy-redis模組

• pip install scrapy-redis
• easy_install scrapy-redis

2. 安裝和運行Redis

• http://redis.io/download
• 運行Redis：redis-server redis.conf
• 清空緩衝：redis-cli flushdb

3. Scrapy配置Redis

• settings.py配置Redis

SCHEDULER = "scrapy_redis.scheduler.Scheduler"
SCHEDULER_PERSIST =True
SCHEDULER_QUEUE_CLASS = ‘scrapy_redis.queue.SpiderPriorityQueue‘
REDIS_URL = None
REDIS_HOST = ‘127.0.0.1‘ #REDIS伺服器的ip
REDIS_PORT = 6379 #REDIS伺服器連接埠

• Spider調用Redis

from scrapy_redis.spiders import RedisSpider
class xxx(RedisSpider):
    ..................
    redis_key = ‘username:password‘
    ..................

Redis與Scrapy