Redis and Scrapy

<title>Redis and Scrapy</title> Redis and Scrapy

Redis is a open source, BSD licensed, Advanced Key-value cache and store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets, sorted sets, Bitma PS and Hyperloglogs. --redis Home Page

1. Installing the Scrapy-redis module

• Pip Install Scrapy-redis
• Easy_install Scrapy-redis

2. Installing and Running Redis

• Http://redis.io/download
• To run Redis:redis-server redis.conf
• Empty the cache:redis-cli flushdb

3. Scrapy Configuring Redis

• settings.py Configuring Redis

"scrapy_redis.scheduler.Scheduler"
SCHEDULER_PERSIST =True
‘scrapy_redis.queue.SpiderPriorityQueue‘
None
‘127.0.0.1‘#REDIS服务器的ip
6379#REDIS服务器端口

• Spider calls Redis

fromimport RedisSpider
class xxx(RedisSpider):
    ..................
    ‘username:password‘
    ..................

Redis and Scrapy