Python scrapy to heavy __python

1, pipleline Add the following code: (after the data crawl after the URL into the Redis to heavy)

Class Redisinsert (object):
   
    def process_item (self,item,spider):
        set_redis_values_1 (item[' url ')
        Return item

2. Middleware Add the following code: (see if the URL crawled before crawling the data)

Class Ingorerequestmiddleware (object):
    def __init__ (self):
        Self.middlewarelogging=getlogger (" Ingorerequestmiddleware ")

    def process_request (self,request,spider):
        if Get_redis_values_1 (request.url):
            self.middlewareLogging.debug ("ignorerequest:%s"% request.url)
            raise Ignorerequest ("ignorerequest:%s"% Request.url)
        Else:
            self.middlewareLogging.debug ("haverequest:%s"% Request.url) return
            None

Ok...