The scrapy custom pipeline class saves collected data to mongodb.

The scrapy custom pipeline class saves collected data to mongodb.

This example describes how to save collected data to mongodb using the scrapy custom pipeline class. Share it with you for your reference. The details are as follows:

# Standard Python library imports# 3rd party modulesimport pymongofrom scrapy import logfrom scrapy.conf import settingsfrom scrapy.exceptions import DropItemclass MongoDBPipeline(object):  def __init__(self):    self.server = settings['MONGODB_SERVER']    self.port = settings['MONGODB_PORT']    self.db = settings['MONGODB_DB']    self.col = settings['MONGODB_COLLECTION']    connection = pymongo.Connection(self.server, self.port)    db = connection[self.db]    self.collection = db[self.col]  def process_item(self, item, spider):    err_msg = ''    for field, data in item.items():      if not data:        err_msg += 'Missing %s of poem from %s\n' % (field, item['url'])    if err_msg:      raise DropItem(err_msg)    self.collection.insert(dict(item))    log.msg('Item written to MongoDB database %s/%s' % (self.db, self.col),        level=log.DEBUG, spider=spider)    return item

I hope this article will help you with python programming.