Scrapy Connecting MongoDB

The work you need to do to connect MongoDB in Scrapy is as follows:

Parts of the 1.settings that need to be set:

# Start pipeline Component Item_pipelines = {    'QianChengWuYu.mongoDBPiplelines.MongoDBPipline': 300 ,}

# connection parameters for Mondb ' mongodb://localhost:27017 '  'scrapy_db'

(1) Where Item_pipelines is set in the Mongodbpipline class in the pipeline file Mongodbpiplelines in the Qianchengwuyu project, the priority level is 300

(2) Where Mongodb_uri is the specified login IP here is localhost, the port is 27017,mongodb_db_name is the database name in MONGODB

The Piplines files in 2.scrapy are as follows:

"""This class is used to insert data into the MONGDB database""" fromPymongoImportmongoclient fromScrapyImportItemclassMongodbpipline (object):defOpen_spider (self, spider):"""This method is used to connect to the database"""Db_uri= Spider.settings.get ('Mongodb_uri','mongodb://localhost:27017') db_name= Spider.settings.get ('Mongodb_db_name','Scrapy_default') Self.db_client= Mongoclient ('mongodb://localhost:27017') self.db=Self.db_client[db_name]defClose_spider (self, spider):"""This method is used to close the database"""self.db_client.close ()defProcess_item (self, item, spider):"""This method is used to insert data"""self.insert_db (item)returnItemdefinsert_db (self, item):ifisinstance (item, Item): Item= Dict (item)#convert a piece of data into a dictionary formatSelf.db.books.insert_one (item)#inserting data into the collection books

Scrapy Connecting MongoDB