"Crawler" saves the captured data--crawlers with MongoDB.

"Crawler" saves the captured data--crawlers with MongoDB.

Video Address

The method of fetching data, the previous lesson should have been told, crawl to take down the data is only the first step, the second step is to save it first. The easiest thing to think about is to save the file, and the course before Python writes the file has already been told. It is possible to save to a file, but do you ever think that it is a little geek to open the whole file and read it every time you use it?
So we usually choose to deposit into the database, easy to write and read the data, and for the most part, the Python data structure of the dict enough for us to go to the structured crawl of the data, then the two can play to the ultimate artifact is--mongodb!

Mongodb

1. Distributed
2. Loose data Structures (JSON)
3. Query Language is powerful

Document

You can be seen as a dict,dict inside you can also nest dict, for example:

1. {"name": "alan", score_list: {"chinese": 90, "english": 80}}

Collection

A set of documents is a bunch of dict.

Database

Multiple collections make up a database

So understand: You can think of MongoDB as a library, every book in the library is a document, a book on a bookshelf is a collection, and each library's bookshelf adds up to a database.

Installation

Official installation method
Students who learn my tutorial should know that I will not give specific steps to encourage people to follow the official documents to explore, shielding hand party.

How to store the captured data in MongoDB

1. Write the captured data into the Dict form you want.
2. Insert to the specified bookshelf
3. It's gone.

Additions and Deletions Example Python2 version

Need to install Pymongo

1. pip install pymongo

mongo_api.py

2. # -*- coding: utf-8 -*-
6. # Define your item pipelines here
8. #
10. # Don‘t forget to add your pipeline to the ITEM_PIPELINES setting
12. # See: http://doc.scrapy.org/en/latest/topics/item-pipeline.html
16. import pymongo
18. import sys
20. import unittest
22. reload(sys)
24. sys.setdefaultencoding(‘utf-8‘)
28. class MongoAPI(object):
30. def __init__(self, db_ip, db_port, db_name, table_name):
32. self.db_ip = db_ip
34. self.db_port = db_port
36. self.db_name = db_name
38. self.table_name = table_name
40. self.= Pymongo. Mongoclient (host=self db_ip, Port=selfdb_port)
42. self.db = self.conn[self.db_name]
44. self.table = self.db[self.table_name]
48. def get_one(self, query):
50. return self.table.find_one(query, projection={"_id": False})
54. def get_all(self, query):
56. return self.table.find(query)
60. def add(self, kv_dict):
62. return self.table.insert(kv_dict)
66. def delete(self, query):
68. return self.table.delete_many(query)
72. def check_exist(self, query):
74. ret = self.get(query)
76. return len(ret) > 0
80. # 如果没有 会新建
82. def update(self, query, kv_dict):
84. ret = self.table.update_many(
86. query,
88. {
90. "$set": kv_dict,
92. }
94. )
96. if not ret.matched_count or ret.matched_count == 0:
98. self.add(kv_dict)
100. elif ret.matched_count and ret.matched_count > 1:
102. self.delete(query)
104. self.add(kv_dict)
108. class DBAPITest(unittest.TestCase):
110. def setUp(self):
112. self.db_api = MongoAPI("127.0.0.1", # 图书馆大楼地址
114. 27017, # 图书馆门牌号
116. "test", # 一号图书室
118. "test_table") # 第一排书架
122. def test(self):
124. db_api = self.db_api
126. db_api.add({"url": "test_url", "k": "v"})
128. self.assertEqual(db_api.get_one({"url": "test_url"})["k"], "v")
132. db_api.update({"url": "test_url"}, {"url_update": "url_update"})
134. ob = db_api.get_one({"url": "test_url"})
136. self.assertEqual(ob["url_update"], "url_update")
140. db_api.delete({"url": "test_url"})
142. self.assertEqual(db_api.get_one({"url": "test_url"}), None)
146. if __name__ == ‘__main__‘:
148. unittest.main()

Previous article

"Crawler" saves the captured data--crawlers with MongoDB.