Scrapy Basics (12) ———— asynchronously export the item data to MySQL

Asynchronously exporting data to MySQL

Last time I said I wrote the database synchronously from item because the network download speed is not the same as the database's I/O speed
So it is possible that the download is fast, but writing to the database is slow, causing the thread to jam; about clogging and non-clogging, synchronous and asynchronous
The relationship, because no contact, so it is not very good understanding, here to see some information, has been recorded
Link 1:https://www.zhihu.com/question/19732473;http://wiki.jikexueyuan.com/project/twisted-intro/

The idea of implementing an asynchronous write to the MySQL database:

1. Write the connection data of the database to the settings file for later custom pipeline use
2, custom pipeline, using the Twisted framework for asynchronous
3, register this pipeline in Settings
Specific code implementation:

1:settings, if the database is already designed, the reference synchronization mechanism is written in MySQL

2:
and importing import mysqldb.cursors

1  fromTwisted.enterpriseImportAdbapi2 classMysqltwistedpipeline (object):3     " "4 asynchronous mechanism to write data to the MySQL database5     " "6     #creates an initialization function that is called the first method when an object is created through this class7     def __init__(self,dbpool):8Self.dbpool =Dbpool9     #To create a static method, the static method has a higher load memory priority than the Init method, and Java's static method is similar.Ten     #before creating a pair of this class, it is loaded into memory, so the Init method can invoke the object generated by this method . One @classmethod A     #fixed-name -     deffrom_settings (cls,settings): -         #first, take out the contents of the connected database in setting and construct a location theDbparms =Dict ( -host= settings["Mysql_host"], -db = settings["Mysql_dbname"], -user = settings["Mysql_user"], +passwd = settings["Mysql_password"], -CharSet ="Utf-8", +             #Cursor Settings ACursorclass =Mysqldb.cursors.DictCursor, at             #set whether encoding uses Unicode -Use_unicode =True -         ) -         #connect to the database through the container provided by the twisted Framework, MYSQLDB is the database module name -Dbpool = Adbapi. ConnectionPool ("MySQLdb", Dbparms) -         returnCLS (dbpool) in  -     defProcess_item (self,item,spider): to         #inserting item data into a database asynchronously using twisted +query =self.dbpool.runInteraction (Self.do_insert,item) -Query.adderrback (Self.handle_error,item,spider)#There is no need to accept the incoming item,spider,handle_error, item,spider) the      *     defDo_insert (self,cursor,item): $         #executes the specific INSERT statement, does not need the commit operation, twisted automaticallyPanax NotoginsengInsert_sql =""" - INSERT INTO jobbole_artical (title,creat_date,url,url_object_id, the Front_image_url2,front_image_path,tags,comment_num, + fav_num,like_num,content A                  ) the VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s) +         """ -Cursor.execute (Insert_sql, (item["title"],item["creat_date"],item["URL"], $item["url_object_id"],item["Front_image_url2"],item["Front_image_path"], $item["Tags"],item["Comment_num"],item["Fav_num"],item["Like_num"],item["content"])) -  -     defHandle_error (self, failure, item, spider): the         #out Asynchronous Insert exception -         Print(failure)

3: Registered in Settings:

At this point, asynchronously writes the data of item to MySQL;
Show results: don't know why, feel asynchronous writes are slightly slower than synchronous writes (same computer and speed conditions), also please understand the big God pointing under

Scrapy Basics (12) ———— asynchronously export the item data to MySQL