Some essays in Ubuntu 16.04 that use Python-scrapy to store crawled data in a MySQL database

The code to save the crawled data to the MySQL database (already able to save the crawled data to a JSON file)

(1) Edit pipeline.py file

　　　　

(2) Edit settings.py file

　　　　

　　　　

Ii. problems with saving data to MySQL database

(1) When the data stored in the database, has been reported 1366 of the error, through the search data, found that the problem of data encoding, in the blog to find several solutions, a better record:

1. First stop the MySQL service, edit the configuration file my.cnf

sudo stop MySQL

sudo vim/etc/mysql.cnf

Add to:

　　　　　　

2. Restart the service

sudo start MySQL

sudo mysql-u root-p

Use root

Show variables like '%char% '; #更改前存在的用户字符集不会改变, the newly created user character set will become UTF8 and re-build the field in the database

This paragraph for reference https://my.oschina.net/sldk/blog/266268

(2) At the time of the crawl save, the data is incomplete, after the inspection found that the data in the crawl has empty data, so in the crawl should do non-empty processing

Third, export the database (in the case of FP) (I am using the root user)

(1) Export data and table structure

Export the SQL script using the mysqldump command (if you do not specify an export path, export to the current path by default)

Format: Mysqldump-u user name-p (password) database name > database name. sql

Mysqldump-u root-p FP > Fp.sql

Prompt for password after enter

(2) Export table structure only

Format: Mysqldump-u user name-p (password)-d database name > database name. sql

Mysqldump-u root-p-D FP > Fp.sql

Second, import the database

Import and export I studied this guy's blog.

https://my.oschina.net/linuxphp/blog/693650

　　

Use Python-scrapy in Ubuntu 16.04 to save crawled data to a MySQL database in some essays