Python crawler project, get news on all websites, save to database, parse HTML page, etc.

1. Requirements Description

Demand:
Crawl all the news from the Tiger sniffing site and save it to the database.
Http://www.huxiu.com

Technology:
1. Crawler
Get the server's resources (URLLIB)
Parsing HTML pages (BeautifulSoup)
2. Database Technology
Database MySQLdb
Analysis of business logic:
(1), Tiger olfactory website news, including home page and pagination information (next page)
(2), need to get the URL of each news from the resources and paging resources of the homepage
How to get the URL:
Parsing the Web site HTML file, if the href attribute of the A tag contains the article field, it is a news
(3) Access the URL of the news, resolve the desired field

http://www.woaipu.com/shops/zuzhuan/61406

http://www.woaipu.com/shops/zuzhuan/61406

Python crawler project, get news on all websites, save to database, parse HTML page, etc.