"Python3 crawler" crawl Blog Home All articles

First, we confirm that the blog home address is: https://www.cnblogs.com/

We open can see there are various articles on the homepage, such as:

Let's take the marked article as an example! Open Web source, search docker, search results such as:

As can be seen from the post-red flag section, we use regular expressions to match the URL, and after we match the URL, we download the corresponding content to store it.

Implementation code

Importurllib.requestImportRe"""Crawl Cnblogs Home all articles"""URL="http://www.cnblogs.com/"Headers= ("user_agent","mozilla/5.0 (Windows NT 6.1; WOW64) applewebkit/537.36 (khtml, like Gecko) chrome/58.0.3029.110 safari/537.36 SE 2.X METASR 1.0") Opener=Urllib.request.build_opener () opener.addheaders=[Headers]urllib.request.install_opener (opener) Data= Urllib.request.urlopen (URL). read (). Decode ("Utf-8","Ignore")#here (. *?) is to match the href after all the content until the "endRule =''result=re.compile (rule). FindAll (data) forIinchRange (0,len (Result)): File="d:/source/"+str (i) +". html"Urllib.request.urlretrieve (Result[i],filename=file)Print("Section"+str (i+1) +"Times Success")

Operation Result:

1th Time Success
2nd time Success
3rd time Success
4th Time Success
5th Time Success
6th time Success
7th Time Success
8th Time Success
9th Time Success
10th Time Success
11th Time Success
12th Time Success
13th Time Success
14th Time Success
15th Time Success
16th Time Success
17th Time Success
18th Time Success
19th Time Success
20th Time Success
[Finished in 9.3s]

After the crawl is complete, we can see the crawled content under the corresponding path

"Python3 crawler" crawl Blog Home All articles