Python crawler Little practice: Crawl any CSDN blog post text content (or can be rewritten to save other elements), indirectly increase the number of blog visits

 

Python is not my main business, the first to learn Python is mainly to learn reptiles, think that they can crawl from the Internet is a very magical and very useful things, because we can get some aspects of data or other things, anyway, useful.

These two days idle nothing, mainly to let the brain relax on the writing crawler to play, on a preliminary use BeautifulSoup to crawl the basic statistics of a CSDN blog (http://blog.csdn.net/hw140701/article/details/55048364), Today, I would like to go straight to the address of a CSDN blog's homepage to crawl all of the blog's article links, and then extract the elements of each article, I am here to extract each blog

First, the main ideas

through the analysis CSDN blog Site source code, we found when we enter a blog page URL, such as: http://blog.csdn.net/hw140701

There are several articles on the homepage, as well as links to articles, and the default is 15 articles. At the bottom of the home blog there will be a page link, such as

, A total of 65 articles in 5 pages, and each page contains 15 links to the article.

So our overall thinking is:

1. Enter the blog home address to get a link to all articles in the current page;

2. Get the link address for each paging

3. Get the link address of all articles on each page by the link address of each page

4. Depending on the link address of each article, get the content of each article until all of the blog posts have been crawled

Second, code Analysis

2.1 Paging Link Source analysis

Use the browser to open the Web address, using the Developer tool to view the blog home site source code, found that the page link address is hidden in the following tags

So we matched all the paging links with the following code

[python] view plain copy

1. Bsobj.findall ("A", Href=re.compile ("^/([a-za-z0-9]+) (/article) (/list) (/[0-9]+) *$"): #正则表达式匹配分页的链接

Bsobj as BeautifulSoup object

2.2 Pagination Each article link source code Analysis

Get each page of the link, on each page of the article link source analysis, the source code is as follows

Through analysis, so we take the following methods to match

[python] view plain copy

1. Bsobj.findall ("A", Href=re.compile ("^/([a-za-z0-9]+) (/article) (/details) (/[0-9]+) *$"))

Or

[python] view plain copy

1. Bsobj.findall ("span", {"Class": "Link_title"})

2.3 Source code Analysis of text content in each article

Through the analysis of the source code of the website in each article, it is found that the content is located in the following location in the source code

So match it with the following code

[python] view plain copy

1. Bsobj.findall ("span", Style=re.compile ("Font-size: ([0-9]+) px"))

3. All codes and results

now enclose all the code, the comment section may be wrong, you can modify it according to this code, to crawl a Csdn any element in the blog

[python] view plain copy

2. #__author__ = ' Administrat
4. #coding =utf-8
6. Import IO
8. Import OS
10. Import sys
12. Import Urllib
14. from urllib.request import Urlopen
16. from Urllib import Request
18. from BS4 import BeautifulSoup
20. Import datetime
22. Import Random
24. Import RE
26. Import Requests
28. Import Socket
30. Socket.setdefaulttimeout (#设置全局超时函数)
36. Sys.stdout = io. Textiowrapper (sys.stdout.buffer,encoding= ' GB18030 ')
38. headers1={' user-agent ': ' mozilla/5.0 (Windows NT 6.1; WOW64; rv:23.0) gecko/20100101 firefox/23.0 '}
40. headers2={' user-agent ': ' mozilla/5.0 (Windows NT 6.3; WOW64) applewebkit/537.36 (khtml, like Gecko) chrome/45.0.2454.101 safari/537.36 '}
42. headers3={' user-agent ': ' mozilla/5.0 (Windows NT 6.1) applewebkit/537.11 (khtml, like Gecko) chrome/23.0.1271.64 safari/ 537.11 '}
46. #得到CSDN博客某一个分页的所有文章的链接
48. Articles=set ()
50. def getarticlelinks (pageurl):
52. #设置代理IP
54. #代理IP可以上http://zhimaruanjian.com/Get
56. Proxy_handler=urllib.request.proxyhandler ({' Post ': ' 210.136.17.78:8080 '})
58. Proxy_auth_handler=urllib.request.proxybasicauthhandler ()
60. Opener = Urllib.request.build_opener (Urllib.request.HTTPHandler, Proxy_handler)
62. Urllib.request.install_opener (opener)
64. #获取网页信息
66. Req=request. Request (pageurl,headers=headers1 or headers2 or headers3)
68. Html=urlopen (req)
70. Bsobj=beautifulsoup (Html.read (), "Html.parser")
72. Global Articles
74. #return Bsobj.findall ("A", Href=re.compile ("^/([a-za-z0-9]+) (/article) (/details) (/[0-9]+) *$"))
76. #return Bsobj.findall ("a")
78. #for articlelist in Bsobj.findall ("span", {"Class": "Link_title"}):
80. for Articlelist in Bsobj.findall ("span", {"Class": "Link_title"}): #正则表达式匹配每一篇文章链接
82. #print (Articlelist)
84. if ' href ' in Articlelist.a.attrs:
86. if articlelist.a.attrs["href"] not in articles:
88. #遇到了新界面
90. newarticle=articlelist.a.attrs["href"]
92. #print (newarticle)
94. Articles.add (newarticle)
96. #articlelinks =getarticlelinks ("http://blog.csdn.net/hw140701")
98. #for list in Articlelinks:
100. #print (list.attrs["href"])
102. #print (list.a.attrs["href"])
106. #写入文本
108. #def data_out (data):
110. # with open ("E:/csdn.txt", "A +") as Out:
112. # out.write (' \ n ')
114. # out.write (data,)
120. #得到CSDN博客每一篇文章的文字内容
122. def Getarticletext (articleurl):
124. #设置代理IP
126. #代理IP可以上http://zhimaruanjian.com/Get
128. Proxy_handler=urllib.request.proxyhandler ({' https ': ' 111.76.129.200:808 '})
130. Proxy_auth_handler=urllib.request.proxybasicauthhandler ()
132. Opener = Urllib.request.build_opener (Urllib.request.HTTPHandler, Proxy_handler)
134. Urllib.request.install_opener (opener)
136. #获取网页信息
138. Req=request. Request (articleurl,headers=headers1 or headers2 or headers3)
140. Html=urlopen (req)
142. Bsobj=beautifulsoup (Html.read (), "Html.parser")
144. #获取文章的文字内容
146. for textlist in Bsobj.findall ("span", Style=re.compile ("Font-size: ([0-9]+) px")): #正则表达式匹配文字内容标签
148. Print (Textlist.get_text ())
150. #data_out (Textlist.get_text ())
154. #得到CSDN博客某个博客主页上所有分页的链接, get a link to each article based on the pagination link and crawl the text of each blog post
156. Pages=set ()
158. def getpagelinks (Bokezhuye):
160. #设置代理IP
162. #代理IP可以上http://zhimaruanjian.com/Get
164. Proxy_handler=urllib.request.proxyhandler ({' Post ': ' 121.22.252.85:8000 '})
166. Proxy_auth_handler=urllib.request.proxybasicauthhandler ()
168. Opener = Urllib.request.build_opener (Urllib.request.HTTPHandler, Proxy_handler)
170. Urllib.request.install_opener (opener)
172. #获取网页信息
174. Req=request. Request (bokezhuye,headers=headers1 or headers2 or headers3)
176. Html=urlopen (req)
178. Bsobj=beautifulsoup (Html.read (), "Html.parser")
180. Links to all Articles #获取当前页面 (first page)
182. Getarticlelinks (Bokezhuye)
184. #去除重复的链接
186. Global Pages
188. for PageList in Bsobj.findall ("A", Href=re.compile ("^/" ([a-za-z0-9]+) (/article) (/list) (/[0-9]+) *$ ")): #正则表达式匹配分页的链接
190. if ' href ' in Pagelist.attrs:
192. if pagelist.attrs["href"] not in pages:
194. #遇到了新的界面
196. newpage=pagelist.attrs["href"]
198. #print (NewPage)
200. Pages.Add (NewPage)
202. #获取接下来的每一个页面上的每一篇文章的链接
204. newpagelink= "http://blog.csdn.net/" +newpage
206. Getarticlelinks (Newpagelink)
208. #爬取每一篇文章的文字内容
210. for Articlelist in articles:
212. Newarticlelist= "http://blog.csdn.net/" +articlelist
214. Print (newarticlelist)
216. Getarticletext (Newarticlelist)
218. #getArticleLinks ("http://blog.csdn.net/hw140701")
220. Getpagelinks ("http://blog.csdn.net/hw140701")
222. #getArticleText ("http://blog.csdn.net/hw140701/article/details/55104018")

Results

In which sometimes garbled, this is due to the existence of a space, and temporarily find a way to solve.

In addition, there are times when the server does not respond to the error, as follows:

Python crawler Little practice: Crawl any CSDN blog post text content (or can be rewritten to save other elements), indirectly increase the number of blog visits