Python implementation of Width-first traversal crawler

Crawler width first traverse python

Online is a well-known crawler tutorial, "write your own manual web crawler," the book all the source code is written in java,

It refers to the Width-first traversal algorithm, idle To do nothing I put him in Python to realize it again. Less than half of the code, Hehe.

Introduction to Width First algorithm

Reference: http://book.51cto.com/art/201012/236668.htm

The entire Width-first crawler process starts with a series of seed nodes, extracts the "child nodes" (I.E. Hyperlinks) from these pages and fetches them in a Queue. The processed links need to be placed in a table (often called a visited table). Before each new link is processed, you need to see if the link already exists in the visited Table. If it exists, the link has been processed, skipped, not processed, or the next step is Processed.

The initial URL address is the seed URL provided in the crawler system (typically specified in the System's configuration file). When parsing the Web page represented by these seed urls, a new URL is generated (such as extracting http://www.admin.com from the <a href= "http://www.admin.com" in the page). then, take the following work:

(1) Compare the parsed link with the link in the visited table if the link does not exist in the visited table, indicating that it has not been accessed.

(2) put the link into the Todo table.

(3) once the processing is complete, get a link from the Todo table and put it directly into the visited Table.

(4) continue the above process for the Web page indicated by this Link. So Cyclical.

Table 1.3 shows the crawl process for the page shown in Figure 1.3.

Table 1.3 Network Crawling

TODO table	Visited table
A	Empty
Bcdef	A
Cdef	a, b
Def	A,b,c

Continuation table

TODO table	Visited table
Ef	A,b,c,d
FH	A,b,c,d,e
Hg	A,b,c,d,e,f
Gi	A,b,c,d,e,f,h
I	A,b,c,d,e,f,h,g
Empty	A,b,c,d,e,f,h,g,i

Width-first traversal is the most widely used crawler strategy in reptiles, and the reason for using the Width-first search strategy is three points:

Important pages tend to be closer to the seeds, such as when we open news sites that are often the hottest news, and as we continue to surf, the pages we see are becoming less important.

The actual depth of the World Wide Web can reach up to 17 levels, but there is always a short path to a web Page. The Width-first traversal will reach this page at the fastest Speed.

Width priority is beneficial to Multi-crawler cooperative crawl, multi-crawler cooperation is usually first crawl site links, crawling closed very strong.

Python implementation of Width-first traversal crawler

Python code

2. #encoding =utf-8
4. From BeautifulSoup import BeautifulSoup
6. Import socket
8. Import Urllib2
10. Import re
14. Class Mycrawler:
16. def __init__ (self,seeds):
18. #使用种子初始化url队列
20. Self.linkquence=linkquence ()
22. If Isinstance (seeds,str):
24. SELF.LINKQUENCE.ADDUNVISITEDURL (seeds)
26. If Isinstance (seeds,list):
28. For I in Seeds:
30. SELF.LINKQUENCE.ADDUNVISITEDURL (i)
32. Print "Add the seeds URL \"%s\ "to the unvisited url list"%str (self.linkQuence.unVisited)
34. #抓取过程主函数
36. def crawling (self,seeds,crawl_count):
38. #循环条件: the link to be crawled is not empty and the page of the zone is not more than Crawl_count
40. While Self.linkQuence.unVisitedUrlsEnmpy () is False and Self.linkQuence.getVisitedUrlCount () <=crawl_count:
42. #队头url出队列
44. Visiturl=self.linkquence.unvisitedurldequence ()
46. Print "Pop out one URL \"%s\ "from unvisited url List"%visiturl
48. If Visiturl is None or visiturl== "":
50. Continue
52. #获取超链接
54. Links=self.gethyperlinks (visiturl)
56. Print "Get%d new Links"%len (links)
58. #将url放入已访问的url中
60. SELF.LINKQUENCE.ADDVISITEDURL (visiturl)
62. Print "visited URL count:" +str (self.linkQuence.getVisitedUrlCount ())
64. #未访问的url入列
66. For link in links:
68. SELF.LINKQUENCE.ADDUNVISITEDURL (link)
70. Print "%d unvisited links:"%len (self.linkQuence.getUnvisitedUrl ())
74. #获取源码中得超链接
76. def gethyperlinks (self,url):
78. links=[]
80. Data=self.getpagesource (url)
82. If data[0]== "200":
84. Soup=beautifulsoup (data[1])
86. A=soup.findall ("a", {"href": re.compile (". *")})
88. For I in a:
90. If I["href"].find ("/http")!=-1:
92. Links.append (i["href"])
94. Return links
98. #获取网页源码
100. def Getpagesource (self,url,timeout=100,coding=none):
102. Try
104. Socket.setdefaulttimeout (timeout)
106. req = urllib2. Request (url)
108. Req.add_header (' user-agent ', ' mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) ')
110. Response = Urllib2.urlopen (req)
112. If coding is None:
114. coding= Response.headers.getparam ("charset")
116. If coding is None:
118. Page=response.read ()
120. Else
122. Page=response.read ()
124. Page=page.decode (coding). encode (' utf-8 ')
126. return ["$", page]
128. Except Exception,e:
130. Print str (e)
132. return [STR (e), None]
136. Class Linkquence:
138. def __init__ (self):
140. #已访问的url集合
142. self.visted=[]
144. #待访问的url集合
146. self.unvisited=[]
148. #获取访问过的url队列
150. def Getvisitedurl (self):
152. Return self.visted
154. #获取未访问的url队列
156. def Getunvisitedurl (self):
158. Return self.unvisited
160. #添加到访问过得url队列中
162. def Addvisitedurl (self,url):
164. Self.visted.append (url)
166. #移除访问过得url
168. def Removevisitedurl (self,url):
170. Self.visted.remove (url)
172. #未访问过得url出队列
174. def unvisitedurldequence (self):
176. Try
178. return Self.unVisited.pop ()
180. Except
182. Return None
184. #保证每个url只被访问一次
186. def Addunvisitedurl (self,url):
188. If url!= "" and URL not in self.visted and URLs not in self.unvisited:
190. Self.unVisited.insert (0,url)
192. #获得已访问的url数目
194. def Getvisitedurlcount (self):
196. Return Len (self.visted)
198. #获得未访问的url数目
200. def Getunvistedurlcount (self):
202. Return Len (self.unvisited)
204. #判断未访问的url队列是否为空
206. def Unvisitedurlsenmpy (self):
208. Return len (self.unvisited) ==0
212. def main (seeds,crawl_count):
214. Craw=mycrawler (seeds)
216. Craw.crawling (seeds,crawl_count)
218. If __name__== "__main__":
220. Main (["http://www.baidu.com", "http://www.google.com.hk"],50)

Python implementation of Width-first traversal crawler