Python's first program small crawler

Recently wanted to get started with Python. A quick way to get started with a language is to write a small demo. The Python demo must be a reptile. The first small reptile program is a little primitive, master do not spray.

The crawler is divided into three parts: crawling the interface, getting the content, and saving the results based on the URLs in the queue.

The program is to the Baidu website Daquan as the seed URL, crawl page URL in order to put in the queue, the crawler from the URL queue in turn to get a new URL to continue to crawl outside.

#-*-Coding:utf-8-*-ImportUrllib2ImportReImportThreadImportTime class html_spider:     def __init__(self):Self.url = []#根据队列中的URL爬取界面     def getpage(self,url):        Try: myresponce = Urllib2.urlopen (URL) mypage = Myresponce.read () Myurl = Re.findall (' href= ' (. *?) "', Mypage,re. S) Self.url.extend (Myurl);except:Print u ' current URL is not valid 'MyPage ="'        returnMyPage#以HTML的形式保存界面     def savepage(self,page):        ifPage! ="':#以时间戳的形式为文件命名f = open (Time.strftime (str (time.time ()), Time.localtime (Time.time ())) +'. html ',' w+ ')#解决pagenama问题, it's best to use save time to nameF.write (page) f.close ()#保持URL队列     def startspider(self):i =1         while 1:ifi = =1: url =u ' http://site.baidu.com/'            Else: url = self.url[i] i + =1            PrintURL page = self. GetPage (URL) self. Savepage (page)#程序main函数Print u ' start crawl page: 'Raw_input (" ") Myspider = Html_spider () myspider.startspider ()

Python first program crawler