Python Distributed crawler principle __python

Reprint please indicate the source: http://blog.csdn.net/yiliumu/article/details/21335245

First, let's take a look at how to get the content of a Web page if it's a normal behavior of people. (1) Open the browser, enter the URL, open the source page
(2) Select the content we want, including title, author, abstract, body and other information
(3) stored in the hard drive

The above three processes, mapping to the technical level, in fact, is: Network request, crawl structured data, data storage.
We use Python to write a simple program that implements the simple crawl feature above.
[Python]  View Plain  copy   #!/usr/bin/python   #-*- coding: utf-8 -*-   ' "'   created on 2014-03-16    @author: kris  '    import  urllib2, re, cookielib      def httpcrawler (URL):         ' '        @summary:  Web crawl        '         content = httprequest (URL)        title  = parsehtml (content)        savedata (title)       def  httprequest (URL):        '        @summary:  Network request        "'          try:            ret = None            sockfile = none           request =  urllib2. Request (URL)            request.add_header (' user-agent ',  ' mozilla/4.0  (compatible; msie 6.0; windows nt 5.2; sv1; .net clr  1.1.4322)            request.add_header (' Pragma '),   ' No-cache ')            opener = urllib2.build_ Opener ()            sockfile = opener.open (Request)            ret = sockfile.read ()         finally:           if SockFile:                sockfile.close ()    &Nbsp;              return ret      def parsehtml (HTML):        "'       @ summary:  Crawl structured data        '        content =  none       pattern =  ' <title> ([^<]*?) </title> '        temp = re.findall (pattern, html)        if temp:           content  = temp[0]              return content           def savedata (data):        ' '        @summary:  data storage        ' '         f =&nbsP;open (' Test ',  ' WB ')