Learn Python 11 Python crawlers and python Crawlers

Learn Python 11 Python crawlers and python Crawlers

After a few days of learning and trying to get a little bit of experience with python crawlers, we gradually find that they have many commonalities and always need to get a series of links to read Web code, obtain the required content and repeat the above work. When we are more and more skilled, we will try to summarize the common characteristics of crawlers and try to write a helper class to avoid repetitive work.

Reference: Tips for using python crawlers to capture websites zz

1. Visit the website # The simplest way to get the webpage code

1 import urllib22 response = urllib2.urlopen("http://www.xx.com")3 print response.read()

2. disguise as a browser (User-Agent, Referer, etc.) # It is better to disguise as a browser to prevent access from being blocked by the server.

1 headers = {2     'User-Agent': 'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)',3     'Referer':'http://www.xx.com/xx',4     'Accept':'application/javascript, */*;q=0.8'5 }6 response = urllib2.Request(url = "http://www.xx.com",data = None,headers = headers)

3. Post Data Transcoding

1 import urllib,urllib22 values = {3     'username':'xxx',4     'password':'xxx',5     'key':'xxx'6 }7 postdata = urllib.urlencode(values)8 response = urllib2.Request(url,data = postdata)

4. Cookies

1 import urllib2,cookielib2 cookie_handler = urllib2.HTTPCookieProcessor(cookielib.CookieJar())3 opener = urllib2.build_opener(cookie_handler)4 urllib2.install_opener(opener)5 response = urllib2.urlopen(url)

5. Proxy Server # The results of repeated accesses to the same website are blocked by ip addresses or limited the number of visits

1 import urllib22 proxy_handler = urllib2.ProxyHandler({"http" : '42.121.6.80:8080'})3 opener = urllib2.build_opener(proxy_handler)4 urllib2.install_opener(opener)5 response = urllib2.urlopen(url)

Q: What if I want to use cookies and proxies together?

A: urllib2.build _ opener can include multiple parameters, such as BaseHandler, ProxyHandler, HTTPHandler, FileHandler, FTPHandler, CacheFTPHandler, etc.

6. gzip # gzip compression is now widely supported. By default, we can obtain the compressed webpage, which greatly improves the efficiency of crawling the webpage and reduces the bandwidth load.

1 import urllib2,zlib2 req = urllib2.Request(url)3 req.add_header('Accept-encoding', 'gzip')4 response = urllib2.urlopen(req, timeout=120)5 html = response.read()6 gzipped = response.headers.get('Content-Encoding')7 if gzipped:8     html = zlib.decompress(html, 16+zlib.MAX_WBITS)

7. Others

Set the thread stack size: the stack size significantly affects python memory usage. The method is as follows:

 1 from threading import stack_size 2 stack_size(32768*16)

Set timeout

1 import socket2 socket. setdefatimetimeout (10) # Set the connection timeout after 10 seconds

Retry upon failure

 1 def get(self,req,retries=3): 2     try: 3         response = self.opener.open(req) 4         data = response.read() 5     except Exception , what: 6         print what,req 7         if retries>0: 8             return self.get(req,retries-1) 9         else:10             print 'GET Failed',req11             return ''12     return data

Based on the above content, we can write our own helper class for ease of configuration and repetitive work:

1 #-*-coding: UTF-8-*-2 import cookielib, urllib, urllib2, socket 3 import zlib, StringIO 4 class HttpClient: 5 _ cookie = cookielib. cookieJar () 6 # proxy settings, add when needed (later set as multi-proxy switchover) 7 #__ proxy_handler = urllib2.ProxyHandler ({"http": '42. 121.6.80: 8080 '}) 8 _ req = urllib2.build _ opener (urllib2.HTTPCookieProcessor (_ cookie) # ,__ proxy_handler) 9 _ req. addheaders = [10 ('accept', 'application/javascript, */*; q = 000000'), 11 ('user-agent', 'mozilla/0.8 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0) ') 12] 13 urllib2.install _ opener (_ req) 14 15 def Get (self, url, refer = None, retries = 3): 16 try: 17 req = urllib2.Request (url) 18 req. add_header ('Accept-encoding ', 'gzip') 19 if not (refer is None): 20 req. add_header ('Referer', refer) 21 response = urllib2.urlopen (req, timeout = 120) 22 html = response. read () 23 gzipped = response. headers. get ('content-encoding') 24 if gzipped: 25 html = zlib. decompress (html, 16 + zlib. MAX_WBITS) 26 return html27 failed t Exception, what: 28 print what29 if retries> 0: 30 return self. get (url, refer, retries-1) 31 else: 32 print "Get Failed", url33 return ''34 # response t urllib2.HTTPError, e: 35 # return e. read () 36 # handle T socket. timeout, e: 37 # return ''38 # response t socket. error, e: 39 # return ''40 41 def Post (self, url, data, refer = None): 42 try: 43 req = urllib2.Request (url, urllib. urlencode (data) 44 # req = urllib2.Request (url, data) 45 if not (refer is None): 46 req. add_header ('Referer', refer) 47 return urllib2.urlopen (req, timeout = 120 ). read () 48 TB t urllib2.HTTPError, e: 49 return e. read () 50 TB t socket. timeout, e: 51 return ''52 blocks t socket. error, e: 53 return ''54 55 def Download (self, url, file): 56 output = open (file, 'wb ') 57 output. write (urllib2.urlopen (url ). read () 58 output. close () 59 60 def getCookie (self, key): 61 for c in self. _ cookie: 62 if c. name = key: 63 return c. value64 return ''65 66 def setCookie (self, key, val, domain): 67 ck = cookielib. cookie (version = 0, name = key, value = val, port = None, port_specified = False, domain = domain, domain_specified = False, domain_initial_dot = False, path = '/', path_specified = True, secure = False, expires = None, discard = True, comment = None, comment_url = None, rest = {'httponly ': None}, rfc2109 = False) 68 self. _ cookie. set_cookie (ck)HttpClient

As for multithreading, refer to the code found on the Internet. Concurrency is also supported...

1 from threading import Thread 2 from Queue import Queue 3 from time import sleep 4 # q is the task Queue 5 # NUM is the total number of concurrent threads 6 # How many JOBS are there 7 q = Queue () 8 NUM = 2 9 JOBS = 1010 # specific processing function, responsible for processing a single task 11 def do_somthing_using (arguments): 12 print arguments13 # This is a working process, constantly retrieve data from the queue and process 14 def working (): 15 while True: 16 arguments = q. get () 17 do_somthing_using (arguments) 18 sleep (1) 19 q. task_done () 20 # fork NUM threads waiting queue 21 for I in range (NUM): 22 t = Thread (target = working) 23 t. setDaemon (True) 24 t. start () 25 # put JOBS into the queue 26 for I in range (JOBS): 27 q. put (I) 28 # Wait for all JOBS to complete 29 q. join ()ThreadDemo

Crawlers rely on a piece of content. A more in-depth crawler framework and html library are now available. Let me consider whether the next content is pygame or django!

Crawler demo github address (just learned to play git): http://git.oschina.net/tabei/Python_spider