[Python] together to write a Python crawler tool class whyspider

I'm glad to have met a like-minded person in GITCAFE who sent the merge request pai_^. I hope more people can participate.

I wrote a lot of small examples of simple Python crawlers. Today I suddenly want to create an open-source toolkit and share the source code with you at gitcafe.

Project Source Address:

Https://gitcafe.com/callmewhy/whyspider

Today I wrote the simplest functions: GET and POST methods.

Other features will continue to be improved on gitcafe,

The next step is to complete the common features of Regular Expression matching encapsulation and header simulation.

Because I have been learning Android recently, the update progress may be a little slower =. =

If you have any ideas, for example, if you want to add some functions to this project, please comment or submit ticket ~ in gitcafe ~

Version 0.2 has just been completed:

#-*-Coding: UTF-8-*-# --------------------------------------- # program: whyspider. py # version: 0.2 # Author: why # Date: # language: Python 2.7.5 # version list: #0.1: add GET and POST #0.2: add the simulation header function # ------------------------------------- import urllib import urllib2import cookielibimport reimport stringclass WhySpider: # initialize crawler def _ init _ (self): self. cookie_jar = cookielib. cookieJar () self. opener = urllib2.build _ opener (urllib2.HTTPCookieProcessor (self. cookie_jar) self. headers = {'user-agent': 'mozilla/5.0 (Windows NT 6.3; WOW64; rv: 28.0) gecko/20100101 Firefox/28.0 '} # Send GET request def send_get (self, get_url): result = "" try: my_request = urllib2.Request (url = get_url, headers = self. headers) result = self. opener. open (my_request ). read () failed t Exception, e: print "Exception:", e return result # Send POST request def send_post (self, post_url, post_data): result = "" try: my_request = urllib2.Request (url = post_url, data = post_data, headers = self. headers) result = self. opener. open (my_request ). read () failed t Exception, e: print "Exception:", e return result # simulate computer def set_computer (self): user_agent = 'mozilla/5.0 (Windows NT 6.3; WOW64; rv: 28.0) Gecko/20100101 Firefox/28.0 'self. headers = {'user-agent': user_agent} # simulate mobile phone def set_mobile (self): user_agent = 'mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) appleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A403 Safari/8536.25 'self. headers = {'user-agent': user_agent}

The call method is simple:

#-*-Coding: UTF-8-*-import whyspider # initialize the crawler object my_spider = whyspider. WhySpider () # simulate the GET operation print my_spider.send_get ('HTTP: // 3.apitool.sinaapp.com /? Why = GetString2333 ') # simulate the POST operation print my_spider.send_post ('HTTP: // batch) # simulate the GET operation print my_spider.send_get ('HTTP: // www.baidu.com /') # Switch to the mobile phone mode my_spider.set_mobile () # simulate the GET operation print my_spider.send_get ('HTTP: // www.baidu.com /')