Summary of common Python crawler tips

Python has been around for more than a year, Python applications most of the scene is the Web rapid development, crawler, automated operations: Write a simple website, write automatic post script, write send and receive mail script, write a simple verification code identification script.

Crawler in the development process also has a lot of reuse process, here to summarize, in the future can save some things.

1, basic crawl web get method

Import Urllib2url = "http://www.baidu.com" response = Urllib2.urlopen (URL) print Response.read ()

Post method

Import Urllibimport Urllib2url = "http://abcde.com" form = {' name ': ' abc ', ' Password ': ' 1234 '}form_data = Urllib.urlencode (form) Request = Urllib2. Request (url,form_data) response = Urllib2.urlopen (request) print response.read ()

2. Using Proxy IP

In the development of the crawler will often encounter IP is blocked, then need to use proxy IP;

There are Proxyhandler classes in the URLLIB2 package that allow you to set up a proxy to access the Web page, as in the following code snippet:

Import Urllib2proxy = Urllib2. Proxyhandler ({' http ': ' 127.0.0.1:8087 '}) opener = Urllib2.build_opener (proxy) Urllib2.install_opener (opener) Response = Urllib2.urlopen (' http://www.baidu.com ') print response.read ()

3. Cookie Processing

Cookies are data (usually encrypted) that are stored on the user's local terminal in order to identify the user, perform session tracking, and Python provides a cookielib module for processing cookies, The primary role of the Cookielib module is to provide objects that store cookies to facilitate access to Internet resources in conjunction with the URLLIB2 module.

Code snippet:

Import Urllib2, cookielibcookie_support= urllib2. Httpcookieprocessor (Cookielib. Cookiejar ()) opener = Urllib2.build_opener (Cookie_support) Urllib2.install_opener (opener) content = Urllib2.urlopen (' Http://XXXX '). Read ()

The key is Cookiejar (), which manages the HTTP cookie value, stores the cookie generated by the HTTP request, and adds a cookie to the outgoing HTTP request. The entire cookie is stored in memory, and the cookie is lost after the Cookiejar instance is garbage collected, and all processes do not need to be operated separately.

4. Disguised as a browser

Some websites resent the crawler's visit, so they refuse to request the crawler. So with URLLIB2 direct access to the site will often appear HTTP Error 403:forbidden situation

Special attention should be paid to some of the headers, which are checked against these headers by the server side.

1.user-agent Some servers or proxies check this value to determine whether a browser-initiated Request

2.content-type when using the REST interface, the Server checks the value to determine how the content in the HTTP Body should be parsed.

This can be done by modifying the header in the HTTP package as follows:

Import urllib2headers = {    ' user-agent ': ' mozilla/5.0 (Windows; U Windows NT 6.1; En-us; rv:1.9.1.6) gecko/20091201 firefox/3.5.6 '}request = urllib2. Request (    url = ' http://my.oschina.net/jhao104/blog?catalog=3463517 ',    headers = headers) Print Urllib2.urlopen (Request). Read ()

5. Page parsing

For page parsing the most powerful of course is the regular expression, this for different sites different users are not the same, do not have too much to explain, with two more good URLs:

Getting Started with regular expressions: http://www.cnblogs.com/huxi/archive/2010/07/04/1771073.html

Online Regular expression test: http://tool.oschina.net/regex/

The second is the analysis of the library, commonly used there are two lxml and BeautifulSoup, for the use of the two introduced two relatively good site:

lxml:http://my.oschina.net/jhao104/blog/639448

Beautifulsoup:http://cuiqingcai.com/1319.html

For these two libraries, my evaluation is, are html/xml processing library, BeautifulSoup pure Python implementation, low efficiency, but functional, such as can be used through the results of the search for an HTML node source code; LXMLC language encoding, efficient, XPath support

6, the Verification Code processing

For some simple verification codes, simple identification can be done. I have only done some simple verification code identification. But some anti-human verification code, such as 12306, can be manually coded by the code platform, of course, this is to pay.

7, gzip compression

Have encountered some Web pages, no matter how the transcoding is a mess of garbled. Haha, that means you don't know. Many Web services have the ability to send compressed data, which can reduce the amount of data transmitted over a network line by more than 60%. This is especially true for XML Web services, because the compression rate of XML data can be high.

However, the general server does not send compressed data for you unless you tell the server that you can process the compressed data.

So you need to modify the code like this:

Import urllib2, httplibrequest = Urllib2. Request (' http://xxxx.com ') request.add_header (' accept-encoding ', ' gzip ')        1opener = Urllib2.build_opener () F = Opener.open (Request)

This is the key: Create a Request object, add a accept-encoding header to tell the server you can accept gzip compressed data

And then unzip the data:

Import Stringioimport Gzipcompresseddata = F.read () Compressedstream = Stringio.stringio (compresseddata) Gzipper = gzip. Gzipfile (fileobj=compressedstream) print gzipper.read ()

8, multi-threaded concurrent crawl

Single thread too slow, you need to multi-threading, here to a simple thread pool template This program is simply printed 1-10, but can be seen to be concurrent.

Although Python's multithreading is very chicken, but for crawler this kind of network frequent type, still can improve the efficiency to a certain extent.

From threading import threadfrom queue import queuefrom time import sleep# Q is the task queue #num is the total number of concurrent threads #jobs is how many tasks q = queue () NUM = 2J OBS = 10# specific processing function, responsible for processing a single task Def do_somthing_using (arguments):    print arguments# This is a worker process, responsible for constantly fetching data from the queue and processing def working (): While    True:        arguments = Q.get ()        do_somthing_using (arguments)        sleep (1)        Q.task_done () #fork Num threads wait for a queue for I in range (NUM):    t = Thread (target=working)    T.setdaemon (True)    T.start () #把JOBS排入队列for I in Range (JOBS):    q.put (i) #等待所有JOBS完成q. Join ()

Summary of common Python crawler tips