Pylibcurl HTTPS search engine network data capture small example, 302moved?google search engine does not let you catch search results?? OK, this article solves the problem

Premise: Operating Platform-win7

First of all, you have Python, and I installed python2.7.9.

Two. Second, you have to install Pylibcurl, installation method: http://pycurl.sourceforge.net/

Three. Finally, you have to write a test case test.py: (Of course, you can see from the code that your computer has an e-drive, otherwise change the code, and then I crawl the data is Google test data)

#! /usr/bin/env python#-*-coding:utf-8-*-# vi:ts=4:etimport sysimport pycurlclass Test:    def __init__ (self):        Self.contents = "    def body_callback (self, buf):        self.contents = self.contents + buf        f = open (' e:\\tmp\\ Tmp.html ', ' W ')        print F        f.write (self.contents) sys.stderr.write ("Testing%s\n"% pycurl.version) T = Test () c = Pycurl. Curl () c.setopt (C.url, ' Https://www.google.com/search?q=testx ') c.setopt (C.useragent, "mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.1; trident/5.0) in my heart of heart. ") C.setopt (C.header, True) c.setopt (C.referer, "Https://www.google.com/search?q=testx") c.setopt (C.cookiefile, "./ COOKIE.txt ") c.setopt (C.cookiejar,"./cookie.txt ") c.setopt (C.writefunction, T.body_callback) c.perform () C.close () Print (t.contents)

Extended:

Do you normally initiate HTTP requests Google will tell you "302 Moved", OK, a closer look at this code, will also solve your problem

Reference:

Http://superuser.com/questions/482470/google-302-moved-in-firefox

Http://stackoverflow.com/questions/22570970/php-search-by-image-google-curl-return-302-moved

Pylibcurl HTTPS search engine network  small example, 302moved?google search engine does not let you catch search results?? OK, this article solves the problem