Use of Python crawler cookies

Cookies, which are data stored on the user's local terminal (usually encrypted) by certain websites in order to identify users and perform session tracking.

For example, some sites need to log in to access a page, before you log in, you want to crawl a page content is not allowed. Then we can use the URLLIB2 library to save our registered cookies, and then crawl the other pages to achieve the goal.

Before that, let's introduce a opener concept.

1.Opener

When you get a URL you use a opener (a urllib2. Openerdirector instances). In front, we are all using the default opener, which is Urlopen. It is a special opener, can be understood as a special example of opener, the incoming parameters are just url,data,timeout.

If we need to use cookies, it is not possible to use this opener, so we need to create more general opener to implement the cookie settings.

2.Cookielib

The primary role of the Cookielib module is to provide objects that store cookies to facilitate access to Internet resources in conjunction with the URLLIB2 module. The Cookielib module is very powerful, and we can use the object of the Cookiejar class of this module to capture cookies and resend them on subsequent connection requests, such as the ability to implement the impersonation login function. The main objects of the module are Cookiejar, Filecookiejar, Mozillacookiejar, Lwpcookiejar.

Their relationship: cookiejar--derived-->filecookiejar--derived-–>mozillacookiejar and Lwpcookiejar

1) Get cookie saved to variable

First, we first use the Cookiejar object to achieve the function of the cookie, stored in the variable, first to feel the

1 #Coding:utf82 3 ImportCookielib4 ImportUrllib25 6 #declaring a Cookiejar object instance to hold a cookie7Cookie =Cookielib. Cookiejar ()8 #creating cookie processors with Httpcookieprocessor objects9Handle =Urllib2. Httpcookieprocessor (Cookie)Ten #build opener with handle OneOpener =Urllib2.build_opener (handle) A #This open method, with the Urllib2 Urlopen method, can be passed to the request -Response = Opener.open ('http://www.baidu.com') -  the  forIinchCookies: -     Print 'Name ='+I.name -     Print 'Value ='+ I.value

Use the above method to save the cookie to a variable, then print out the value in the cookie and run the result as follows

1Name =Baiduid2Value = 6e0127b9536de7ee8a68d8b5ae016cca:fg=13Name =Bidupsid4Value =6e0127b9536de7ee8a68d8b5ae016cca5Name =H_ps_pssid6Value =1465_13550_21110_17001_21672_221587Name =PSTM8Value = 14910373929Name =BdsvrtmTenValue =0 OneName =Bd_home AValue = 0

2) Save cookies to file

In the above method, we save the cookie in the cookie variable, what if we want to save the cookie to a file? At this point, we need to use

Filecookiejar This object, where we use its subclass Mozillacookiejar to save cookies.

1 #Coding:utf82 3 ImportCookielib4 ImportUrllib25 6 #set the Save cookie file in the sibling directory7File_name='Cookie.txt'8 #declaring a Cookiejar object instance to hold a cookie9Cookie =Cookielib. Mozillacookiejar (file_name)Ten #creating cookie processors with Httpcookieprocessor objects OneHandle =Urllib2. Httpcookieprocessor (Cookie) A #build opener with handle -Opener =Urllib2.build_opener (handle) - #This open method, with the Urllib2 Urlopen method, can be passed to the request theResponse = Opener.open ('http://www.baidu.com') -  -Cookie.save (Ignore_discard=true,ignore_expires=true)

The two parameters about the last Save method are described here:

The official explanations are as follows:

Gnore_discard:save even cookies set to is discarded. Ignore_expires:save even cookie that has expiredthe file is overwritten if it already exists

Thus, ignore_discard means that even if the cookie is discarded, it will be saved, ignore_expires means that if the cookie already exists in the file, overwrite the original file, and here we set both to true. After the operation, the cookies will be saved to the Cookie.txt file, and we'll look at the contents as follows

3) Obtain a cookie from the file and access

So we've already saved the cookie to the file, and if you want to use it later, you can use the following method to read the cookie and visit the website and feel

1 #Coding:utf82 ImportUrllib23 ImportCookielib4 #creating an Instance object5Cookie =Cookielib. Mozillacookiejar ()6 #read cookie content from file to variable7Cookie.load ('Cookie.txt', ignore_discard=true,ignore_expires=True)8 #Create request9Request = Urllib2. Request ('http://www.baidu.com')Ten #use the Build_opener method to create a opener OneOpener =Urllib2.build_opener (urllib2. Httpcookieprocessor (cookie)) A  -req =Opener.open (Request) - PrintReq.read ()

Imagine, if our cookie.txt file is stored in a person login Baidu cookie, then we extract the contents of this cookie file, you can use the above method to simulate the person's account login Baidu.

4) Use cookies to simulate website Login

below to my blog garden For example (the account password are fake yo, do not believe you can try to drop), using cookies to achieve the simulation login, and the cookie information saved to a text file, to feel the cookie Dafa it!

1 #Coding:utf82 3 ImportUrllib4 ImportUrllib25 ImportCookielib6 7file_name ='Cookie1.txt'8 #declares a Mozillacookiejar object instance to hold the cookie, and then writes the file9Cookie =Cookielib. Mozillacookiejar (file_name)TenOpener =Urllib2.build_opener (urllib2. Httpcookieprocessor (cookie)) Onedata =Urllib.urlencode ({ A     'username':'username', -     'pwd':'Password', -     }) the #log in to the URL of the blog Park -Login_url ='Https://passport.cnblogs.com/user/signin?ReturnUrl=http%3A%2F%2Fwww.cnblogs.com%2F' - #impersonate the login and save the cookie to the variable -result =Opener.open (login_url,data) + #Save cookies to file -Cookie.save (Ignore_discard=true, ignore_expires=True) + #use cookies to request access to another URL ASelect_url='http://www.cnblogs.com/qianyuliang/p/6656580.html' at #Request Access -result =Opener.open (Select_url) - PrintResult.read ()

The principle of the above procedure is as follows

Create a opener with a cookie, save the logged-in cookie when accessing the URL of the login, and then use this cookie to access other URLs.

Use of Python crawler cookies