Python-based request library, demo Login Csdn Blog

Before the crawler with URLLIB2 to achieve, also used Scrapy crawler frame, this test requests, just start with, really better than urllib2, package better, easy to use a lot.

Installing the Requests Library

The easiest way is to use PIP to install: Pip install requests, if you need to install a specific version, then add the version number: Pip Install requests = = 1.9.7, so it's done.

Quick Start with small examples

Here is one of the simplest examples:

The first line, the introduction of the requests library, which is inevitable.

The second line, through get method to get Baidu home content.

The third line, the return of the response content, output.

It is very simple, so you can send a GET request, similarly, you can also use requests.post,requests.put,requests.options, Requests.head, send the request.

Analog Login Csdn

We need other assistive tools.

Browser: Firefox

Browser plugin: Tamper Data,firebug

We need to tamper data to intercept the request because Chrome does not have a plugin for this feature, so this can only be done using Firefox (except for blocking requests for Chrome, other jobs can use chrome).

Analyzing the logon process

1. Open the login page

We first open the login page of csdn:https://passport.csdn.net/account/login?ref=toolbar, this link, the front part is the login URL, the argument after the question mark, Referer, is where you jump from, maybe a page jump to login, toolbar is my own click on the top navigation bar, and then jump to the login page.

2. Clearing the relevant cookie

In order to eliminate unnecessary disturbances, we first clear out all relevant cookies so that we can analyze which parameters are necessary.

3. Login Process Analysis

After clearing the cookie, we refresh the page,Https://passport.csdn.net/account/login?ref=toolbar, and retrieve the corresponding cookie.

Then we started using tamper data to intercept the request.

We click on start Tamper , fill in the user name and password on the webpage, click "Login", will make a request, then tamper data will intercept this request, ask if we intercept, click Tamper, we can before the request is submitted, View the contents of the request and delete it.

Send a login form in the past, the login was successful.

4. Start the demo login

Knowing the login process, we begin to write the login code.

At this point, the login is successful.

5. Optimization

When you log in successfully, you will ask, how do I know the login success? When you try to crawl the content of http://write.blog.csdn.net/postlist , you will find a 403 error, why?

very simple, The user agent is not modified, we use the default User agent, This is not a normal user, so the site was rejected. We'll just add it:

Something

We all know that the cookie has a validity period, when doing debugging, every modification, it is necessary to simulate login once, so bad, we want to save cookies, so next time we do not need to resend the login request.

The complete code is as follows:

　　

Python-based request library, demo Login Csdn Blog