python-Crawler-requests Library term Post Login

The requests library is powerful, supports HTTP connection retention and connection pooling, supports the use of cookies to maintain sessions, supports file uploads, supports automatic encoding of response content, supports internationalized URLs and automatic post data encoding.

You can send a GET request with no parameters, or you can send a GET request with parameters, modify headers, and so on.

The main exhibition here sends a POST request, which is passed through the data parameter.

For example: Login to Chinaunix website, log in by login name, password.

By viewing the source code of the Chinaunix website, you can see the URL of the login page:

Http://bbs.chinaunix.net/member.php? Mod=logging&action=login&loginsubmit=yes&loginhash=licac

Different computer login URLs may not be the same, please check the specific page source code.

In order to deal with the site's anti-crawler, you can modify headers to simulate Web login. Specific as follows:

Import requestsconn = requests.session () url = ' Http://bbs.chinaunix.net/member.php?mod=logging&action=login &LOGINSUBMIT=YES&LOGINHASH=LICAC ' postdata = {    ' username ': ' * * * ',    ' password ': ' * * * '}headers = {            ' User-agent ': ' mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_0) applewebkit/537.36 (khtml, like Gecko) chrome/55.0.2883.95 safari/537.36 '}rep = conn.post (URL, dat A=postdata,headers=headers) with open (' 1.html ', ' WB ') as F:    F.write (rep.content)

The login name and password in the code are replaced by your own early registration, otherwise you won't be logged in.

The requests library automatically saves cookies without having to set them up separately.

Import requestsconn = requests.session () url = ' Http://bbs.chinaunix.net/member.php?mod=logging&action=login &LOGINSUBMIT=YES&LOGINHASH=LICAC ' postdata = {    ' username ': ' zhaoxn04 ',    ' password ': ' wobugaosuni2004 '}headers = {            ' user-agent ': ' mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_0) applewebkit/537.36 (khtml, like Gecko) chrome/55.0.2883.95 safari/537.36 '}rep = conn.post (URL, dat A=postdata,headers=headers) with open (' 1.html ', ' WB ') as F:    F.write (rep.content) url1 = ' http://bbs.chinaunix.net/ thread-4246512-1-1.html ' REP1 = Conn.get (URL1, Headers=headers) with open (' 2.html ', ' WB ') as F:    F.write ( Rep1.content)

python-Crawler-requests Library term Post Login