Python Crawler prep Knowledge

Source: Internet
Author: User

1.http Programming Knowledge

    • Working mode of client and server in HTTP

Client and server establish a reliable TCP link (this link is a long-time, time-out break policy in HTTP1.1)

Client communicates with server via socket, sends request and accepts response

The HTTP protocol is stateless, meaning that each request is independent of each other, and neither the client nor the server logs the customer's behavior.

The client adds headers to the HTTP request to tell the server what he requested, the acceptable format

    • Common request methods are get and post

Get:client Request a file

Post:client sending data for server processing

For example, enter http://www.baidu.com/

The resulting request headers is:

Get: The request way/represents the root directory http/1.1 represents the protocol version adopted

Host: Requesting hosts

Connection: Keep a long connection,

Cache-control: Cache-related

User-agent: Tell the server my client's identity, including browser version, etc.

Accept: Supported content types, sequencing indicates the order in which the browser is loaded sequentially

Accept-encoding: Allows the server to compress the transmitted content in a few compressed formats

Accept-language: The language in which the returned information is displayed

Accept-charset: A set of character encodings supported by the browser

Cookies: Cache-related

Reference blog:

http://technique-digest.iteye.com/blog/1174581

Http://www.cnblogs.com/ShaYeBlog/archive/2012/09/11/2680485.html

http://blog.csdn.net/bingjing12345/article/details/9819731

2. URLLIB2 Related content

class Urllib2. Request (url[, data][, headers][, origin_req_host][, unverifiable])

URL: should be a string

Data: is a urllib.urlencode () encoded post-encoded string

Headers: Used to spoof User_agent, to spoof access from a script visit disguised as a browser.

Example code:

Import Urllib

Import Urllib2

url = ' http://www.someserver.com/cgi-bin/register.cgi '

User_agent = ' mozilla/4.0 (compatible; MSIE 5.5; Windows NT) '

Values = {' name ': ' Why ',

' Location ': ' SDU ',

' Language ': ' Python '}

headers = {' User-agent ': user_agent}

data = Urllib.urlencode (values)

req = Urllib2. Request (URL, data, headers)

Response = Urllib2.urlopen (req)

The_page = Response.read ()

Reference Blog: http://blog.csdn.net/pleasecallmewhy/article/details/8923067

3. Save the following code in HTML format, open it with the corresponding browser, and get the version information of the browser.

The user_agent of Sogou browser

Baidu Browser's User_agent

Google Chorme's User_agent

Python Crawler prep Knowledge

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.