1.http Programming Knowledge
- Working mode of client and server in HTTP
Client and server establish a reliable TCP link (this link is a long-time, time-out break policy in HTTP1.1)
Client communicates with server via socket, sends request and accepts response
The HTTP protocol is stateless, meaning that each request is independent of each other, and neither the client nor the server logs the customer's behavior.
The client adds headers to the HTTP request to tell the server what he requested, the acceptable format
- Common request methods are get and post
Get:client Request a file
Post:client sending data for server processing
For example, enter http://www.baidu.com/
The resulting request headers is:
Get: The request way/represents the root directory http/1.1 represents the protocol version adopted
Host: Requesting hosts
Connection: Keep a long connection,
Cache-control: Cache-related
User-agent: Tell the server my client's identity, including browser version, etc.
Accept: Supported content types, sequencing indicates the order in which the browser is loaded sequentially
Accept-encoding: Allows the server to compress the transmitted content in a few compressed formats
Accept-language: The language in which the returned information is displayed
Accept-charset: A set of character encodings supported by the browser
Cookies: Cache-related
Reference blog:
http://technique-digest.iteye.com/blog/1174581
Http://www.cnblogs.com/ShaYeBlog/archive/2012/09/11/2680485.html
http://blog.csdn.net/bingjing12345/article/details/9819731
2. URLLIB2 Related content
class Urllib2. Request (url[, data][, headers][, origin_req_host][, unverifiable])
URL: should be a string
Data: is a urllib.urlencode () encoded post-encoded string
Headers: Used to spoof User_agent, to spoof access from a script visit disguised as a browser.
Example code:
|
Import Urllib
Import Urllib2
url = ' http://www.someserver.com/cgi-bin/register.cgi '
User_agent = ' mozilla/4.0 (compatible; MSIE 5.5; Windows NT) '
Values = {' name ': ' Why ',
' Location ': ' SDU ',
' Language ': ' Python '}
headers = {' User-agent ': user_agent}
data = Urllib.urlencode (values)
req = Urllib2. Request (URL, data, headers)
Response = Urllib2.urlopen (req)
The_page = Response.read () |
Reference Blog: http://blog.csdn.net/pleasecallmewhy/article/details/8923067
3. Save the following code in HTML format, open it with the corresponding browser, and get the version information of the browser.
The user_agent of Sogou browser
Baidu Browser's User_agent
Google Chorme's User_agent
Python Crawler prep Knowledge