HTTP protocol request actual Combat--get Request example crawl Baidu search keyword corresponding to the page information

1 through analysis search keyword corresponding URL found field name and content are in the URL can be simplified to www.baidu.com/s?wd= search keywords

According to the analysis of the law can be constructed by the use of crawling crawler to automatically search for a keyword on Baidu

The code is as follows:

Import urllib.request   Importing Urllib.request
KEYWD = ' high with same '         specified keyword
Keywd_cod = Urllib.request.quote (KEYWD) encodes the keyword
url = ' https://www.baidu.com/s?wd= ' input URL
Url_all = URL +keywd_cod Constructs the final crawl URL
Req =urllib.request.request (URL) Constructs a request object with Urllib.request.Request and assigns it to a variable req

data = Urllib.request.urlopen (req). Read () Open the corresponding Request object by Url.request.urlopen ()
Fandle = open (' d:/siq/part4/10.html ', ' WB ')
Fandle.write (data)
Writes the crawled content to an HTML file
Fandle.close () End



Summary of Ideas:
1 build the corresponding URL address, which contains information such as the field name and field contents of the GET request and the URL address satisfies the GET request Format "http://url?" Field Name 1 = Field contents 1& Field Name 2 = Field Content 2 "
2 Build the Request object with the corresponding URL as the parameter
3 Open the built Request object via Urlopen ()
4 Read content and save

HTTP protocol request actual Combat--get Request example crawl Baidu search keyword corresponding to the page information