Python crawler (2)

I. Encoding and decoding of URLs　

The Urllib and urllib2 included in Python2 are the modules that accept URL requests. But in the Python3, there is no urllib2. In fact, the function of URLLIB2 can be implemented with Urllib in Python3.

Usually coding work, we use the Urllib.parse.urlencode () function, to help us to convert key:value (like a Python dictionary) such as a key value pair into a string such as "Key=value", decoding work can be used unquote () function to implement.

A generic HTTP request submits data that needs to be encoded into a URL-encoded format, then as part of the URL, or as a parameter to the request object.

Second, write a simple reptile program

First we use VIM to create a new Python file in Cmder, our goal is to download the home page of the Shanghai chain.

Web site can be opened in the browser analysis of the URL

The first page URL is: https://sh.lianjia.com/ershoufang/

The second page address is: https://sh.lianjia.com/ershoufang/pg2/

The third page address is: https://sh.lianjia.com/ershoufang/pg3/

............

When we add "pg1/" behind the https://sh.lianjia.com/ershoufang/, we find that we can also get the first page. The crawler can be written according to this rule.

1  fromUrllibImportRequest2 3 4 defHtmlspider (url,startpage,endpage):5     6     #role: Responsible for processing URLs, assigning each URL to send requests7     8      forPageinchRange (startpage,endpage+1):9Filename="Section"+ str (page) +"page. html"Ten      One      A         #combine as full URL -Fullurl=url +Str (page) -  the         #call LoadPage () to send a request to get an HTML page -Html=loadPage (fullurl,filename) -  -         #writes the obtained HTML page to the local disk file + writepage (html,filename) -  +  A  at defloadPage (fullurl,filename): -    -Response=Request.urlopen (FullUrl) -     returnResponse.read () -  -  in  - defwritepage (html,filename): to     """ + Save the server's response file to a local disk - where filename is the local Disk file name the     """ *     Print("is storing"+filename) $with open (filename,'WB') as F:Panax Notoginseng f.write (HTML) -  the  +     Print("--"*30) A  the  + if __name__=="__main__": -     #enter the starting and ending pages you want to download, and note the conversion to int type $Startpage=int (Input ("Please enter the start page:")) $Endpage=int (Input ("Please enter the termination page:")) -  -Url="https://sh.lianjia.com/ershoufang/" the  - Htmlspider (url,startpage,endpage)Wuyi  the     Print("Download Complete! ")

Of course, this is just a very simple download page of the small crawler. But we can see the basic process of crawler work.

Iii. about get requests and post methods

Get requests are generally used for us to get data to the server, if we use Baidu search a keyword, we can see in the request section, http://www.baidu.com/? After a long string, which is to include the key words we want to query, So we can use the default get method to send the request. The Get method is accessed directly as a link, and all parameters are included in the link.

When sending a POST request, pay special attention to some of the properties of headers: content-length (Form data length), X-requested-with:xmlhttprequest (Ajax asynchronous request), and so on. Post does not display all parameters on the URL, and the server uses Requeste.form to get the submitted data.

"For a crawler engineer, we have to focus on the source of the crawler."

　　

Python crawler (2)