Python crawlers web pages and python crawlers web pages

Python crawlers web pages and python crawlers web pages

Preparation Process

1. webpage capturing process

Prepare an http request-> submit the corresponding request-> obtain the returned response (http response)-> obtain the webpage source code

2. GET or POST

3. Headers (optional)

In some cases, direct capturing is forbidden. In this case, you need to provide a Headers to tell the other party that I am not a robot.

For example:

1 def getHtml(url):2     header={'User-Agent' : 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:14.0) Gecko/20100101 Firefox/14.0.1','Referer' : '******'}3     request=urllib2.Request(url,None,header)4     response=urllib2.urlopen(request)5     text=response.read()6     return text

4. Post Data (optional)

In some cases, You need to log on to a website. In this case, you need to submit the account and password, and use Post Data.

In IE, it is called request body, and chrome becomes Post Data

5. Cookie (optional)

Generally, simulated login involves cookies.

6. Others (proxy, maximum timeout)

Content Analysis

1. For the html source code, call the BeautifulSoup Library

2. Regular Expression

 

 

This article references http://www.crifan.com/summary_about_flow_process_of_fetch_webpage_simulate_login_website_and_some_notice/

It can be said that it is a simplified version. If you want to see it, you can read the original article.

PS: the blogger provides a lot of computer information, which can help a lot after reading. If you are interested, you can check it out.