python: 爬蟲利器requests

標籤：python 爬蟲

requests並不是系統內建的模組，他是第三方庫，需要安裝才能使用

requests庫使用方式

閑話少說，來，讓我們上代碼：
簡單的看一下效果：

import requestsrequests = requests.session()headers = {    ‘User-Agent‘:‘Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0‘}url = "http://httpbin.org"response = requests.get(url, headers=headers, timeout=None)print(response.text)print(response.cookies)print(response.content)print(response.content.decode("utf-8"))print(respone.json())

基本的post請求：

data = {    "name":"zhaofan",    "age":23}response = requests.post("http://httpbin.org/post",data=data)print(response.text)

對於無效的網站認證要求方法：

import requestsfrom requests.packages import urllib3urllib3.disable_warnings()response = requests.get("https://www.12306.cn",verify=False)print(response.status_code)

代理設定：

import requestsproxies= {    "http":"http://127.0.0.1:9999",    "https":"http://127.0.0.1:8888"}response  = requests.get("https://www.baidu.com",proxies=proxies)print(response.text)如果代理需要設定賬戶名和密碼,只需要將字典更改為如下：proxies = {"http":"http://user:[email protected]:9999"}如果你的代理是通過sokces這種方式則需要pip install "requests[socks]"proxies= {"http":"socks5://127.0.0.1:9999","https":"sockes5://127.0.0.1:8888"}

逾時設定

通過timeout參數可以設定逾時的時間

沒有逾時時間，一直等待timeout=None

異常捕捉：

import requestsfrom requests.exceptions import ReadTimeout,ConnectionError,RequestExceptiontry:    response = requests.get("http://httpbin.org/get",timout=0.1)    print(response.status_code)except ReadTimeout:    print("timeout")except ConnectionError:    print("connection Error")except RequestException:    print("error")

python: 爬蟲利器requests