012 Python Crawler Project 1

# Python Crawler Project 1
Python Web page request
Requests
POST
GET

Page status Code

1 # -*-coding:utf-8-*- 2  from Import BeautifulSoup 3 Import Requests 4 5 " http://www.baidu.com " 6 unknow = requests.get (URL)7print(type (unknow))8  Print(unknow)

　　　　

Match content by tag

1 #-*-coding:utf-8-*-2  fromBs4ImportBeautifulSoup3 ImportRequests4 5URL ="http://zz.ganji.com/fang1/"6R =requests.get (URL)7Soup = BeautifulSoup (R.text,'lxml')8  forIteminchSoup.find_all ('DD'):9     ifitem['class'] == ['Dd-item','title']:Ten         #print (item) One         Print(item.a.string) A         Print("----------------------------------------------------")

Copying copy selector via browser

　　

1 #-*-coding:utf-8-*-2  fromBs4ImportBeautifulSoup3 ImportRequests4 5URL ="http://zz.ganji.com/fang1/"6R =requests.get (URL)7Soup = BeautifulSoup (R.text,'lxml')8 9 #Price AcquisitionTentitle = Soup.select ('DL > Dd.dd-item.info > Div.price > Span.num') One Print(title) A      -Title2 = Soup.select ('DL > Dd.dd-item.size > Span.first.js-huxing') - Print(Title2)

1 title = Soup.select ('dl > Dd.dd-item.info > Div.price > Span.num') 
    2print(title)3print(type (title[0]))

The type of title or tag tag

SOUP.BODY.DIV.DIV.A Way to get

1 #-*-coding:utf-8-*-2  fromBs4ImportBeautifulSoup3 ImportRequests4 5URL ="http://zz.ganji.com/fang1/"6R =requests.get (URL)7Soup = BeautifulSoup (R.text,'lxml')8 Print(SOUP.BODY.DIV.DIV.A)

1  fromBs4ImportBeautifulSoup2 ImportRequests3 4 defIsdiv (tag):5     returnTag.name = ='Div'6 7URL ="http://zz.ganji.com/fang1/"8R =requests.get (URL)9Soup = BeautifulSoup (R.text,'lxml')Ten  OneValue =Soup.find_all (Isdiv) A Print(value)

Python sends Web requests using a proxy

1 Import Requests    2 " http " " http://10.10.1.10:3128 " " HTTPS " " http://10.10.1.10:1080 " , }    3 requests.get ("http://example.org", Proxies=proxies)  

012 Python Crawler Project 1