Python crawler requests+beautifulsoup crawl information Company Overview code example

The first time to write a more complete crawler, self-feeling very poor Ah, code low, inefficient, and did not save to local files or databases, forcing the use of a wave of multithreading caused the data order changed ...

Stick here, warning.

#-*-coding:utf-8-*-"""Created on Wed Jul 21:41:34 2018@author:brave-manblog:http://www.cnblogs.com/zrmw/"""ImportRequests fromBs4ImportBeautifulSoupImportJSON fromThreadingImportThread
# get the full name of the listed company, English names, addresses, legal representatives (you can also get any company information you want to obtain)defgetdetails (URL): Headers= {"user-agent":"mozilla/5.0 (Windows NT 6.1; WOW64; rv:6.0) gecko/20100101 firefox/6.0"} res= Requests.get ("{}". Format (URL), headers =headers) res.encoding="GBK"Soup= BeautifulSoup (Res.text,"Html.parser") Details= {"Code": Soup.select (". Table") [0].td.text.lstrip ("Stock Code:") [: 6],                "Entire_name": Soup.select (". Zx_data2") [0].text.strip ("\ r \ n"),                "English_name": Soup.select (". Zx_data2") [1].text.strip ("\ r \ n"),                "Address": Soup.select (". Zx_data2") [2].text.strip ("\ r \ n"),                "legal_representative": Soup.select (". Zx_data2") [4].text.strip ("\ r \ n")}    #This translates details into a JSON string format for post-storage processingJD =json.dumps (Details) jd1=Json.loads (JD)Print(JD1)
# This function is used to obtain the stock code of the listed company defGetCode (): Headers= {"user-agent":"mozilla/5.0 (Windows NT 6.1; WOW64; rv:6.0) gecko/20100101 firefox/6.0"} res= Requests.get ("http://www.cninfo.com.cn/cninfo-new/information/companylist", headers =headers) res.encoding="gb1232"Soup= BeautifulSoup (Res.text,"Html.parser")#Print (Soup.select (". Company-list"))L =[] L1=[] L2=[] L3=[] L4= []     forIinchSoup.select (". Company-list") [0].find_all ("a"): Code= I.text[:6] L1.append (code) forIinchSoup.select (". Company-list") [1].find_all ("a"): Code= I.text[:6] L2.append (code) forIinchSoup.select (". Company-list") [2].find_all ("a"): Code= I.text[:6] L3.append (code) forIinchSoup.select (". Company-list") [3].find_all ("a"): Code= I.text[:6] L4.append (code) L=[L1, L2, L3, L4]Print(l[0])returngetAll (L)defGetAll (L):defT1 (L): forIinchl[0]: Url_sszb="http://www.cninfo.com.cn/information/brief/szmb{}.html". Format (i) getdetails (URL_SSZB)defT2 (L): forIinchL[1]: Url_zxqyb="http://www.cninfo.com.cn/information/brief/szsme{}.html". Format (i) getdetails (URL_ZXQYB)defT3 (L): forIinchL[2]: Url_cyb="http://www.cninfo.com.cn/information/brief/szcn{}.html". Format (i) getdetails (URL_CYB)deft4 (L): forIinchL[3]: Url_hszb="http://www.cninfo.com.cn/information/brief/shmb{}.html". Format (i) getdetails (URL_HSZB)#TT1 = Thread (target = t1, args = (L,))#TT2 = Thread (target = t2, args = (L,))#tt3 = Thread (target = t3, args = (L,))#TT4 = Thread (target = T4, args = (L,))#    #Tt1.start ()#Tt2.start ()#Tt3.start ()#Tt4.start ()#    #Tt1.join ()#Tt2.join ()#Tt3.join ()#Tt4.join ()T1 (l) T2 ( L) T3 (l) T4 (L)if __name__=="__main__": GetCode ()

does not consider the actual production of the unexpected situation, such as the delay in the speed of network problems.

The speed is really slow, there is time to share to everyone selenium + browser crawl information of the method code. Good Night ~

Python crawler requests+beautifulsoup crawl information Company Overview code example