Python crawler crawl room World data-into MySQL database

Python crawl room World City data

With the rise of the internet era, technology is changing rapidly, mastering a new technology for career development has a profound significance, do the first demo, will be in the crawler and data analysis to facilitate deeper research, I will not do detailed documentation, where there is insufficient place, I hope that Daniel guidance to explain. Nonsense not much to say, on the code.

The skills you need:

(1) familiar with the front-end knowledge will debug the browser

(2) Skilled Python basic knowledge, familiar with some of the commonly used libraries to master

(3) Master the General relational database

Import requests as Reqimport Timeimport pandas as Pdfrom BS4 import beautifulsoupfrom sqlalchemy import Create_enginegloba L infodef gethouseinfo (URL): info = {}soup = BeautifulSoup (req.get (URL). Text, "Html.parser") Resinfo = Soup.select (". Tab-cont-right. Trl-item1 ") # Get the house, floor, unit price, direction, floors, decoration situation for the RE in resinfo:tmp = Re.text.strip (). Split (" \ n ") name = Tmp[1].  Strip () if ("Facing" in name): Name = Name.strip ("Door") if ("Floor" in name): Name = name[0:2]if ("upper layer" in name): Name = "Floor" if ("Decoration degree" in Name): name = "renovated" info[name] = Tmp[0].strip () Xiaoqu = Soup.select (". Rcont. Blue") [0].text.strip () info["cell name"] = Xiaoquzongjia = Soup.select (". Tab-cont-right. Trl-item") info["Total price"] = zongjia[0].textreturn Infodomain = "http:// esf.anyang.fang.com/"city =" house/"#获取总页数def gettotalpage (): res = Req.get (domain+city+" i31 ") soup = BeautifulSoup ( Res.text, "Html.parser") EndPage = Soup.select (". Page_al a"). Pop () [' href ']pagenum = Endpage.strip ("/"). Split ("/") [1]. Strip ("i3") print ("Loading ... Total "+pagenum+" page data ...) return pagenum#Pagination Crawl Data def pagefun (i):p ageurl = domain + City + "i3" +iprint (pageurl+ "Loading ... section "+i+" page data ...) res = Req.get (pageurl) soup = BeautifulSoup (Res.text, "html.parser") houses = Soup.select (". Shop_list DL ") Pageinfolist = []for house in houses:try:# print (domain + house.select (" a ") [0][' href ']) info = gethouseinfo (domain + hous E.select ("a") [0][' href ']) pageinfolist.append (info) print (info) except Exception as E:print ("----> Exception, skip continue execution", E) DF = PD. DataFrame (pageinfolist) return dfconnect = Create_engine ("mysql+pymysql://root:[email protected]:3306/ Houseinfo?charset=utf8 ") for I in Range (1,int (Gettotalpage ()) +1): Try:df_onepage = Pagefun (str (i)) except Exception as E: Print ("Exception", E) pd.io.sql.to_sql (Df_onepage, "City_house_price", Connect, schema= "Houseinfo", if_exists= " Append ")

　　

Python crawler crawl room World data-into MySQL database