Python crawls the first online loan, China's online loan financial management daily return index, and python crawls

Python crawls the first online loan, China's online loan financial management daily return index, and python crawls

Link: http://www.p2p001.com/licai/index/id/147.html

 

The link for obtaining data is similar to http://www.p2p001.com/licai/shownews/id/454.html:

Library:

Requests (For human) re (regular) pandas (used to process data) BeautifulSoup (used to parse WEB text) the logic thinking captured this time after the Code:

# Coding utf-8import requestsimport reimport pandasfrom bs4 import BeautifulSoupuser_agent = 'user-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Maxthon 2.0) 'headers = {'user-agent': user_agent} # define the function, get the link for each daily report, and return def get_newsurl () in the form of a list (): newsurl = [] url1 = 'HTTP: // www.p2p001.com/licai/index/id/147/p/'num = 1 url2+'.html 'while num <= 22: url = url1 + str (num) + url2 try: r1 = requests. get (url, headers = headers) failed T: print ('wrong % s' % url) else: s1 = BeautifulSoup (r1.text, 'lxml ') for x in s1.find _ all (href = re. compile ('licai/shownew'): newsurl. append (x ['href ']) num = num + 1 return newsurl # defines the function. The obtained data returns def get_info (): url = 'HTTP: // www.p2p001.com 'date = [] zonghe = [] one = [] one_three = [] three_six = [] six_twelve = [] twelve_most = [] for y in get_newsurl (): try: main_url = url + y r2 = requests. get (main_url, headers = headers) failed T: print ('wrong % s' % main_url) else: s2 = BeautifulSoup (r2.text, 'lxml') date. append (s2.find (text = re. compile ('statistical date') [5:]) rate = s2.find _ all ('td ') zonghe. append (rate [10]. string) one. append (rate [11]. string) one_three.append (rate [12]. string) three_six.append (rate [13]. string) six_twelve.append (rate [14]. string) twelve_most.append (rate [15]. string) p = {'date': Date, 'composite ': zonghe, '1 month': one, '1-3 month': one_three, '3-6 month': three_six, '6-12 month': six_twelve, '12 months and above ': twelve_most} return p # pandas storage data p = pd. dataFrame (get_info ())

P. to_csv ('f: // 1 // rate1.csv ', index = False, columns = ['date',' ', '1 month ', '1-3 month', '3-6 month', '6-12 month', '12 months and above '], Header = ['date', 'Total', '1 month', '1-3 month', '3-6 month ', '6-12 month', '12 months and above ']) 

 

 

Summary and reflection:

1. For ease of processing, the database is not used to store data, but pandas is used to save the data in csv format on the local disk F

2. Define the first function object get_newsurl and return the financial index daily report link in the form of a list. The second function traverses the return values of the first function for data collection.

3. Why not put a series of pandas operations in the function object get_info to directly complete a series of operations?

At that time, we considered efficiency and flexibility. One: If you leave pandas data operations in get_info, the overall efficiency of calling get_info () will be much lower than that of two: taking it out for separate processing, greatly improving flexibility, call get_info () to return the required data in a dictionary. After obtaining the data, I can do anything I want to do, instead of using a function to directly break the collected data into the dungeon, the data is free. 4. capture ideas ① first discover the daily report's link rules, that is to say, there are 22 pages, 20 Daily Report index data per page, because only one simple capture, therefore, this value can be written to the dead. This is what the first function does. Return the href value ② To Go To The daily report page, capture the data we need, and return it in a dictionary.

③ Process and store data (pandas)

Note: Data from the first net loan http://www.p2p001.com/

I am a little white, with limited capabilities and poor performance. Especially in terms of logic and thinking, the experts need to see a lot of advice and an axe to buddyquan. QQ: 1749061919 Crawlers