Python Crawler Combat: Stock Data directed crawler __python

The author of this article: Hang

This article source: https://segmentfault.com/a/1190000010520835

function Introduction

Objective: To obtain the name and transaction information of all stock in SSE and SSE.

Output: Saved to file.

Technical route: Requests-bs4–re

Language: python3.5

Description

Site Selection principle: Stock information static exists in the HTML page, non-JS code generation, no ROBBTS protocol restrictions.

Selection method: Open the Web page, view the source code, search the Web page of the stock price data exists in the source code.

such as open Sina stock URL: Link description (http://finance.sina.com.cn/realstock/company/sz000877/nc.shtml), as shown in the following image:

On the left side of the page, the interface shows the stock price of Tianshan shares is 13.06. To the right for the Web page source code, in the source code query 13.06 found not found. Therefore, the data used to determine the Web page using JS generated, not suitable for this project. So change to a Web page.

Then open the Baidu Stock URL: Link description (https://gupiao.baidu.com/stock/sz300023.html), as shown in the following figure:

From the above figure can be found in Baidu stock data is generated by HTML code, in line with the requirements of our project, so in this project to choose Baidu Stock site.

Because Baidu shares only a single stock of information, so also need the current stock market all the stock list, where we choose east Net, URL is: Link description (http://quote.eastmoney.com/stocklist.html), the interface as shown in the following image:

Principle Analysis

See Baidu Stock of each stock URL: https://gupiao.baidu.com/stock/sz300023.html, you can find that the URL has a number 300023 is exactly the number of this stock, SZ said the Shenzhen exchange. So the program structure we construct is as follows:

Step 1: Get the stock list from East net;

Step 2: Get the stock code, and add to the Baidu Stock link, and finally to these links to access the stock information;

Step 3: Store the results in a file.

Then look at Baidu Stock Information page source code, found that each stock information in the HTML code stored in the following way:

Therefore, when we store information for each stock, we can refer to how the HTML code in the above image is stored. Each information source corresponds to an information value, which is stored in the form of a key-value pair. The way in which key-value pairs can be used in Python is a dictionary type. Therefore, in this project, use a dictionary to store information about each stock, then use the dictionary to record all the stock information, and finally output the data in the dictionary to the file.

Code writing

The first is to get the HTML page data program, here do not do more introduction, the code is as follows:

#获得html文本

def gethtmltext (URL):

Try

r = Requests.get (URL)

R.raise_for_status ()

r.encoding = r.apparent_encoding

Return R.text

Except

Return ""