Python uses multi-threaded and multi-process to get all stock real-time data

 Python uses multi-threaded and multi-process to get all stock real-time data 

The previous day briefly introduced how Python obtains historical data and real-time data, so what do you do if you want to get real-time pen data for all listed companies?
It must be thought that a list of all listed companies to store the stock code, and then infinite loop to get it?
Now there are more than 3,400 shares in Shenzhen and Shanghai, if you do so, it takes more than 10 20 seconds to get real-time data for all stocks at once, even more, and it's very easy to hang up the program because of waiting for timeouts, which is certainly not possible if your model has a high quality requirement for real-time data. Even if data quality is not taken into account, it is not always possible to get the data script to hang.
The first thing that comes to mind is multithreading and multi-process. However, because Python globally explains the existence of locks, in multithreaded situations, even in the case of multi-core CPUs, you can only execute one thread at a time, even so, multithreading takes time for all stock real-time data, still much less than using single-threaded threads in a cycle. This is because Python takes data from the network to the local, is IO-intensive tasks, Python multithreading can still greatly improve performance, details are not described here.
In order to make full use of the bandwidth resources, IO resources, the use of multi-threaded and multi-process two ways to obtain stock data,
First we need to have a comparison of all the stock code files, I have been ready for everyone, attention to the public "good data" reply "stock" Four words can be downloaded link, compression package provides multi-threaded and multi-process to obtain stock real-time data two cases, wherein the stocks file is stored in the stock code file, All listed company codes are provided as of today (2017-11-23).

Here to provide multi-threaded examples, multi-process examples please check in the "Good data", while the code is also ready for everyone, the above reply to the "stock" to get the download link directly there is a Python source file, download after modifying the path directly can run, The Python version used in the article was shared yesterday in the anaconda of the python3.5, the code is still written using Jupyter, the operating system for WIN10, on Linux only need a little change on the implementation.

Import PandasAs PDImport NumPyAs NPImport TushareAs TSImport OSImport timeFrom threadingImport ThreadDefGet_data '%y-%m-%d-%h-%m-%s ', Time.localtime (Time.time ())) data=ts.get_realtime_quotes (stock) print (data) Path= F:\\ Stocks ' Data.to_csv (path+ ' \ \ \ ' +date+ ' _ ' +stock,mode= '%y-%m-%d-%h-%m-%s ') print (time1) with Open ( ' F:\stocks\stock_codes\stocks ') as F: try: while true:line=next (f). Strip () T=thread (target=get_data,args= (line,)) T.start () except stopiteration: passtime2=time.strftime (%y-%m-%d-%h-%m-%s ') print (time2)          

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18
• 19
• 20
• 21st
• 22
• 23
• 24
• 25

Where time1 is to get the start time of all stocks, time2 is the end time, the specific time spent, please download the meta-generation to test themselves

2017-11-23-21-16-23
2017-11-23-21-16-33

This is the time that I have commented on the Prinf (data) line, it took exactly 10 seconds, if your network is better, it may be faster, of course, this is not the quickest way, the fastest way can be achieved within one second.

The data has been stored in the specified directory, using the multi-process example please download the download link for you.
Here is the "data of the best" QR code, I will share in the public number of quantitative modeling cases, results and so on.

Python uses multi-threaded and multi-process to get all stock real-time data