Python processing concurrency and python processing concurrency

Python processing concurrency and python processing concurrency

The system programmers often attack threads. The application scenarios they consider may not be met by general application programmers in their whole life ...... For application programmers, in 99%, they only need to know how to derive a bunch of independent threads and then use the queue to collect results.

 

Example: three network download styles 

To efficiently process network I/O, you need to use concurrency. Because the network has a high latency, it is recommended to do other things before receiving the network response in order not to waste the CPU cycle.

To illustrate this through code, I wrote three sample programs to download the National Flag images of 20 countries from the Internet. The first sample program flags. py is sequentially downloaded. After an image is downloaded and saved on the hard disk, the next image is requested. The other two scripts are concurrently downloaded: almost all images are requested at the same time, and one file is saved every time a file is downloaded. The flags_threadpool.py script uses the concurrent. futures module, while the flags_asyncio.py script uses the asyncio package.

Run the flags. py, flags_threadpool.py, and flags_asyncio.py scripts.

$ Python3 flags. pyBD br cd cn de eg et fr id in ir jp mx ng ph pk ru tr us vn ← after each script is run, the country code downloaded during the download is first displayed, the last message is displayed, indicating that it takes 20 flags downloaded in 7.26 s to consume flags. the average usage time of The py script to download 20 images is 7.18 seconds $ python3 flags. pyBD br cd cn de eg et fr id in ir jp mx ng ph pk ru tr us VN20 flags downloaded in 7.20 s $ python3 flags. pyBD br cd cn de eg et fr id in ir jp mx ng ph pk ru tr us VN20 flags downloaded in 7.09 s $ python3 flags_threadpool.pyDE BD CN JP ID EG NG BR RU CD IR MX US ph fr pk vn in et TR20 flags downloaded in 1.37 s average flags_threadpool.py script average time 1.40 seconds $ python3 flags_threadpool.pyEG BR FR IN BD JP DE RU PK PH CD MX ID US NG TR CN VN ET IR20 flags downloaded in 1.60 s $ python3 average de eg cn id ru in vn et mx fr cd ng us jp tr pk br ir PH20 flags downloaded in 1.22 s $ python3 flags_asyncio.py average flags_asyncio.py script 1.35 seconds bd br in id tr de cn us ir pk ph fr ru ng vn et mx eg jp CD20 flags downloaded in 1.36 s $ python3 flags_asyncio.pyRU CN BR IN FR BD TR EG VN IR ph cd et id ng de jp pk mx US20 flags downloaded in 1.27 s $ python3 flags_asyncio.pyRU in id de br vn pk mx us ir et eg ng bd fr cn jp ph cd tr country code Order: for concurrent download scripts, the download sequence varies by 20 flags downloaded in 1.42 s.

The performance of the two concurrently downloaded scripts is not significantly different, but it is more than 5 times faster than that of the sequentially downloaded scripts. This is only a very small task. If you increase the number of downloaded files to several hundred, the concurrent download script can be 20 times faster or more than the sequential download script.

Script downloaded in sequence