Python multi-Threading problem data lookup and summary by Tsy

Python multi-Threading problem data lookup and summary by Tsy

Statement:

1) this report by the blog Park Bitpeach Write, copyright, free reprint, please specify the source, and do not for commercial purposes.

2) If there are any infringing text or images in this document, please contact the author Bitpeach delete the appropriate section.

3) the contents of this document relate to Python multithreading problem, does not introduce the concept of multithreading, does not introduce multi-threaded program module, just to discuss the problem of multithreading, and find some materials for supporting and learning.

4) For reference purposes only, do not use as evidence to prove the purpose, please make your own choice, verify the reference.

5) Hyperlinks in this article, please do not directly click, for the convenience of reading, please select "Open in New tab page".

I'm sorry, I'm not editing in the blog editor, so I don't have time to adjust the window parameters of hyperlinks.

( 0 ) Description

The layout style of this article is different from before, the main reason this article is not so formal. When you recently wrote a python script for the Needleman-wunsch algorithm or the smith-waterman algorithm , The time-space complexity of the 22 alignment is very high when the number of sample sequences is found to reach 1000 in the process of the alignment sequence. So I want to use multi-threaded solution. But there are some problems with it, in order to think and summarize the relevant knowledge of multithreading, so make a simple transcript. This article is actually the equivalent of a memo, make a log, easy to continue learning and supplement multithreading. If there is also a study sequence comparison algorithm, welcome to discuss together.

if it's a beginner of sequence alignment, I was looking at the blog Park<million Cang a millet>the text comparison algorithm ⅰ--LDalgorithm and the text comparison algorithm (ⅱ--)Needleman/wunschalgorithm, the former is easy to understand the predecessor of sequence alignment, the latter helps to understand the global sequence alignment. CSDNNetizens<HUGEHEADHUG>reproduced in the blog "Dynamic Programming sequence comparison:Needleman-wunschAlgorithms andSmith-watermanalgorithm, for the understanding of global sequence alignment and local sequence comparison, although reproduced in this article, I did not find the source of the original text, and the theory of the algorithm is less introduced, mainly code, but I think it is very useful. Finally, I was readingIBMthe technical topic document "Dynamic programming and Gene sequence alignment", I've always feltIBMof these documents, written very well, not only this multi-sequence, including other topics of some useful algorithms, written very good. In the analysis of the origin of the algorithm, classics. In the algorithm's text description, the typesetting is beautiful, the statement is fluent, the module idea is clear. In the code implementation, there is a little detail, code style robust typical. This articleIBMthe article on multi-sequence alignment is very good. Some of the above articles are some reading materials that I have learned about multiple sequence alignment. It was originally usedCwritten, and the reference isCSDNBlogs,<nkliming>Netizens reproduced the blog "Needleman-wunsch Algorithm", this post is reproduced in theC + +code that is well worth reading and learning.

And then I learned.python, learn to write based onpythonalgorithm for multi-sequence alignment. But due to the complexity of spacepythonafter all, it's an explanatory language, so I'm afraid I can't keep upCof the speed, considered several methods. One is to usePyPyto accelerate, withPyPyCommunity Moderator After the discussion, foundPyPySome third-party libraries are not well supported and may need to start from now (2014.12), wait two years to develop well, andvitrualenvCurrent Recent version (1.11.6) does not yet support the default designationPyPy, causingPyPywith thepythonIt can be messy to use in an operating system. The second is to usePsyco, but it has not supportedpython2.7,PsycoThe latest version supported ispython2.5. Third, the multithreading method, is also the main issue discussed in this paper. Four is, usingCPython, but there is no time to learn, and IClearn good rotten, really want to cover the face.

( a ) Multi-threaded learning reference

(1.1) A simple introduction to multithreading

first of all, see the blog Park < Insect Master > Original blog post, the python Multithreading is so simple "and the continuation of the article, the author tries to use a simple example, a preliminary understanding of multithreading. After I read it, I felt very good, easy to use quickly and quickly understand. As this article explains the necessary functions of multithreading, it is very good to be quick to get started.

If you want to make some progress, read this section of the module that is not explained, such as lock <ASTRALWIND> python thread guide, this blog post is great, from basic concepts to thread (threading) The Netizen article is similar, actually also has an article w3cschool The article " python multi-Threading ", this article also introduces the module large and small parameters.

second, if you want to practice, do some small experiments. Can uphold the reading<Insect Master>of the blog, that blog as a simple small experiment, again suitable, easy to understand. You can also read the article on Bole's online website, "UnderstandingPythonis translated from a foreign language article. can also readChinaunixBlog Users<that piece is still sea>written by the original blog "pythoninThreadingmodule in detail (i) ". Blog Park's<Slice Inspiration>The original blog "PythonMulti-threaded learning and he wrote another article on thePythonMulti- Threaded application, the content of the practice of the introduction of more, is also a good choice.

(1.2) Multi-threading problem generation

Next, I willThreadingthe module is applied to my actual situation. In fact, the sample is +Sequence , each length of the sequence is approximately $characters. Operating conditions are, only forNeedlemanoperation, the sample 22 is compared to the operation, the output of two pairs of original sequence, output two pairs of sequence after the output backtracking the longest similar sequence. Single thread case near -seconds. and using multi-threading, in the same situation, it costs nearly -seconds. I was puzzled until Baidu came to aCSDNPost "pythoncan multithreading improve efficiency? , a moderator of several threads or an experienced person, points outpythonhave aGIL(Global interpreter Lock). This post gives a few comments: first, the language to write a higher-temporal complexity of the algorithm part, such asmatlab,C + +, of course, the example in the post isLua, and looked at it. A university research group in Brazil1993years of development, suddenly awe, I development although rapid, when there is the ability to develop language AH? I have never heard of any programming language developed in China (please do not speak easy language, thank you.) While not discriminating against the easy language, butLualanguage as the best choice of embedded scripts, is clearly a global promotion, and easy language is obviously not a global promotion, but also feel that the current decade is not to promote the global. ）。 Second, the use of a co-process or a multi-procedure approach. Third, use unlock or speed up plug-ins or other methods. The first one is not discussed, the second I have not yet collated material. For the third one, the main is to unlockGIL(individuals do not quite understand, see Blog Park<sanlo>User's Blog "in the Extension module onpythonof theGIL), or the method of specifying multicore operating devices may be more efficient. The main summarized materials are as follows:

for (W) netizens wrote thread safety and python gil Span style= "font-family: the song Body;" > ". Discussion on the translation of foreign articles into Chinese gil "panic." Issue, in Open source Chinese Community magazine most difficult question", It is also a very interesting historical science.

　　there is an article on the internet entitled python concurrent Pain ", but also to talk about this problem, but the source of the original I can not find, most are reproduced, so the hyperlink is a reprint of the link. This paper explains the Gil, explains the contradiction between thecurrent thread pool and GIL , and also introduces other possible ways to solve the problem, such as multi-process + co-procedure. There is also a Sina blog technical essay "Python two or three", also briefly mentions the problem of concurrency.

so knowGILIn addition to reading the introductory article above, if you want to go deeper, you can seeCSDNBlog's<I2Cbus>User original article "Python GILthe adoption of a series of examples of understandingPythonof theGIL", the author wrote several articles aboutGILseries of discussion articles, very praise! He took the lead in this article and drew some conclusions. And the author has done a very rigorous follow-up, correcting his previous erroneous conclusions and benefitingGIL's acting exploration, the author of this blog gives someGILthe other links are explained. Which means this blog post is just touchingGILquestion, the author has not done, wrote the "Again talk about the Gil of Python."

then< I2cbus >netizens continue to write the original technical article "Python GILthe series ofPython 2.7ImprovedGILa trial ofpython27the ImprovementGILperformance, made a summary of the discussion, and provided links to some materials in the blog post. The author then uses the setup process to run theCPUto improvePythonthe performance, wrote thePython GILthe series is run through the setup process.CPUto improvePythonthe performance of the program and thePython GILthe series is run through the setup process.CPUto improvePythonProgram Performance (cont.) blog post, which specifies multi-core for threads when threadingCPUOne , avoid switching waste time, achieve and improvePythonMulti-core operation (previously, single-threaded operation is faster than multithreading.) This article may be able to solve, for me, I have not had time to chew absorption, only from the thinking and theoretically think it can be solved, welcome someone to do the experiment).

Citing the comments of netizens, I think the comments are very good: human beings are always taking pains to invent simple language, and at the same time do not bother to add to it difficult to use new features.

This article is temporarily over, what new discoveries, will continue to add.

Python data finding and summarizing for multithreading problems by Tsy