Python multi-threaded, multi-process

Source: Internet
Author: User

First, the multi-threaded, multi-process use and similarities and differences, but also related to the queue, memcache, redis operation, etc.:

1, in Python, if a program is IO-intensive operation, using multi-threaded, operation-intensive operations using multiple processes.

However, in fact, in Python, only support a multi-threaded CPU, multiple tasks are switching execution, and can not be executed in parallel, so sometimes, multithreading is not faster than a single thread, in our understanding, subconsciously will think

Multithreading is certainly faster than a single-threaded, but in fact, multithreading will only be blocked in the case of wired to play an effect, the following we look at an example:

1 ImportOs,sys,json2 Importthreadpool,threading3  fromCollectionsImportordereddict4 ImportCollections,time5 ImportRandom6 Importdatetime7 8 Print(Datetime.datetime.now ())9 defanalysed ():Ten     #count_dict = {} OneWith open ('Log.log') as Log_file: A          forLineinchlog_file.readlines (): -line =Line.strip () -             if '. Action?' inchLine : theUrl_api = Line.split () [6].split ('. Action') [0] -                 ifUrl_api.startswith ('http'): -Count =Count_dict.setdefault (url_api,0) -Count + = 1 +COUNT_DICT[URL_API] =Count -                 Else: +                     Continue A             Else: at                 Continue -  -Order_dict = collections. Ordereddict (Reversed (sorted (Count_dict.items (), key=LambdaX:x[1]))) -     #return order_dict -      forKeyinchOrder_dict.keys (): -         if '?' inchKey: in             Continue -         Else: to             #Print (Key,order_dict.get (key)) +             Pass -     Print(Datetime.datetime.now ()) theCount_dict = {} *t = Threading. Thread (target=analysed,args=()) $T.start ()

Explanatory notes:

First we look at the purpose of this script implementation, mainly is to open a log file, which is the Nginx access log, extract part of the URL, and then go to reorder output duplicate URL number, (Here I do not output, because it is convenient to see the time

The bottom two lines are multi-threaded, using the threading thread method, there are two parameters, one is target, to specify the function of the call, and the other is args, to specify the parameters passed, the other is not too much

Explained, look directly at the running time:

"2016-07-22 09:56:33.716978
2016-07-22 09:56:39.375297 "For a total of 6 seconds, we will not print out all the scripts, just change the last two lines to analysed (count_dict), call the function directly, and

The result of the output is:

"2016-07-22 10:19:57.294831
2016-07-22 10:20:02.549330 "; Total time 5 seconds

Summary:

From the output of the results can be seen, the use of multithreading than single-threaded time is longer, because the script does not cause the process of thread blocking problem, so the execution time is similar, where multithreading is not what to do

Use, but also to verify what we said above (thread blocking script instance will not show you, as long as you understand this truth OK)

2, said the multi-threading, the following we talk about the role of the thread pool, before the thread pool, we first look at the queue:

The queue can have many forms (1, FIFO, 2, advanced after out), in fact, regardless of the form, its role is to put the task in the inside, and then the process or thread to take the task to execute, below we define a queue, and list some of the commonly used

Method:

Myqueue = queue. Queue (); There can be a lot of parameters in parentheses (where maxsize = num) Specifies the maximum length of the queue

Myqueue.put (); A task is placed in parentheses to put the task in the queue

Myqueue.get (); This method is to get the task in the queue, and if there are no tasks in the queue, it will remain in the waiting state.

Myqueue.qsize (); This method is to get the length of the queue

Myqueue.empty (); Returns true if the queue is empty, otherwise false

Myqueue.full (); Returns true if the queue is full, and vice versa false

Attach an example:

1 ImportQueue2 ImportThreading3 Import Time4 5Q = queue. Queue (20)6 7 defProductor (ARG):8Q.put ('Buy Tickets')9 Ten #There are 6 people who buy tickets at the same time, but three servers can only process three requests at a time.  One  forIinchRange (6): At = Threading. Thread (target=productor,args=(i,)) - T.start () -  the defConsumer (ARG): -      whileTrue: -         Print(Arg,q.get ()) -Time.sleep (2) + #description for three servers at the same time, a single server can only process one request at a time -  forIinchRange (3): +t = Threading. Thread (target=consumer,args=(i,)) AT.start ()

Let's look at the code first and then list the results of the output; This code means that 6 months at the same time to buy tickets, and then three threads at the same time, buy tickets will be placed in the queue, that is q.put (' buy tickets ')

Thread processing inside threading. Thread (target=consumer,args=(i,)), the meaning of the parentheses is: target indicates the function to be called by the thread, and args is the argument to pass, where

We only pass the first few threads to perform the task, and then the consumer function executes the queue to fetch the task, that is, Q.get (). After reading this code, we look at the results of the output:

1 0 Buy Tickets 2 1 buy Tickets 3 2 buy tickets 40 Buy Tickets 5 1 buy tickets 6 2 Buy Tickets

In fact, we look at it is not very clear, in fact, after the output, the program has not stopped, the above we say the queue, but also simply mentioned, because, the last Q.get () will always wait for the output,

If we don't want the code to wait so long, we can add a timeout to Q.get (), which can be written in this mode: Q.get (timeout=5), wait 5 seconds.

Python multi-threaded, multi-process

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.