First, the multi-threaded, multi-process use and similarities and differences, but also related to the queue, memcache, redis operation, etc.:
1, in Python, if a program is IO-intensive operation, using multi-threaded, operation-intensive operations using multiple processes.
However, in fact, in Python, only support a multi-threaded CPU, multiple tasks are switching execution, and can not be executed in parallel, so sometimes, multithreading is not faster than a single thread, in our understanding, subconsciously will think
Multithreading is certainly faster than a single-threaded, but in fact, multithreading will only be blocked in the case of wired to play an effect, the following we look at an example:
1 ImportOs,sys,json2 Importthreadpool,threading3 fromCollectionsImportordereddict4 ImportCollections,time5 ImportRandom6 Importdatetime7 8 Print(Datetime.datetime.now ())9 defanalysed ():Ten #count_dict = {} OneWith open ('Log.log') as Log_file: A forLineinchlog_file.readlines (): -line =Line.strip () - if '. Action?' inchLine : theUrl_api = Line.split () [6].split ('. Action') [0] - ifUrl_api.startswith ('http'): -Count =Count_dict.setdefault (url_api,0) -Count + = 1 +COUNT_DICT[URL_API] =Count - Else: + Continue A Else: at Continue - -Order_dict = collections. Ordereddict (Reversed (sorted (Count_dict.items (), key=LambdaX:x[1]))) - #return order_dict - forKeyinchOrder_dict.keys (): - if '?' inchKey: in Continue - Else: to #Print (Key,order_dict.get (key)) + Pass - Print(Datetime.datetime.now ()) theCount_dict = {} *t = Threading. Thread (target=analysed,args=()) $T.start ()
Explanatory notes:
First we look at the purpose of this script implementation, mainly is to open a log file, which is the Nginx access log, extract part of the URL, and then go to reorder output duplicate URL number, (Here I do not output, because it is convenient to see the time
The bottom two lines are multi-threaded, using the threading thread method, there are two parameters, one is target, to specify the function of the call, and the other is args, to specify the parameters passed, the other is not too much
Explained, look directly at the running time:
"2016-07-22 09:56:33.716978
2016-07-22 09:56:39.375297 "For a total of 6 seconds, we will not print out all the scripts, just change the last two lines to analysed (count_dict), call the function directly, and
The result of the output is:
"2016-07-22 10:19:57.294831
2016-07-22 10:20:02.549330 "; Total time 5 seconds
Summary:
From the output of the results can be seen, the use of multithreading than single-threaded time is longer, because the script does not cause the process of thread blocking problem, so the execution time is similar, where multithreading is not what to do
Use, but also to verify what we said above (thread blocking script instance will not show you, as long as you understand this truth OK)
2, said the multi-threading, the following we talk about the role of the thread pool, before the thread pool, we first look at the queue:
The queue can have many forms (1, FIFO, 2, advanced after out), in fact, regardless of the form, its role is to put the task in the inside, and then the process or thread to take the task to execute, below we define a queue, and list some of the commonly used
Method:
Myqueue = queue. Queue (); There can be a lot of parameters in parentheses (where maxsize = num) Specifies the maximum length of the queue
Myqueue.put (); A task is placed in parentheses to put the task in the queue
Myqueue.get (); This method is to get the task in the queue, and if there are no tasks in the queue, it will remain in the waiting state.
Myqueue.qsize (); This method is to get the length of the queue
Myqueue.empty (); Returns true if the queue is empty, otherwise false
Myqueue.full (); Returns true if the queue is full, and vice versa false
Attach an example:
1 ImportQueue2 ImportThreading3 Import Time4 5Q = queue. Queue (20)6 7 defProductor (ARG):8Q.put ('Buy Tickets')9 Ten #There are 6 people who buy tickets at the same time, but three servers can only process three requests at a time. One forIinchRange (6): At = Threading. Thread (target=productor,args=(i,)) - T.start () - the defConsumer (ARG): - whileTrue: - Print(Arg,q.get ()) -Time.sleep (2) + #description for three servers at the same time, a single server can only process one request at a time - forIinchRange (3): +t = Threading. Thread (target=consumer,args=(i,)) AT.start ()
Let's look at the code first and then list the results of the output; This code means that 6 months at the same time to buy tickets, and then three threads at the same time, buy tickets will be placed in the queue, that is q.put (' buy tickets ')
Thread processing inside threading. Thread (target=consumer,args=(i,)), the meaning of the parentheses is: target indicates the function to be called by the thread, and args is the argument to pass, where
We only pass the first few threads to perform the task, and then the consumer function executes the queue to fetch the task, that is, Q.get (). After reading this code, we look at the results of the output:
1 0 Buy Tickets 2 1 buy Tickets 3 2 buy tickets 40 Buy Tickets 5 1 buy tickets 6 2 Buy Tickets
In fact, we look at it is not very clear, in fact, after the output, the program has not stopped, the above we say the queue, but also simply mentioned, because, the last Q.get () will always wait for the output,
If we don't want the code to wait so long, we can add a timeout to Q.get (), which can be written in this mode: Q.get (timeout=5), wait 5 seconds.
Python multi-threaded, multi-process