Python Multithreaded Learning Notes

Before using the Linux shell to write multithreaded statistics Youku Web site members, now use Python to implement again, as a note of learning notes.

Python multithreading can create threads by importing the thread module, using functions in thread, or by inheriting the threading class.

Each process has its own address space, memory, data stacks, and secondary data that records its motion trajectory.

The execution of Python code is controlled primarily by the Python virtual machine (also known as the Interpreter Gil Main loop), and access to the Python virtual machine is controlled by the Python global Interpreter lock Gil, which guarantees that only one thread is running at any one time.

The reason for using the thread module is not recommended: There are many reasons for doing so, and obviously one reason is that when the main thread exits, all other threads are exited without being cleared. However, another module threading to ensure that all "important" child threads are exited before the process ends. Threading is more advanced than thread.

Another reason not to use thread is that when your process should end completely without control, when the main thread ends, all threads will be forced to end, no warning and no normal cleanup work. As we said before, at least the threading module ensures that the process does not exit until an important child thread exits.

The use of import thread in Python 3.3 is an error because thread support is not turned on and the Python interpreter needs to be recompiled to run.

Python provides several multithreaded programming modules: Thread, threading, queue
Single Thread

In the MS-DOS era a few years ago, operating system processing problems are single task, I want to listen to music and watch movies two things, then must first arrange order.

(All right!) We don't struggle with the use of listening to music and seeing in the DOS era. ^_^)

From time import Ctime,sleep

def music ():
For I in range (2):
Print "I is listening to music. %s '%ctime ()
Sleep (1)

def move ():
For I in range (2):
Print "I is at the movies! %s '%ctime ()
Sleep (5)

if __name__ = = ' __main__ ':
Music ()
Move ()
Print "All over%s"%ctime ()

We first listened to a piece of music, through a for loop to control the music played two times, each music playback needs 1 seconds, sleep () to control the length of the music playback. And then we watched a movie,

Every movie takes 5 seconds, because it looks so good, so I look at it two times through the for loop. At the end of the whole leisure and recreational activity, I passed

Print "All over%s"%ctime ()
Looking at the current time, it's almost bedtime.

Run Result:

>>=========================== Restart ================================
>>>
I was listening to music. Thu APR 17 10:47:08 2014
I was listening to music. Thu APR 17 10:47:09 2014
I was at the movies! Thu APR 17 10:47:10 2014
I was at the movies! Thu APR 17 10:47:15 2014
All over Thu APR 17 10:47:20 2014

　　

In fact, music () and move () should be viewed as musical and video players, as well as what songs and videos to play should be decided by us when we use them. So, we've transformed the code above:

#coding =utf-8
Import threading
From time import Ctime,sleep

def music (func):
For I in range (2):
Print "I is listening to%s.%s"% (Func,ctime ())
Sleep (1)

def move (func):
For I in range (2):
Print "I is at the%s! %s '% (Func,ctime ())
Sleep (5)

if __name__ = = ' __main__ ':
Music (U ' Love Business ')
Move (U ' Avatar ')

Print "All over%s"%ctime ()

The music () and move () are treated with parameters. Experience the Chinese classic songs and large and Western culture.

Run Result:

>>> ======================== Restart ================================
>>>
I was listening to love business. Thu APR 17 11:48:59 2014
I was listening to love business. Thu APR 17 11:49:00 2014
I am at the Avatar! Thu APR 17 11:49:01 2014
I am at the Avatar! Thu APR 17 11:49:06 2014
All over Thu APR 17 11:49:11 2014

Multithreading

Technology in the development of the Times in progress, our CPU is also getting faster, CPU complaints, p big things accounted for me a certain amount of time, in fact, I do a lot of work at the same time no problem, so the operating system has entered a multitasking era. We listen to music eating hotpot is not a dream.

Python provides two modules to implement multithreading thread and threading, thread has some shortcomings, in threading got made up, in order not to waste you and time, so we directly learn threading can be.

Start learning Python threads
There are two ways to use threads in Python: Functions or wrapping thread objects with classes.
Function: Invokes the Start_new_thread () function in the thread module to produce a new thread. The syntax is as follows:
Thread.start_new_thread (function, args[, Kwargs])
Parameter description:
function-thread functions.
Args-the argument passed to the thread function, he must be a tuple type.
Kwargs-Optional parameters.
Instance:
#!/usr/bin/python
#-*-Coding:utf-8-*-

Import Thread
Import time

# define a function for the thread
def print_time (ThreadName, delay):
Count = 0
While Count < 5:
Time.sleep (Delay)
Count + 1
Print '%s:%s '% (ThreadName, Time.ctime (Time.time ()))

# Create two threads
Try
Thread.start_new_thread (Print_time, ("Thread-1", 2,))
Thread.start_new_thread (Print_time, ("Thread-2", 4,))
Except
Print "Error:unable to start Thread"

While 1:
Pass
The results of the above program output are as follows:
Thread-1: Thu 22 15:42:17 2009
Thread-1: Thu 22 15:42:19 2009
Thread-2: Thu 22 15:42:19 2009
Thread-1: Thu 22 15:42:21 2009
Thread-2: Thu 22 15:42:23 2009
Thread-1: Thu 22 15:42:23 2009
Thread-1: Thu 22 15:42:25 2009
Thread-2: Thu 22 15:42:27 2009
Thread-2: Thu 22 15:42:31 2009
Thread-2: Thu 22 15:42:35 2009
The end of a thread generally relies on the natural end of the thread function, or it can be called thread.exit () in a thread function, and he throws Systemexit exception to exit the thread.
Threading Module
Python provides thread support through two standard library thread and threading. Thread provides low-level, raw threads, and a simple lock.
Other methods provided by the thread module:
Threading.currentthread (): Returns the current thread variable.
Threading.enumerate (): Returns a list containing the running thread. Running refers to threads that start and end after the thread has started, not including before and after the start.
Threading.activecount (): Returns the number of threads running, with the same result as Len (Threading.enumerate ()).
In addition to using methods, the thread module also provides the thread class to handle threads, and the thread class provides the following methods:
Run (): The method used to represent thread activity.
Start (): Starts thread activity.
Join ([TIME]): Wait until thread aborts. This blocks the calling thread until the thread's join () method is called aborted-gracefully quits or throws an unhandled exception-or an optional timeout occurs.
IsAlive (): Returns whether the thread is active.
GetName (): Returns the name of the thread.
SetName (): Sets the thread name.
To create a thread using the threading module
Use the threading module to create the thread directly from the threading. Thread inheritance, and then rewrite the __init__ method and the Run method:
#!/usr/bin/python
#-*-Coding:utf-8-*-

Import threading
Import time

exitflag = 0

Class Mythread (threading. Thread):   #继承父类threading. Thread
    def __init__ (self, ThreadID, name, counter):
         Threading. Thread.__init__ (self)
        self.threadid = ThreadID
         self.name = name
        self.counter = counter
    def run (self):                    #把要执行的代码写到run函数里面 thread runs the run function directly after it is created
         print "Starting" + Self.name
        print_time (Self.name, Self.counter , 5
        print "exiting" + self.name

def print_time (ThreadName, Delay, counter):
While counter:
If Exitflag:
Thread.exit ()
Time.sleep (Delay)
Print '%s:%s '% (ThreadName, Time.ctime (Time.time ()))
Counter-= 1

# Create a new thread
Thread1 = Mythread (1, "Thread-1", 1)
Thread2 = Mythread (2, "Thread-2", 2)

# Open Thread
Thread1.start ()
Thread2.start ()

Print "Exiting Main Thread"
The above procedure implementation result is as follows;
Starting Thread-1
Starting Thread-2
Exiting Main Thread
Thread-1: Thu Mar 21 09:10:03 2013
Thread-1: Thu Mar 21 09:10:04 2013
Thread-2: Thu Mar 21 09:10:04 2013
Thread-1: Thu Mar 21 09:10:05 2013
Thread-1: Thu Mar 21 09:10:06 2013
Thread-2: Thu Mar 21 09:10:06 2013
Thread-1: Thu Mar 21 09:10:07 2013
Exiting Thread-1
Thread-2: Thu Mar 21 09:10:08 2013
Thread-2: Thu Mar 21 09:10:10 2013
Thread-2: Thu Mar 21 09:10:12 2013
Exiting Thread-2
Thread synchronization
If multiple threads work together on a data modification, unpredictable results may occur, and multiple threads need to be synchronized in order to ensure the correctness of the data.
Using lock and Rlock of the thread object enables simple thread synchronization, both of which have a acquire method and a release method, and you can place the action between the acquire and released methods for data that requires only one thread to operate at a time. As follows:
The advantage of multithreading is that you can run multiple tasks at the same time (at least it feels that way). However, when a thread needs to share data, there may be a problem with data synchronization.
Consider a situation where all elements in a list are 0, and the thread "set" changes all elements from the back to 1, while thread "print" is responsible for reading the list and printing it backwards.
Then, when the thread "set" starts to change, the thread "print" Prints the list, and the output becomes half 01 and a 1, which is the data's different steps. In order to avoid this situation, the concept of lock is introduced.
There are two states of a lock-locked and unlocked. Whenever a thread such as "set" wants to access the shared data, the lock must first be obtained, and if there are other threads such as "print" that are locked, then the thread "set" is paused, which means the synchronization is blocked, and the thread "set" continues after the thread "print" is accessed and the lock is released.
After this processing, the print list is either full output 0, or all output 1, no more than half 1 and a half 1 embarrassing scene.
Instance:
#!/usr/bin/python
#-*-Coding:utf-8-*-

Import threading
Import time

Class Mythread (threading. Thread):
    def __init__ (self, ThreadID, name, counter):
         Threading. Thread.__init__ (self)
        self.threadid = ThreadID
         self.name = name
        self.counter = counter
    def run (self):
        print "starting" + Self.name
       # Get lock, return true after successful lock
       # Optional Timeout parameter will block until you get a lock
       # Otherwise timeout will return false
         threadlock.acquire ()
        print_time (Self.name, Self.counter, 3
        # release Lock
         Threadlock.release ()

def print_time (ThreadName, Delay, counter):
While counter:
Time.sleep (Delay)
Print '%s:%s '% (ThreadName, Time.ctime (Time.time ()))
Counter-= 1

Threadlock = Threading. Lock ()
Threads = []

# Create a new thread
Thread1 = Mythread (1, "Thread-1", 1)
Thread2 = Mythread (2, "Thread-2", 2)

# Open a new thread
Thread1.start ()
Thread2.start ()

# Add thread to Thread list
Threads.append (THREAD1)
Threads.append (THREAD2)

# Wait for all threads to complete
For T in Threads:
T.join ()
Print "Exiting Main Thread"
Thread priority queues (queue)
A synchronized, thread-safe queue class is provided in the Python queue module, including FIFO (first in first out) queue Queue,lifo (back-first out) queue Lifoqueue, and priority queue priorityqueue. These queues implement the lock primitives and can be used directly in multiple threads. You can use queues to implement synchronization between threads.
Common methods in the queue module:
Queue.qsize () returns the size of the queue
Queue.empty () returns True if the queue is empty, false instead
Queue.full () returns True if the queue is full, false instead
Queue.full and maxsize size correspond
Queue.get ([block[, timeout]]) Gets the queue, timeout wait time
Queue.get_nowait () quite queue.get (False)
Queue.put (item) write queue, timeout wait time
Queue.put_nowait (item) quite Queue.put (item, False)
Queue.task_done () After completing a work, the Queue.task_done () function sends a signal to the queue that the task has completed
Queue.join () actually means wait until the queue is empty, and then do something else
Instance:
#!/usr/bin/python
#-*-Coding:utf-8-*-

Import Queue
Import threading
Import time

exitflag = 0

Class Mythread (threading. Thread):
    def __init__ (self, ThreadID, name, Q):
        Threading. Thread.__init__ (self)
        self.threadid = ThreadID
         self.name = name
        self.q = q
 & nbsp;  def run (self):
        print "starting" + self.name
         Process_data (Self.name, self.q)
         print "exiting" + self.name

def process_data (ThreadName, Q):
While not exitflag:
Queuelock.acquire ()
If not Workqueue.empty ():
data = Q.get ()
Queuelock.release ()
print '%s processing%s '% (threadname, data)
Else
Queuelock.release ()
Time.sleep (1)

Threadlist = ["Thread-1", "Thread-2", "Thread-3"]
NameList = ["One", "two", "Three", "Four", "Five"]
Queuelock = Threading. Lock ()
Workqueue = Queue.queue (10)
Threads = []
ThreadID = 1

# Create a new thread
For Tname in Threadlist:
Thread = Mythread (ThreadID, Tname, Workqueue)
Thread.Start ()
Threads.append (thread)
ThreadID + 1

# Fill Queue
Queuelock.acquire ()
For word in NameList:
Workqueue.put (Word)
Queuelock.release ()

# Wait for the queue to clear
While not Workqueue.empty ():
Pass

# notification Thread It's time to quit
Exitflag = 1

# Wait for all threads to complete
For T in Threads:
T.join ()
Print "Exiting Main Thread"
Results of the above program execution:
Starting Thread-1
Starting Thread-2
Starting Thread-3
Thread-1 processing One
Thread-2 processing Two
Thread-3 processing Three
Thread-1 processing Four
Thread-2 processing Five
Exiting Thread-3
Exiting Thread-1
Exiting Thread-2
Exiting Main Thread

Learn notes

Import threading
Import Urllib2
Import Urllib
Import Base64
Import time

Class Getyouku:
def __init__ (Self,thread_num):
Self.thread_count=self.thread_num=thread_num
Self.lock=threading. Lock ()

def _getyoukucount (SELF,II):
Thread_id=int (Threading.currentthread (). GetName ())
self.str=2
Self.str+=ii
While Self.str > 1 and self.str<300:
Self.base64_str=base64.b64encode (str (SELF.STR))
Url= "Http://i.youku.com/u/U" +self.base64_str
Req=urllib2. Request (URL)
print ' Self.str: ' +str (SELF.STR) +urllib2.urlopen (req). Read ()

Self.str+=1


Self.lock.acquire ()
Self.thread_count-= 1
Self.lock.release ()


def run (self):
For I in Range (Self.thread_num):
Q=threading. Thread (Target=self._getyoukucount,name=str (i), args= (I,))
Q.setdaemon (True)
Q.start ()

While self.thread_count>0:
Time.sleep (0.01)

if __name__ = = ' __main__ ':
D=getyouku (THREAD_NUM=20)
D.run ()