Python Basics-Day Nineth-Paramiko modules, processes, threads

the content of this article:

1.paramiko module Use

2. Introduction to Processes and threads

3.python Methods for calling Threads

4.join-Waiting for thread execution

5. Daemon Threads

6.GIL-Global Interpreter lock

7. Mutual exclusion Lock

8. Signal Volume

9. Events

10. Queues

One, Paramiko module use

Introduction to 1.paramiko Modules

? Paramiko is an SSH-based connection to a remote server and performs related operations (Sshclient and Sftpclinet, one is a remote connection, one is an upload download service), which can be used to command or file operations on a remote server, and it is worth saying that The remote management of fabric and ansible is the use of Paramiko to reality.

2. Using the Paramiko module do sshclient: Used to connect remote service and execute basic commands

To start the SSH program on the server

①sshclient does not encapsulate the use of transport

import?paramiko#? Create an SSH object Ssh?=?paramiko. Sshclient () #? Allow connections to hosts that are not in the ~/.ssh/known_hosts file Ssh.set_missing_host_key_policy (Paramiko. Autoaddpolicy ()) #? Connect to Server Ssh.connect (hostname= ' c1.salt.com ',? port=22,?username= ' root ',? password= ' 123 ') #? Execute the command, Do not perform the command Stdin,?stdout,?stderr?=?ssh.exec_command (' DF ') of top and so on without constant flushing #? Get command result Result?=?stdout.read () #? The command result obtained is bytes type print (Result.decode (encoding= "Utf-8")) #? Close Connection Ssh.close ()

The use of ②sshclient encapsulated transport

Import?paramikotransport?=?paramiko. Transport ((' hostname ',?))???? Build Connection Transport.connect (username= ' Wupeiqi ',? password= ' 123 ')???? # Build a connection #? Create an SSH object Ssh?=?paramiko. Sshclient ()???? #? Sshclient is the definition of how to transfer the command, how to interact with the file ssh._transport?=?transport#? Execute the command, do not perform a command like top such as a constant flush stdin,?stdout,?stderr?=?ssh.exec_ Command (' DF ') #? Get commands result Result?=?stdout.read () #? The command result obtained is the bytes type print (Result.decode (encoding= "Utf-8")) #? Close Connection Transport.close ()

③ private key connection based on SSH-free login

import?paramiko#? Specifies a private key Private_key?=?paramiko that uses SSH-free login. Rsakey.from_private_key_file ('/home/auto/.ssh/id_rsa ') Transport?=?paramiko. Transport ((' hostname ', ")") Transport.connect (username= ' root ',? pkey=private_key) Ssh?=?paramiko. Sshclient () Ssh._transport?=?transportstdin,?stdout,?stderr?=?ssh.exec_command (' DF ') transport.close ()

3. Use the Paramiko module for sftpclient: For connecting to a remote server and performing an upload download

To start the SSH program on the server

① based on user name password upload download, sftpclient package Transport usage

Import?paramikotransport?=?paramiko. Transport ((' hostname ',? 22))?? #? Establish a connection Transport.connect (username= ' root ',? password= ' 123 ')?? # Build a connection #? Create an Sftp object Sftp?=?paramiko. Sftpclient.from_transport (transport)?? #? Sftpclient is the definition of how to transfer files, how to interact with files #? Uploading location.py to the server?/tmp/test.pysftp.put ('/tmp/location.py ',? ' /tmp/test.py ') # Remove_path? Download to local? Local_pathsftp.get (' Remove_path ',? ' Local_path ') #? Close Connection Transport.close ()

② upload and download of private keys based on ssh-free login

import?paramiko#? Specifies a private key Private_key?=?paramiko that uses SSH-free login. Rsakey.from_private_key_file ('/home/auto/.ssh/id_rsa ')? Transport?=?paramiko. Transport ((' hostname ', ")") Transport.connect (username= ' root ',? Pkey=private_key?)? Sftp?=?paramiko. Sftpclient.from_transport (transport) # location.py? Upload to server?/tmp/test.pysftp.put ('/tmp/location.py ',? ' /tmp/test.py ') # Remove_path? Download to local? Local_pathsftp.get (' Remove_path ',? ' Local_path ')? Transport.close ()

Ii. Introduction to Processes and threads

1. Introduction to the process

? Each process provides the resources required to execute the program. A process has a virtual address space, executable code, an open handle to a system object, a security context, a unique process identifier (PID), an environment variable, a priority class, a minimum and maximum working set size, and at least one thread of execution. Each process starts with a single thread, often called the main thread, but can create additional threads from any of its threads.

? The process itself cannot be executed and must be executed by thread;

2. Introduction to Threading

? A thread is the smallest unit that the operating system is able to dispatch (the smallest unit of the operating system's CPU is the thread). It is included in the process and is the actual operating unit of the process.

?? A thread is an execution context (directive), which is all the information required by the CPU to execute the instruction stream.

?? A thread refers to a single sequential control flow in a process in which multiple threads can be concurrent and each thread performs different tasks in parallel.

3. The difference between a process and a thread

? The thread shares the address space of the process that created it, and the process has its own address space (as is the case with multiple child processes of the same parent process). A thread in the same process is a shared data, and the process is not sharing data (as is the case with multiple child processes of the same parent process)

?? A thread can access the data segment of its process directly, and the child process has a copy of the parent process data segment.

?? Threads can communicate directly with other threads of the same process, and processes must use interprocess communication (IPC) to communicate with sibling processes.

?? New threads are easy to create, and new processes require replication (cloning) of the parent process.

?? Threads can have considerable control and manipulation of other threads of the same process, and processes can only be controlled and manipulated by the child process.

?? Changes to the main thread (cancellation, priority changes, and so on) may affect the behavior of other threads in the same process, and changes to the parent process do not affect the child process (as long as the parent process is not deleted, the child process is not affected).

Third, Python method of calling thread

1. How to call threads directly

import?threadingimport?time#? Define the function that the thread will run #? The function name can be arbitrarily named Def?run (n):???? Print ("Task",? n)???? Time.sleep (2) if?__name__?==? " __main__ ":???? #?group default is empty, reserved for future extensions to the Threadgroup class implementation???? #?target is a callable object that is called by the Run method function. The default is empty, which means nothing is done???? #?name is the thread name. By default, a unique name is constructed: Thread-n,n is a decimal number???? #?args is the parameter tuple of the callable object called by Target, which defaults to (). Even if there is only one argument, add a comma???? #?kwargs is a dictionary of keyword parameters for callable objects called by target???? T1?=?threading. Thread (target=run,?args= (1,))?? # Create a thread instance???? T2?=?threading. Thread (target=run,?args= (2,))?? # generate another thread instance???? T1.start ()?? # Start a thread???? T2.start ()?? #? Start another thread

2. How to inherit the calling thread (custom thread Class)

Import?threadingimport?timeclass? MyThread (Threading. Thread):???? "" " Custom thread Class "" "???? Def?__init__ (self,?num):???????? First refactor the constructor and then inherit the parent class's constructor???????? Super (Mythread,?self). __init__ ()???????? Self.num?=?num???? Define the method function that the thread will run???? Note that the method function name must be run, because the program has been written dead, will automatically call the Run method function???? Def?run (self):???????? Print ("Number of runs:%s"?%?self.num)???????? Time.sleep (3) if?__name__?==? ' __main__ ':???? T1?=? MyThread (1)?? # Generate a thread instance and pass the parameters???? T2?=? MyThread (2)?? # generate another thread instance and pass the parameters???? T1.start ()?? # Start a thread???? T2.start ()?? #? Start another thread

3. Other methods of threading

? Print (thread instance. GetName ()): Gets the thread name;

?? Thread instance. SetName (name): Sets the name for the thread;

?? Thread instance. Setdaemon (): Sets the thread as a background thread (daemon) or foreground thread, by default all threads are foreground threads. The thread instance name. Setdaemon () represents the setting of the current thread to the foreground thread, and the thread instance name. Setdaemon (True) represents the setting of the current thread as a background thread (daemon thread).

?? Thread instance. Join (Timeout): Waits for the thread to terminate, timeout is the number of seconds to wait, and timeout is empty to wait until the thread terminates. After the waiting thread executes, the mainline friend continues to execute, and the method makes multithreading meaningless;

?? Thread instance. Run (): The thread is automatically executed by the CPU after the thread object's Run method;

?? Print (Threading.active_count ()): View the current number of active threads;

?? Print (Threading.current_thread ()): View the current thread instance, the main thread is called Mainthread, the child thread is called thread-n;

4. Note After the program turns on the thread

? The main program is the main thread in the execution;

? After the main thread creates the child threads, the main thread does not wait for the child threads to complete before executing down. This means that the main thread and the child thread are parallel;

Iv. join-Wait for thread execution

Thread instance. Join (Timeout): Waits for the thread to terminate, timeout is the number of seconds to wait, and timeout is not specified to wait until the thread terminates. After the waiting thread finishes executing, the main line friend continues to execute;

Import?threadingimport?timedef?run (n):???? Print ("Task",? n)???? Time.sleep (2)???? Print ("Task?done",? n)???? Print ("View current Thread",? Threading.current_thread ()) if?__name__?==? " __main__ ":???? T_obj_list?=? []???? Start_time?=?time.time ()???? For?i?in?range (3):?? # Start 3 Threads???????? T?=?threading. Thread (target=run,?args= ("t-%s"?%?i,))?? # Generate a thread instance???????? T.start ()?? # Start a thread???????? T_obj_list.append (t)???? Print ("Current number of active threads",? Threading.active_count ())???? For?item?in?t_obj_list:???????? Item.join ()?? # Wait for thread to terminate???? Print ("Total program execution time?%s? Seconds"?%? ( Time.time ()?-? start_time))???? Print ("View current Thread",? Threading.current_thread ())

Five, The Guardian thread

1. What is a daemon thread

? When the main thread finishes executing, any daemon thread terminates automatically, regardless of whether or not the execution completes. The program will only wait for the main thread, the non-daemon thread to complete before exiting the program;

?? The main thread cannot be set as a daemon thread;

?? The daemon thread has been listening for the main thread to exit;

2. How to set the daemon thread

? Thread instance. Setdaemon (): Sets the thread as a background thread (daemon) or foreground thread, by default all threads are foreground threads. The thread instance name. Setdaemon () represents the thread instance that sets the current thread to the foreground thread. Setdaemon (True) represents the setting of the current thread as a background thread (daemon thread).

? ? If it is a background thread, during the main thread execution, the background thread is also in progress, and after the main thread executes, the background thread stops regardless of whether it succeeds or not.

?? If it is a foreground thread, the foreground thread is also in progress , and after the main thread executes, wait for the foreground thread to finish executing, the program stops .

? Note that it must be set before starting the thread, or it will not be set;

3. Example

Import?threadingimport?timedef?run (n):???? Print ("Task",? n)???? Time.sleep (2)???? Print ("Task?done",? n)???? Print ("View current Thread",? Threading.current_thread ()) if?__name__?==? " __main__ ":???? For?i?in?range (3):?? # Start 3 Threads???????? T?=?threading. Thread (target=run,?args= ("t-%s"?%?i,))?? # Generate a thread instance???????? T.setdaemon (True)?? Set the current thread as a daemon thread. Be sure to set the???????? before starting the thread T.start ()?? # Start a thread???? Print ("Current number of active threads",? Threading.active_count ())???? Print ("View current Thread",? Threading.current_thread ())

Vi. GIL-Global interpreter lock

?? The role of the Global Interpreter Lock (GIL) is to have only one thread call the CPU at a time;

?? No matter how many CPUs are on the machine, the Global Interpreter Lock (GIL) allows only one thread to perform the operation at the same time, no matter how many threads are started by CPython;

?? Because of random scheduling between threads, dirty data may appear when multiple threads modify the same data at the same time, so a global interpreter lock (GIL) is present, allowing only one thread to perform operations at the same time.

Seven, mutual exclusion lock

1. Why you should have a mutex

? A process can start multiple threads, and multiple threads share the memory space of the parent process, which means that each thread can access the same data. Assuming that there are now a, b two threads, at this point to the number minus 1 operations, because 2 threads are concurrently running concurrently, so 2 threads are likely to take the number=100 this initial variable to the CPU to calculate, when a thread end result is 99, But at this point the result of the B-thread operation is also 99, two threads simultaneously assigns the result of the CPU operation to the number variable, the result is 99. The result of this calculation is a problem, how to do? It is very simple that each thread will modify the data in order to avoid having to modify it when it is not finished, so you can add a lock to the data so that other threads will have to wait for you to modify the data and release the lock before accessing the data.

2. Examples of Mutex lock

Import?threadingdef?operation ():???? Global?number?? Get this global variable in each thread???? Print (' Thread instance?%s? ' Gets the number value for?%s? '? %? (Threading.current_thread (),? number))???? Lock.acquire ()?? #? lock???? before modifying data Number?-=?1?? # 1 Operation???? for this public variable Lock.release ()?? #? Release the lock number?=?10 after the modification is complete?? #? Set a shared variable thread_list?=? []lock?=?threading. Lock ()?? #? Generate global lock For?i?in?range:???? T?=?threading. Thread (target=operation)???? T.start ()???? Thread_list.append (t) #? Wait for all threads to finish, or the main thread will execute down, it is possible that some threads have not finished executing, the main thread prints a number value incorrectly for?t?in?thread_list:???? T.join () print (' The value of number after all thread modification is complete ',? ')

3. Recursive lock

Sometimes it is necessary to modify the data to require a multi-layer lock (The large lock also contains a sub-lock), if the use of ordinary locks to lock the data, will appear after modifying the data can not release the lock, lock dead. Recursive locks are used for this situation;

4. Example of a recursive lock

IMPORT?THREADINGDEF?RUN1 ():???? Print ("Fetch data for the first time")???? Lock.acquire ()?? # Change the data before adding a small lock???? Global?num?? This global variable is obtained in the thread???? Num?+=?1???? Lock.release ()?? #? Release the small lock when the modification is complete???? RETURN?NUMDEF?RUN2 ():???? Print ("Fetch data for the second time")???? Lock.acquire ()?? # Change the data before adding a small lock???? Global?num2?? This global variable is obtained in the thread???? Num2?+=?1???? Lock.release ()?? #? Release the small lock when the modification is complete???? RETURN?NUM2DEF?RUN3 ():???? Lock.acquire ()?? # Change the data before adding a big lock???? RES?=?RUN1 ()???? Print ('--------between the RUN1 function and the RUN2 function-----')???? RES2?=?RUN2 ()???? Lock.release ()?? #? Release the large lock after the modification is complete???? Print (the value returned by the "run1 function is",? res)???? Print (the value returned by the "run2 function is",? res2) if?__name__?==? ' __main__ ':???? Num,?num2?=?0,?0???? Lock?=?threading. Rlock ()?? # Create a recursive lock???? For?i?in?range (2):?? # Open Two threads???????? T?=?threading. Thread (TARGET=RUN3)?? # Open Threads Call the RUN3 function first???????? T.start () While?threading.active_count ()?! =?1:?? The child thread has not finished executing???? Print ("Number of currently active threads",? Threading.active_count ()) Else:???? Print ('----all threads have finished executing---')???? Print ("Num's value is", "num")???? Print ("num2 value is",? num2)

Eight, the signal volume

1. Signal Volume Introduction

? mutexes allow only one thread to change data at the same time, while Semaphores (Semaphore) allow a certain number of threads to change data at the same time.

? The principle of releasing threads: when a thread in the allowed thread performs a release lock first, the program immediately releases the new thread, stabilizes the number of allowed threads in the set, and does not wait until all the allowed threads have finished executing and then unify the release;

?? Note that because the semaphore (Semaphore) is simultaneously allowing a certain number of threads to change the data, if these allowed threads change the same data, then there is a possibility that the result of the change will be wrong.

2. Examples of semaphores

Import?threadingimport?timedef?run ():???? Semaphore.acquire ()?? # lock???? Time.sleep (????) Print ("Thread?%s?" Running "?%?threading.current_thread (). Name)???? Semaphore.release ()?? # Release the lock if?__name__?==? ' __main__ ':???? # Generate a semaphore instance and set up to allow up to 3 threads to modify the data at the same time???? Semaphore?=?threading. Boundedsemaphore (3)???? For?i?in?range (10):?? # Open 10 Threads???????? T?=?threading. Thread (Target=run)???????? T.start () While?threading.active_count ()?! =?1:???? Passelse:???? Print (' All threads are executed ')

IX. Events

1. Introduction to Events

? The events of the Python thread are used for the main thread to control the execution of other threads, and the events provide three methods. Set, wait, clear.

? Event handling mechanism: A tag is defined globally, and if the tag value is false, the thread blocks when the Event.wait method is executed, and if the token value is true, the thread does not block when the Event.wait method is executed.

? Any thread can set tags, reset tags, and wait for the same event;

2. Examples of events

Import?threadingimport?timeevent?=?threading. Event () def?traffic_lights ():???? Count?=?1???? #? After the event object is created, the tag defaults to False, which sets the token to True, which represents the green light???? Event.set ()???? While? True:???????? IF?5?<?COUNT?<=?10:?? # Red light time 5 seconds???????????? Event.clear ()?? Set the tag to False to indicate a red light???????????? Print ("\033[41;1m red Light \033[0m")???????? ELIF?COUNT?>?10:?? # Green time 5 seconds???????????? Event.set ()?? Set the tag to true to represent a green light???????????? Count?=?1?? # Reset the Count???????????? Print ("\033[42;1m green light \033[0m")???????? else:???????????? Print ("\033[42;1m green light \033[0m")???????? Time.sleep (1)???????? Count?+=?1def?car (*args):???? While? True:???????? If?event.is_set ():?? Mark True to represent a green light???????????? Print ("%s pass"?%?args[0])???????????? Print ("%s pass"?%?args[1])???????????? Time.sleep (1)???????? else:???????????? Print ("Stop Waiting")???????????? Event.wait ()?? # Handle blocking status, wait for the flag to be set light_obj?=?threading. Thread (target=traffic_lights) Light_obj.start () For?i?in?range (2):?? # Open two-car thread???? Car_obj?=?threading. Thread (target=car,?args= ("Tesla",? " Benz "))???? Car_obj.start ()

Ten, the queue

1. Introduction to Queues

? Queue thread programming is particularly useful when information must be securely exchanged between multiple threads.

?? Queues are used for inter-thread communication, allowing data to be shared between threads.

? Two functions of the queue: decoupling and improving operation efficiency;

? The difference between a queue and a list: The data in the queue is taken away, and the data from the list is copied, and only the data is deleted manually;

2. Mode of the queue

① first in, first out mode

Queue object name? =?queue. Queue (maxsize=0)

② after-in first-out mode

Queue object name? =?queue. Lifoqueue (maxsize=0)

③ set priority when data is placed in the queue, by priority out queue (number of small precedence, first letter precedence), formatted as a tuple: (priority, data)

Queue object name? =?queue. Priorityqueue (maxsize=0)

Note that the default value of MaxSize is zero, and the size of the queue is infinite when the value of maxsize is less than or equal to zero.

3. Other methods of queuing

Queue object. Put (Item,?block=true,?timeout=none): The queue is not full and the item is placed in a queue. When the queue is full:

?? The value of Block true,timeout is none: The method function blocks until there is free space in the queue and puts the item in the queue;

?? The value of the block is a positive value of True,timeout: The method function blocks until the time-out period is set, and if there is free space in the queue during the timeout period, the item is placed in the queue, and the queue is thrown at the set time-out. The full exception.

?? The value of block is false: throws the Queue.full exception directly (in this case, the time-out is ignored);

Queue object. put_nowait (): The queue is not full and the item is placed in the queue. Queues are thrown when the queue is full. The full exception.

Print (Queue object. Get (Block=true,?timeout=none)): When there are items in the queue, they are immediately removed from the queue and an item is returned. When there are no items in the queue:

?? The value of Block true,timeout is none: The method function blocks until the item is in the queue, and the item is removed from the queue and returned;

?? The value of block true,timeout is a positive number: The method function blocks until the timeout time is set, and if there is an item in the queue, it is removed and an item is returned, and the queue is thrown when the time-out is set. The exception of empty;

?? The value of block is false: throws the Queue.empty exception directly (in this case, the time-out is ignored);

Print (Queue object. get_nowait ()): When there are items in the queue, they are immediately removed from the queue and an item is returned. A queue is thrown when there are no items in the queues. The exception of empty.

Print (Queue object. Qsize ()): Returns the number of items in the queue;

Print (Queue object. Empty ()): Returns True if the queue is empty, otherwise false;

Print (Queue object. Full ()): Returns False if the queue returns true;

4. Examples of queues

Import?threadingimport?timeimport?queuedef?producer ():???? Count?=?1???? While? True:???????? The queue is not full and the bones are placed in the queue???????? #? When the queue is full, the thread blocks until there is free space in the queue and puts the bone in the queue???????? Q.put ("Bone%s"?%?count)???????? Print ("Bone%s produced"?%?count)???????? Count?+=?1???????? Time.sleep (0.5) Def?consumer (name):???? While? True:???????? When there is a bone in the queue, it is immediately removed from the queue and a bone is returned???????? #? When there is no bone in the queue, the thread keeps blocking the bones in the queue and removes the bone from the queue and returns???????? Print ("%s" fetch to%s? and eat it "?%? ( Name,?q.get ()))???????? Time.sleep (1) q?=?queue. Queue (Ten) p?=?threading. Thread (Target=producer) c1?=?threading. Thread (target=consumer,?args= ("Husky",)) c2?=?threading. Thread (target=consumer,?args= ("Teddy",)) P.start () C1.start () C2.start ()

Python Basics-Day Nineth-Paramiko modules, processes, threads