Python multi-process (1)--subprocess and Popen ()

The main modules involved in Python multi-process include:

• Subprocess: You can execute other programs or commands in the current program;
• Mmap: Provides an inter-process communication mechanism based on memory;
• Multiprocessing: Provides multi-process programming interfaces that support multiprocessor technology, and the interface is designed to maximize consistency with the threading module for easy understanding and use.

This article focuses on the subprocess module and the Popen class it provides, and how to use it to create new child processes in a process. In addition, the other methods and properties provided by the Subprocess module are briefly described, which, although not Popen powerful tools, are very convenient and efficient in some cases.

The directory of this article is as follows:

1. subprocess. Popen class

2. Properties of the Popen object

3. Methods for Popen objects

4. Other easy ways to subprocess modules

5. Other properties of the Subprocess module

6. subprocess module-defined exceptions

Subprocess. Popen class

by calling:

Subprocess. Popen (args, bufsize=0, Executable=none, Stdin=none, Stdout=none, Stderr=none, Preexec_fn=none, Close_fds=false, shell= False, Cwd=none, Env=none, Universal_newlines=false, Startupinfo=none, creationflags=0)

　　creates and returns a child process , and executes the specified program in the process.

Instantiation Popen can customize the environment of a child process with many parameters, but only one parameter is required, that is, the positional parameter, args , and the remaining named parameters are described in detail below.

Parameter description:

• args: The path to the command or executable to execute. A sequence of strings (usually a list), the first element of the list is the executable's path, and the remainder is passed to the program's parameters, and if there are no parameters to pass to the program, the args parameter can be just a string.
• bufsize: Control stdin, stdout, stderr and other parameters specify the buffer of the file, and open the file of the parameters in the opening () function bufsize the meaning is the same.
• executable: If this parameter is not None, the substitution parameter args as the executable program;
• stdin: Specifies the standard input for the child process;
• stdout: Specifies the standard output of the child process;
• stderr: Specifies the standard error output of the child process;

For stdin, stdout and stderr , if they are None (by default), the child process uses the same standard stream file as the parent process.

If the parent process wants to communicate with the child process through the communicate () method, the corresponding parameter must be subprocess. PIPE (see example 4 below);

Of course stdin, stdout and stderr can also be open file objects, if open in a reasonable way, such as stdin corresponding files must be readable and so on.

• preexec_fn: The default is None, otherwise it must be a function or callable object, execute the function first in the child process, and then execute the program or shell specified for the child process.
• Close_fds: Boolean variable, true, forces all files except Stdin,stdout and stderr to be closed before the child process executes;
• ShelL: Boolean variable that explicitly requires the shell to run the program with the parameter executable to specify what shell the child process is running in--if Executable=none and Shell=true, The/bin/sh is used to execute the program specified by args, that is, Python starts with a shell and then uses the shell to interpret the command that is specified to run.
• CWD: A string representing the path specifying the working directory in which the child process is running, requiring that the directory be present;
• env: Dictionaries, keys, and values are strings that define environment variables for a child process;
• universal_newline: Boolean variable, True whenstdout and stderr are opened in Universal line break (Universal newline) mode,
• startupinfo: see next parameter;
• Creationfalgs: The last two parameters are parameters that are only in windows and are passed to the Win32 CreateProcess API call.

Similar to creating a child process in Linux, the parent process does not automatically wait for the child process to execute after the child process is created, and the parent process before the child process will cause the child process to become an orphan process, and the orphan process is uniformly taken over by the INIT process, responsible for its termination of the recycle work.

If the parent process terminates after the child process, but the child process terminates, the parent process does not make the final recycle, and the remaining data structure of the child process is called the zombie process. A large number of zombie processes will consume system resources, so it is necessary for the parent process to wait and reclaim the child process in time, unless it can confirm that it is terminated before the child process, thus transitioning the recycle work to the INIT process.

This wait and reclaim process is the wait () function, which is described later in this article.

Example 1:

Create a child process, and then execute a simple command

>>> import subprocess>>> p = subprocess. Popen (' ls-l ', shell=true) >>> total 164-rw-r--r--  1 root root   133 Jul  4 16:25 admin-openrc.sh-rw-r--r--  1 root root   268 Jul 15:55 admin-openrc-v3.sh...>>> p.returncode>> > p.wait () 0>>> p.returncode0

p = subprocess can also be used here. Popen ([' ls ', '-cl ']) to create a child process.

Popen properties of an object

Popen creates a child process that has some useful properties, assuming P is a child process created by Popen, the properties of P include:

1.

P.pid

The PID of the child process.

2.

P.returncode

This property represents the return state of the child process, and ReturnCode may have multiple conditions:

• none--the child process has not been completed;
• ==0--the child process to exit normally;
• > 0--Sub-process abnormal exit, ReturnCode corresponding to the error code;
• < 0--process was killed by the signal.

3.

P.stdin, P.stdout, P.stderr

Some initial files corresponding to the child process, if called popen (), the corresponding parameter is subprocess. Pipe, the corresponding property here is a file object that wraps the pipe.

　　

Methods for Popen objects

1.

P.poll ()

Check if the child process P has terminated, return the P.returncode property (refer to the properties of the Popen object below);

2.

P.wait ()

Wait for the child process p to terminate, return the P.returncode property;

Attention:

Wait () immediately blocks the parent process until the child process finishes!

3.

P.communicate (Input=none)

and the child process p communication, the data in the parameter input (string) is sent to the stdin of the child process, while reading the data from the stdout and stderr of the child process, until EOF.

　　return value:

Binary groups (Stdoutdata, stderrdata) represent data that is read from standard and standard errors, respectively.

Parent process calls P.communicate () and child process communication have the following limitations:

(1) Only through the pipeline and the child process communication, that is, only call Popen () to create the child process when the parameter stdin=subprocess. PIPE to send data to the stdin of the subprocess via P.communicate (input), only the parameters stout and stderr are subprocess. PIPE to receive data from the child process via p.communicate (), otherwise the corresponding position in the two-tuple received is none.

(2) The data that the parent process reads from the child process is cached in memory, so Commucate () is not suitable for data that is too large to be exchanged with the child process.

Attention:

Communicate () immediately blocks the parent process until the child process finishes!

4.

P.send_signal (signal)

Send a signal signalto the child process;

5.

P.terminate ()

Terminating the subprocess p is equal to sending a SIGTERM signal to the child process;

6.

P.kill ()

Kill the child process p, equal to send SIGKILL signal to the child process;

Other ways to subprocess modules

1.

Subprocess.call (args, *, Stdin=none, Stdout=none, Stderr=none, Shell=false)

The parent process creates the child process executor directly, and then waits for the child process to complete

return value:

Call () returns the exit state of the child process, which is the Child.returncode property;

2.

Subprocess.check_call (args, *, Stdin=none, Stdout=none, Stderr=none, Shell=false)

The parent process directly creates the child process executor, and then waits for the child process to complete, with specific parameters that can be used, referring to the Popen class described above.

　return value:

The function returns 0 regardless of the success of the child process;

If the exit state of the child process is not 0,check_call () throws an exception Calledprocesserror, the exception object contains the Child.returncode corresponding return code.

Example 2:

Check_call () Normal and error execution commands

>>> p = subprocess.check_call ([' Ping ', '-C ', ' 2 ', ' www.baidu.com ']) ping www.a.shifen.com (220.181.111.188) 56 ( Bytes of data.64 bytes from 220.181.111.188:icmp_seq=1 ttl=42 time=37.4 ms64 bytes from 220.181.111.188:icmp_seq=2 t TL=42 time=37.3 ms---www.a.shifen.com ping statistics---2 packets transmitted, 2 received, 0% packet loss, time 1001MSRT T Min/avg/max/mdev = 37.335/37.410/37.486/0.207 ms>>> print P0>>> p = subprocess.check_call ([' Ping ', '-C ', ' 4 ', ' [email protected]$#@! ( *^.com ']) ping:unknown host [email protected]$#@! (*^.comtraceback (Recent): File "<stdin>", line 1, <module> file "/usr/lib/python2.7/subpr ocess.py ", line 540, in Check_call raise Calledprocesserror (Retcode, cmd) subprocess.Calledprocesserror: Command ' [' ping ', '-C ', ' 4 ', ' [email protected]$#@! ( *^.com '] ' returned Non-zero exit status 2>>> print P0

　　

3.

Subprocess.check_output (args, *, Stdin=none, Stderr=none, Shell=false, Universal_newlines=false)

The parent process creates the child process executor directly, returning the output of the child process as a string .

return value:

The output of a child process in the form of a string, however,

If the exit status of the child process is not 0, the exception calledprocesserror is thrown, and the exception object contains the Child.returncode corresponding return code.

　　Attention:

There is no parameter stdoutin the function signature of Check_output (), and when the method is called, the output of the child process is returned to the parent process by default.

Example 3:

Check_output () called child process normal with error exit

>>> subprocess.check_output (["echo", "Hello world!"]) ' Hello world!\n ' >>> subprocess.check_output ("Exit 1", Shell=true) Traceback (most recent call last):   ... Subprocess. Calledprocesserror:command ' exit 1 ' returned Non-zero exit status 1

Attention:

When using the three methods mentioned above: Call (), Check_call (), and Check_output (), try not to set the parameters stderr and stdout to subprocess. Pipe, these functions by default will wait for the child process to complete, the child process produces a large amount of output data if the pipeline is blocked, the parent process waits for the child process to complete may cause a deadlock.

　　

Subprocess other properties of the module

Subprocess. PIPE

When you call several of the functions provided by this module, you open a pipeline for the standard stream file as the value of the std* parameter.

Example 4:

Connecting standard stream files with pipelines

Import subprocesschild1 = subprocess. Popen ([' ls ', '-l '], stdout=subprocess. PIPE) child2 = subprocess. Popen ([' WC ', '-l '], stdin=child1.stdout, stdout=subprocess. PIPE) out = Child2.communicate () child1.wait () child2.wait () print (out)

Here, the standard output of the subprocess child1 is used as the standard input for the child process child2, and the parent process reads Child2 's standard output through communicate ().

Subprocess. STDOUT

When invoking several functions provided by this module, the standard error output of the child process is printed to standard output as the value of the stderr parameter.

Subprocess module-defined exceptions

Exception subprocess. Calledprocesserror

(1) When the exception may be thrown: call Check_call () or Check_output (), the exit status of the child process is not 0 o'clock.

(2) The exception contains the following information:

• ReturnCode: The exit status of the child process;
• CMD: the command specified when the child process was created;
• Output: If this exception is thrown when calling Check_output (), this contains the output of the child process, otherwise the property is none.

Summarize

This article describes the basic usage of Python subprocess, which can be used to create child processes in the Python process, and if only interested in the execution exit state of a child process, you can call the Subprocess.call () function. Consider using Subprocess.check_call () and subprocess.check_output if you want the exception handling mechanism to resolve the situation where the child process exits unexpectedly. If you want to get the output of a child process, you can call Subprocess.check_output (), but Popen () is undoubtedly the most powerful.

The drawback of the subprocess module is that the default provides a limited means of communication between parent and child processes, only pipelines, and also creates child processes that are designed to execute external programs or commands.

There are many ways to communicate between processes in Linux, and it is possible for a child process to continue calling after creation

Python multi-process (1)--subprocess and Popen ()