Python invokes an external subprocess that implements the asynchronous standard input and output through the pipeline.

We often encounter the need to implement a complex functional module through C + + or other lower-level languages, and to build a Web-based demo and query data. Thanks to the power and simplicity of the Python language, the Flask framework and the JINJA2 module feature provide a convenient web development capability for Python, which is ideal for building demos. At the same time, Python can easily interact with code in other languages. So we chose Python as the tool to develop the demo. Suppose we need to invoke the module (providing the underlying service) to read the data through a standard input loop, and write the results out to the output after processing, which is common in Linux environments and relies on the powerful redirection capabilities of Linux. Unfortunately, however, the underlying module has a very heavy initialization process, so we are not able to regenerate the sub-processes that call the underlying module every time the request is queried. The solution is to generate only one child process and then interact with the child process through the pipeline (pipe) for each request.

Python's subprocess module makes it easy to generate sub-processes, similar to the Linux system called Fork and exec. The Popen object of the Subprocess module may invoke an external executable program in a non-blocking manner, so we use the Poen object to implement the requirements. If we want to write data to the standard input stdin of the child process, Then, when creating the Popen object, you need to specify the parameter stdin as Subprocess.pipe, and similarly, if we need to read the data from the child process's standard output, we need to specify the stdout of the parameter when creating the Popen object. Let's look at a simple example:

From subprocess import Popen, Pipep = Popen ("Less", Stdin=pipe, Stdout=pipe) p.communicate (' line number%d.\n '% x)

The communicate function returns a two-tuple (Stdoutdata, Stderrdata) that contains the standard output of the child process and the output data that marked the error. However, because the communicate function of the Popen object blocks the parent process, it also closes the pipe, so that each Popen object can only be called once communicate function, and if there are multiple requests that must regenerate the Popen object (reinitialize the child process), Not be able to meet our needs.

Therefore, we can only achieve our needs by writing and reading data to the stdin and stdout objects of the Popen object. Unfortunately, however, the Subprocess module only runs at the end of the child process and reads the standard output only once. Both subprocess and os.popen* only allow input and output one time, and the. Output to is read only when the process Termin Ates.

After a few studies I found that the FCNTL function of the Fcntl module could be used to change the standard output of the process to a non-blocking way to achieve our goal. The problem that bothered me for a long time finally got the perfect solution. The code is as follows:

#!/usr/bin/python #-*-Coding:utf-8-*-# author:weisu.yxd@taobao.comfrom subprocess Import P       Open, Pipeimport fcntl, Osimport timeclass Server (object): Def __init__ (self, args, server_env = None): If server_env: self.process = Popen (args, Stdin=pipe, Stdout=pipe, Stderr=pipe, env=server_env) else:self.process = Popen ( Args, Stdin=pipe, stdout=pipe, stderr=pipe) flags = Fcntl.fcntl (Self.process.stdout, Fcntl. F_GETFL) Fcntl.fcntl (Self.process.stdout, Fcntl. F_SETFL, Flags | Os. O_nonblock) def send (self, data, tail = ' \ n '): self.process.stdin.write (data + tail) Self.process.stdin.flush () de    F recv (self, t=.1, e=1, Tr=5, stderr=0): Time.sleep (t) if tr < 1:TR = 1 x = Time.time () +t r = " PR = self.process.stdout If STDERR:PR = Self.process.stdout while Time.time () < X or R: R = Pr.read () if R is none:if e:raise Exception (message) Else: Break Elif R:return R.rstrip () Else:time.sleep (Max ((X-time.time ())/tr, 0) ) return R.rstrip () if __name__ = = "__main__": Serverargs = ['/home/weisu.yxd/qp/trunk/bin/normalizer ', '/home/weisu.yx D/qp/trunk/conf/stopfile.txt '] Server = Server (serverargs) Test_data = ' In the Cloud ', ' ladder ', ' Mossad ', ' Alisa ', ' IDB ', ' Ali Big Data ' for X in Test_data:server.send (x) print x, Server.recv ()

In addition, when calling some external programs, you may need to specify the appropriate environment variables, as follows:

  my_env = Os.environ  my_env["ld_library_path"] = "/path/to/lib"  server = server. Server (cmd, my_env)

