Detailed description of deadlock traps when using Python + Java to call Shell scripts, pythonshell

Detailed description of deadlock traps when using Python + Java to call Shell scripts, pythonshell

Preface

A recent requirement is to regularly determine whether the execution conditions of a job meet and trigger a Spark job. When writing a Spark job, it is encapsulated into a Jar package and then passed in the required parameters for execution using Shell scripts, considering that the judgment condition logic is complex, it is not conducive to development and testing to be completed by using only Shell scripts. Therefore, the research uses the Spark script method called by Python and Java respectively.

Python 3.6.4 and JDK 8

Python

The subprocess library is mainly used. The Python API changes frequently, and the run method is added after 3.5, which greatly reduces the difficulty of use and the probability of encountering bugs.

subprocess.run(["ls", "-l"])subprocess.run(["sh", "/path/to/your/script.sh", "arg1", "arg2"])

Why can the run method reduce the probability of encountering bugs?

Before the run method is available, we generally call other advanced methods, that is, the Older high-level API, such as call, check_all, or directly create a Popen object. Because the default output is the console, if you are not familiar with the API or have not carefully read the doc, you want to wait for the sub-process to finish running and get the output.stdout = PIPEWhen wait is added, when the output content is too large, the Buffer will be full, and the process will remain waiting for reading, forming a deadlock. This strange phenomenon occurs when a Spark log is output to the console. The following script can be used to simulate it:

# a.shfor i in {0..9999}; do echo '***************************************************'done 

p = subprocess.Popen(['sh', 'a.sh'], stdout=subprocess.PIPE)p.wait()

While the call method directly calls wait internally to produce the same effect.

To avoid deadlocks, you must manually process the input and output before calling the wait method, or use the recommended communicate method. The communicate method generates internal read threads to read stdout stderr separately, thus avoiding Buffer full write. The new run method mentioned earlier calls communicate internally.

stdout, stderr = process.communicate(input, timeout=timeout)

Java

After Python is finished, Java is much simpler.

JavaRuntime.getRuntime().exec()Or ProcessBuilder calls the external script:

Process p = Runtime.getRuntime().exec(new String[]{"ls", "-al"});Scanner sc = new Scanner(p.getInputStream());while (sc.hasNextLine()) { System.out.println(sc.nextLine());}// orProcess p = new ProcessBuilder("sh", "a.sh").start(); p.waitFor(); // dead lock 

Note that:Here, the stream direction is relative to the main program, sogetInputStream() Is the output of the sub-process, andgetOutputStream() Is the input of a child process.

For the same Buffer reason, if the waitFor method is called to wait for the sub-process to complete execution and the output is not processed in time, a deadlock will occur.
Java APIs are rarely changed, so they do not provide new run methods like Python, but the open-source community also provides its own solutions, such as commons exec, or http://www.baeldung.com/run-shell-command-in-java, or the solution provided by alvin alexander (although incomplete ).

// Commons exec. To obtain the output, CommandLine = commandLine is more complex than python. parse ("sh. sh "); ByteArrayOutputStream out = new ByteArrayOutputStream (); PumpStreamHandler streamHandler = new PumpStreamHandler (out); Executor executor = new mongocute (commandLine); String output = new String (out. toByteArray ());

However, the idea is the same as that of Python. It is to enable a new thread in the background to read the output of the sub-process to prevent the Buffer from being full.

Another unified idea is that we recommend that you use arrays or lists to separate input shell commands into multiple segments. In this way, the system will handle special characters such as spaces.

Refer:

Https://dcreager.net/2009/08/06/subprocess-communicate-drawbacks/ https://alvinalexander.com/java/java-exec-processbuilder-process-1 https://www.javaworld.com/article/2071275/core-java/when-runtime-exec-won-t.html

Summary

The above is all the content of this article. I hope the content of this article has some reference and learning value for everyone's learning or work. If you have any questions, please leave a message to us, thank you for your support.