"Python" uses the UNIX pipeline pipe to process stdout real-time data

Now there is a real-time grab packet processing program, the approximate process is to use the Tshark capture package--real-time upload, if the log is possible to write, but the log file cutting needs to be executed on a timed basis. Because some of the content in log needs to be processed in real time, the delay time can lead to data error, so the thought of a Unix-like pipeline, real-time processing out of standard output content processing, similar to the production of consumer mode.

Scene Commentary

The simple process is

程序输出内容到stdout--> python pipe--> python 处理程序读取管道内的输出内容，然后后续处理

Under normal circumstances can be used in the producer consumer mode or queue to get it done, but the first program here is a third party, so only from the stdout start processing, but also what cards play what card it.

Case simulation

A printing method to simulate the STDOUT output program, which is a handler for the Python pipeline.

echo.pyRandom event output in 1 seconds one line, total 10 rows

#coding:utf-8#author：orangleliu#title: echo.pyimport timeimport sysimport1while10:    delay = random.randint(0,100)/100.0#1秒内的随机时间    sys.stdout.write("Talking every %s seconds, blabbed %i times\n" % (delay, ii))    #如果没有flush 后面的程序无法读取的    sys.stdout.flush()    1    time.sleep(delay)

handler.py2 seconds to read the standard output, then processing, slower than the output program

#coding: Utf-8#orangleliu#title: handler.py fromSubprocessImportPopen, PIPE fromOsImportKillImportSignalImportTimetalkpipe = Popen ([' python ',' echo.py '], shell=False, Stdout=pipe)Try: while True: line = Talkpipe.stdout.readline ()ifLinePrint "SERVER HEARD", Line.strip ()Else:Print "No Data"Time.sleep (2)exceptKeyboardinterrupt:Print "Killing child ..."Kill (Talkpipe.pid, signal. SIGTERM)

Summary of results

 TimePython handler.pyserver HEARD talkingevery 0.35Seconds, blabbed1  TimesSERVER HEARD Talkingevery 0.87Seconds, blabbed2  TimesSERVER HEARD Talkingevery 0.31Seconds, blabbed3  TimesSERVER HEARD Talkingevery 0.05Seconds, blabbed4  TimesSERVER HEARD Talkingevery 0.9Seconds, blabbed5  TimesSERVER HEARD Talkingevery 0.74Seconds, blabbed6  TimesSERVER HEARD Talkingevery 0.33Seconds, blabbed7  TimesSERVER HEARD Talkingevery 0.63Seconds, blabbed8  TimesSERVER HEARD Talkingevery 0.47Seconds, blabbed9  TimesNo Datano Datano Datano Datano data^ckilling Child...python handler.py0.03s user0.02s system0% CPU27.522Total

echo.py Total is 10 times printing, each output is within 1 seconds, handler.py 2 seconds to process, you can normal data processing, no data will be taken, according to their own logic to stop processing, or do not do processing, waiting for the next data arrival can be. Basically able to reach the original idea.

Copyright NOTICE: This article is Orangleliu (http://blog.csdn.net/orangleliu/) original article, the article reproduced please declare.

"Python" uses the UNIX pipeline pipe to process stdout real-time data