How much does shell command Buffer know about playing the shell?

1. Questions:

in the afternoon, a classmate asked such a question:

Tail-n +$ (tail-n1/root/tmp/n)-f/root/tmp/ip.txt 2>&1| awk ' argind==1{i=$0;next}{i++;if ($0~/file truncated/) {i=0};p rint "---" I;print i >> "/root/tmp/n"} '/root/tmp/n-  Seq >/root/tmp/ip.txt && tail-f/root/tmp/n

The two statements are executed on the two terminals of the same machine, and you will find that the tail of the second statement is not traceable to the result, and the first statement clearly has the result output.

let's start with a brief introduction to the first statement:

This statement is a real-time tail a log, and implements two small functions:

when the file is rewritten, the line number of the file is set to 0, and when the process is hung, the process can be restarted from the last place to start tail, similar to "Breakpoint continuation."

students who are unfamiliar with awk seem to be struggling, so let's simplify the scenario and write a simple test case to simulate the statement above (1):

{echo 21;sleep 10;echo 22;}| awk ' {print >> '/root/tmp/n '} '

you will find that the value of n does not change when the screen output is 21, but when the entire echo execution is complete, the value of n changes, outputting 21, 22.

for this, you can easily write another test case:

while [[$i-lt 10]]; Do ((i++)); echo $i |awk ' {print >> '/root/tmp/n '} '; Sleep 2; Done

So why does this case see the value of N changing in real time? Don't worry, after reading this article, you will find the answer yourself. ^ _ ^

the problem with the statement (1) is that a concept under the shell triggers: buffer

The students who wrote the program should know that the disk and memory, memory and CPU IO interaction speed is not a magnitude, then in order to improve the efficiency of data access, generally in the software program, hardware design using the buffer design, when the buffer full will be requested once IO operation, Instead of a character or a byte of the way to request IO operation, specifically generally interactive will be no buffer or small buffer, non-interactive operation generally buffer will be relatively large, because the user "real-time" requirements are not so high ~ ~

the redirection of the statement (1) is a typical non-interactive operation, and the user cannot see the changes in the log data in real time due to the buffer's cause.

2. Solution:

knowing why, there are several ways in which redirects in awk can be output without buffer in real time:

{echo 21;sleep 10;echo 22;}| awk ' {print >> '/root/tmp/n '; Fflush ("")} ' {echo 21;sleep 10;echo 22;}| awk ' {print >> '/root/tmp/n '; System ("")} ' {echo 21;sleep 10;echo 22;}| awk ' {print >> '/root/tmp/n '; Close ("/root/tmp/n")} ' {echo 21;sleep 10;echo 22;}| awk ' {System ("echo" $ >>/root/tmp/n ")} ' {echo 21;sleep 10;echo 22;}| awk ' {print | ' Cat->>/root/tmp/n "} '

a description of Fflush is as follows:

fflush ([file]) Flush any buffers associated with the open output file or pipe file. If file is missing and then standard output is flushed. If file is the null string, then all open output files and pipes has their buffers flushed.

here, there are some students may have doubts: there is still any way to verify the cause of buffer?

In fact, you can increase your output on the line:

{seq 5000;sleep 10;seq 1000;}| awk ' {print >> '/root/tmp/n '} '

3. By and by

In fact, many of the commands under the Linux shell are designed with a buffer, such as grep, for example, once a classmate asked me:

tail-f logfile | grep ' Ooxx ' Why can't you see the results? Is there something in the log? and so on ...

In this article, we summarize the buffer problem of common commands and the countermeasures:

grep (e.g. GNU version 2.5.1)	--line-buffered
sed (e.g. GNU version 4.0.6)	- u,--unbuffered
awk (GNU awk)	Use the fflush () function
awk (Mawk)	- W Interactive
tcpdump, Tethereal	- L

For example, the grep buffer problem mentioned above:

Tail-f/var/log/foo | grep--line-buffered

There are also special commands or toolkits to solve this problem, such as Stdbuf, unbuffer,stdbuf, or simply call the C library to disable buffer:

Setvbuf (stdout, 0, _IONBF, 0);

4, Refer:

[1] 9.1.4 input/output Functions

Https://www.gnu.org/software/gawk/manual/html_node/I_002fO-Functions.html

[2] What is buffering? Or, why does my command line produce no output:tail-f logfile | grep ' foo bar ' | Awk...

http://mywiki.wooledge.org/BashFAQ/009

[3] How to fix stdio buffering

Http://www.perkin.org.uk/posts/how-to-fix-stdio-buffering.html

[4] buffering in standard streams

http://www.pixelbeat.org/programming/stdio_buffering/

[5] about the pipeline shutdown problem in awk after executing a shell through a pipeline

http://hi.baidu.com/leejun_2005/item/26a5f8273e7e3555c28d5970

[6] Why does awk does full buffering if reading from a pipe

Http://unix.stackexchange.com/questions/33650/why-does-awk-do-full-buffering-when-reading-from-a-pipe

How much does shell command Buffer know about playing the shell?