A way for Linux shells to read files line by row

A way for Linux shells to read files line by row

In Linux there are many ways to read a file row by line, the most common is the method in the following script, and is the most efficient, most used method. In order to give you an intuitive feeling, we will test the execution efficiency of various methods by generating a large file.

Method 1:while the most efficient and most commonly used method in the loop.

function While_read_line_bottm () {
While Read line
Do
Echo $LINE
Done < $FILENAME
}

Note: I'm used to calling this method read drastic, because it takes a file to be executed at the end, as if it were to be read in after execution.

Method 2: Redirect method; Piping method: Cat $FILENAME | While Read line

Function While_read_line () {
Cat $FILENAME | While Read line
Do
Echo $LINE
Done
}
Note: I only call this method the pipe method, compared to everyone should be able to see it. When the pipeline is met, the output of the command on the left side of the pipeline is entered as input to the command on the right side of the pipeline.

Method 3: File Descriptor method

Function while_read_line_fd () {
Exec 3<&0
Exec 0< $FILENAME
While Read line
Do
Echo $LINE
Exec 0<&<3
}

Note: This method is divided into 2 steps, first, to close the file descriptor 0 by redirecting all content to the file descriptor. For this we used the grammar exec 3<&0. The second part broadcasts the input file to the file descriptor 0, which is the standard input.

Method 4 for Loop.

function For_in_file () {
For i in ' Cat $FILENAME '
Do
Echo $i
Done
}
Note: This is a way to read the contents of a file through a for loop rather than everyone is familiar with, here not much to say.


Test each method to see that the method performs the most efficiently.

First, we use the script (see the script attached) to generate a 70000-line file, the file location in/scripts/bigfile. Then test the execution efficiency of each method with the following script, which is simple and no longer explained.

#!/bin/Bashfilename=" $"Timefile="/tmp/loopfile.out">$TIMEFILE SCRIPT=$(basename$0) functionusage () {Echo-E"\nusage: $SCRIPT file \ n"Exit1} functionWhile_read_bottm () { whileRead Line DoEcho$LINE Done<$FILENAME}functionWhile_read_line () {Cat$FILENAME | whileRead Line DoEcho$LINE Done }  functionwhile_read_line_fd () {exec3<&0exec0<$FILENAME whileRead Line Do  Echo$LINE Doneexec0<&3} functionFor_in_file () { forIinch`Cat$FILENAME ' DoEcho$i Done} if[$#-lt1] ; Thenusagefi Echo-E"\ Starting file processing of each method\n" Echo-E"Method 1:" Echo-E"function While_read_bottm"  TimeWhile_read_bottm >>$TIMEFILEEcho-E"\ n" Echo-E"Method 2:"Echo-E"function While_read_line" TimeWhile_read_line >>$TIMEFILEEcho-E"\ n"Echo-E"Method 3:"Echo "function While_read_line_fd" TimeWHILE_READ_LINE_FD >>$TIMEFILEEcho-E"\ n"Echo-E"Method 4:"Echo-E"function For_in_file" TimeFor_in_file >> $TIMEFILE

After executing the script: [[email protected] shell]#./while/scripts/bigfile
What the script outputs:
Method 1:
function While_read_bottm

Real 0m5.689s
User 0m3.399s
SYS 0m1.588s


Method 2:
function While_read_line

Real 0m11.612s
User 0m4.031s
SYS 0m4.956s


Method 3:
function while_read_line_fd

Real 0m5.853s
User 0m3.536s
SYS 0m1.469s


Method 4:
function For_in_file

Real 0m5.153s
User 0m3.335s
SYS 0m1.593s


Let's sort by speed for each method.
Real 0m5.153s Method 4 (For loop method)
Real 0m5.689s Method 1 (while drastic method)
Real 0m5.853s Method 3 (identifier method)
Real 0m11.612s Method 2 (Pipeline method)

This shows that in each method, the For statement is the most efficient, while reading and writing files in the while loop,
While Read line
Do
Echo $LINE

Done < $FILENAME

The most efficient way to execute.