4 ways for the shell to read files line-by-row _linux shell

Source: Internet
Author: User

There are many ways to read one file at a row in Linux, the most common of which is the method in the following script, and the most efficient and most used method. In order to give you an intuitive feeling, we will test the execution efficiency of various methods by generating a large file.

Method 1:while the most efficient and commonly used method in the loop.

Copy Code code as follows:

function While_read_line_bottm () {
While Read line
Todo
Echo $LINE
Done < $FILENAME
}

Note: I'm used to calling this a read, because it requires the execution of a file at the end of it, like reading the file when it's done.

Method 2: Redirect method; Piping method: Cat $FILENAME | While Read line

Copy Code code as follows:

Function While_read_line () {
Cat $FILENAME | While Read line
Todo
Echo $LINE
Done
}

Note: All I have to do is call this a plumbing method, which we should be able to see. The output of the command on the left side of the pipe will be input to the command on the right of the pipe when the pipe is met.

Method 3: File Descriptor method


Copy Code code as follows:

Function while_read_line_fd () {
Exec 3<&0
Exec 0< $FILENAME
While Read line
Todo
Echo $LINE
Exec 0<&<3
}

Note: This method is divided into 2 steps, first, by redirecting all content to the file descriptor 3来 closes the file descriptor 0. For this we used the syntax of exec 3<&0. The second part is to broadcast the input file to the file descriptor 0, the standard input.

Method 4 for Loop.


Copy Code code as follows:

function For_in_file () {
For i in ' Cat $FILENAME '
Todo
Echo $i
Done
}

Note: This is the way to read the contents of a file through a for loop, which is not much said here. Test each method to see that the method is performing the most efficient.

First we use a script (see the attachment) to generate a 70000-line file, the file location in/scripts/bigfile. Then test the execution efficiency of each method through the following script, which is simply not explained.

Copy Code code as follows:

#!/bin/bash
Filename= "$"
timefile= "/tmp/loopfile.out" > $TIMEFILE
script=$ (basename $)
function usage () {
Echo-e "\nusage: $SCRIPT file \ n"
Exit 1
}
function While_read_bottm () {
While Read line
Todo
Echo $LINE
Done < $FILENAME
}
function While_read_line () {
Cat $FILENAME | While Read line
Todo
Echo $LINE
Done
}
function while_read_line_fd () {
EXEC 3<&0
EXEC 0< $FILENAME
While Read line
Todo
Echo $LINE
Done
EXEC 0<&3
}
function For_in_file () {
For i in ' Cat $FILENAME '
Todo
Echo $i
Done
}
If [$#-lt 1]; Then
Usage
Fi
Echo-e "\ Starting file processing of each method\n"
Echo-e "Method 1:"
Echo-e "Function While_read_bottm"
Time While_read_bottm >> $TIMEFILE
Echo-e "\ n"
Echo-e "Method 2:"
Echo-e "Function While_read_line"
Time While_read_line >> $TIMEFILE
Echo-e "\ n"
Echo-e "Method 3:"
echo "Function While_read_line_fd"
Time WHILE_READ_LINE_FD >> $TIMEFILE
Echo-e "\ n"
Echo-e "Method 4:"
Echo-e "Function For_in_file"
Time For_in_file >> $TIMEFILE

After executing the script: [Root@localhost shell]#./while/scripts/bigfile
Script output:

Copy Code code as follows:

Method 1:
function While_read_bottm
Real 0m5.689s
User 0m3.399s
SYS 0m1.588s
Method 2:
function While_read_line
Real 0m11.612s
User 0m4.031s
SYS 0m4.956s
Method 3:
function while_read_line_fd
Real 0m5.853s
User 0m3.536s
SYS 0m1.469s
Method 4:
function For_in_file
Real 0m5.153s
User 0m3.335s
SYS 0m1.593s

Here's how to sort each method by speed.
Copy Code code as follows:

Real 0m5.153s Methods 4 (For loop method)
Real 0m5.689s methods 1 (while algorithm)
Real 0m5.853s Methods 3 (identifier method)
Real 0m11.612s Methods 2 (Piping method)

This shows that in each method, the For statement is the most efficient, while in the while loop, when reading and writing files,
Copy Code code as follows:

While Read line
Todo
Echo $LINE
Done < $FILENAME

The method performs the most efficiently.

Related Article

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.