Read large files row by line in Python

In our daily work, inevitably there will be processing log files, when the file is small, the basic need not be careful of what, directly with File.read () or ReadLines () on it, but if it is a 10G size log file read, that is, the file is larger than the size of the memory, There is a problem with this, and the entire file is loaded into memory causing memoryerror ... That is, a memory overflow occurs.

Here are a few ways to share the solution:

Iterate over the file object:

With open (' file_name ', ' r ') as file: For line in    file:        print Line

Advantages:

• With statement, the file object automatically closes the file stream after the code block exits, the file read data is abnormal, and exception capture is handled

• When iterating over a file object, internally, it buffers IO (optimized for expensive IO operations) and memory management, so you don't have to worry about large files.

• This is the Pythonci perfect way to be efficient and fast

Disadvantage: The data content of each row cannot be larger than the memory size, otherwise it will causeMemoryError

Use yield

Normal use of the above way, But if you encounter the entire file only one row, and according to specific characters to split, the above way is not, this time yield is very useful.

Take a chestnut, the form of log is like this.

• 2018-06-18 16:12:08,289-main-debug-do something{|} .....

• with {|} As a separator.

• def read_line (filename, split, size):    with open (filename, ' r+ ') as file:        buff = "while        True: while            spli T in buff:                position = Buff.index (split)                yield buff[:p osition]                buff = buff[(position +len (split)):]            Chunk = file.read (size)            if not chunk:                yield buff                break            buff = Buff +chunk        

Pros: Do not limit the size of each row of data, even if the entire large file has only one row.
Disadvantage: speed is slower than the above method.
To parse:

• First: Define a buffer buff

• The loop is judged if the split delimiter is in the buffer buff, then the Find delimiter appears in the position and yield back.

• Update the buff to continue the second step

• If the split delimiter is not in the buffer buff, the read (size) character

• If the chunk is empty, jump out of the loop, or update the buff to continue the second step

So we need to use that kind of method, generally use the first kind. Encounter only one row of data, and the data is particularly large, it is necessary to consider whether you offend the programmer, deliberately to give you such a file.

Read large files row by line in Python