How to read large files using Python
Background
When I recently processed a text File (the File size is about 2 GB), there was a memoryError error and the File Reading was too slow. Later I found two methods for fast Large File Reading, this article describes the two read methods.
Preparations
When talking about "text processing", we usually refer to the content to be processed. Python reads the content of a text file into a string variable that can be operated easily. The file object provides three "read" Methods:. read (),. readline (), and. readlines (). Each method can accept a variable to limit the amount of data read each time, but they usually do not use variables. . Read () reads the entire file every time. It is usually used to put the file content into a string variable. However. read () generates the most direct string representation of the file content, but it is unnecessary for continuous row-oriented processing. If the file is larger than the available memory, this processing is impossible. The following is an example of the read () method:
Try:
F = open ('/path/to/file', 'R ')
Print f. read ()
Finally:
If f:
F. close ()
Calling read () will read all the content of the file at a time. If the file has 10 Gb, the memory will pop up. Therefore, you can call the read (size) method repeatedly to ensure security, read a maximum of size bytes each time. In addition, you can call readline () to read a row of content at a time, call readlines () to read all the content at a time, and return the list by row. Therefore, you need to decide how to call it as needed.
If the file is small, read () is the most convenient one-time read. If the file size cannot be determined, it is safer to call read (size) repeatedly. If it is a configuration file, it is the most convenient to call readlines:
For line in f. readlines ():
Process (line )#
Multipart read
It is easy to think of splitting a large file into several small files for processing. After processing each small file, this part of memory is released. Iter and yield are used here:
Def read_in_chunks (filePath, chunk_size = 1024*1024 ):
"""
Lazy function (generator) to read a file piece by piece.
Default chunk size: 1 M
You can set your own chunk size
"""
File_object = open (filePath)
While True:
Chunk_data = file_object.read (chunk_size)
If not chunk_data:
Break
Yield chunk_data
If _ name _ = "_ main __":
FilePath = './path/filename'
For chunk in read_in_chunks (filePath ):
Process (chunk) # <do something with chunk>
Use With open ()
With statement to open and close the file, including throwing an internal block exception. For line in f file object f is regarded as an iterator and will automatically adopt buffer IO and memory management, so you don't have to worry about large files.
The Code is as follows:
# If the file is line based
With open (...) as f:
For line in f:
Process (line) # <do something with line>
Optimization
There is no problem with using with open for millions of rows of large data, but different parameters here will also lead to different efficiency. After testing, when the initial parameter is "rb", the efficiency is six times that of "r. We can see that binary reading is still the fastest mode.
With open (filename, "rb") as f:
For fLine in f:
Pass
Test results: the rb mode is the fastest, with 2.9 rows traversing for seconds. It can basically meet the needs of medium and large file processing efficiency. If the read speed is changed from rb (second-level read) to r (read mode), the speed is 5-6 times slower.
Conclusion
When using python to read large files, you should let the system handle them. The simplest way is to hand it over to the interpreter to manage your work. At the same time, you can select different read parameters to achieve higher performance based on different requirements.