Python handles memory problems with large files

Summary:

Students often encounter the situation to deal with large files, now is the big Data age, some files frequently dozens of g, we in the processing of such a file when the memory is accidentally burst, or the program was forced to kill off.

The reason is that you can read all the contents of the file into memory at once. Python has a way to read a file over a period of time.

Body:

Yes, it is using iterator, also known as iterators, the example code is as follows.

Cat test.py

f = open (' Data ', ' R ')

For line in F:

line = Line.split (";;")

Lines.append (line)

If Len (lines) >= 10000:

# Consume the lines which has been read

Print lines

Del lines[:]

If lines:

# Consume the lines which has been read

Print lines

Cat data

73231701-201610;; shop_id::::73231701;; Shop_name:::: Encounter Phantom;; Platform_name::::xxxx;; Shop_type:::: individual sellers;; Shop_loc::::xxx;; Gold_seller:::: No;; Market_name::::taobao;; Description::::null;; Service::::null;; Logistics::::null;; Shop_owner:::: Yangyang 103105;; create_time::::2012-08-15;; Credit:::: Love 4;; Shop_age::::4;; Co_name::::null;; Shop_link::::https://shop73231701.example.com

73295319-201610;; shop_id::::73295319;; Shop_name:::: Beautiful for you the most beautiful; Platform_name::::xxxx;; Shop_type:::: individual sellers;; Shop_loc::::xxx;; Gold_seller:::: No;; Market_name::::taobao;; Description::::null;; Service::::null;; Logistics::::null;; shop_owner::::chenyan121166563;; create_time::::2012-08-20;; Credit::: Diamond 3;; Shop_age::::4;; Co_name::::null;; Shop_link::::

Https://shop73295319.example.com

The above file will actually be very long, I just wrote two lines here, for reference only.

"For lines in F" will only read a row of data to memory each time, we can set a buffer, for example, every 10000 rows with the list of the temporary storage, processing finished before continuing to read the file.

This allows for a period of time to read the contents of the file into memory. Isn't it cool!

Give it a try!

This article from "Linux operation and Maintenance" blog, declined reprint!

Python handles memory problems with large files