Lazy Loading of PHP and the use of iterator

Lazy Loading of PHP and the use of iterator

Recently changed the work, to build station software, our company built station software template system out of the problem, indicating that the memory exceeds the maximum value. Memory problems The simplest way is to modify the php.ini to improve the memory_limit, but as the number of templates continues to increase, the need for memory exceeded the limit, so the boss put this problem let me see, see what I have a good way.

I get this problem, the first is to understand the original code ideas, analysis of the main cause of the problem, the problem of the page is very simple, is a template display page, each page shows 16 templates, do pagination processing, and the background of processing is not simple, the first template data is not read from the database, Instead, we get a serialized string from the server, take out the array of information for all the templates, then cache the information (read the cache at a later time), then do the filtering and paging in the array, and remove the 16 template display.

Then analyze the problem, it is obvious that the problem here is to remove the information array of all the template, the number of templates is huge, but in fact, each page needs only 16, but this is the implementation of all the memory added.

When the problem is found, how to solve it? So I asked the former staff to listen to their opinions. They provide a lot of advice, have to put the template into the database processing, there will be paging and other operations all put to our server processing, etc., but the program needs to our background server big processing improvement, and the change of things too large, the introduction of more problems, how to do?

1. Lazy Loading

Later I thought, since the problem is serialized out all the data, if we can in the file, only to take out the data we need, because I have Java experience, I first think of is the XML SAX processing, but I am not familiar with the XML processing in PHP, to no avail.

But gave me the correct idea, a lot of PHP memory is too large problem, is due to the PHP developer for convenience, all the required data preloaded into memory (array), and then through processing, the problem is that the required memory far exceeds its actual needs, resulting in an excessive memory problem. The workaround is lazy loading, so that the data is loaded into memory for processing when it is used.

The problem here is clearly the problem of serialization, PHP's serialize mechanism is a pre-loaded policy that does not support lazy loading. Need to find a support for lazy loading serialization, can read only one template each time the content processing, soon CSV format appears, CSV support this behavior unit of reading, and for the original system mechanism does not need to have a big change, so the original system will be CSV serialization changes.

2.Iterator interface Use

CSV reads use while to traverse, and the original system uses an array to traverse, because the system code filters many conditions, involving many code changes, and the previous array using Key=>value to implement, and CSV can only use 0,1,2 digital access. In order to minimize the original code changes, we first processed the CSV, let its first line array of the key value, starting from the second line to display the template actual value value, position and key one by one corresponding, this also satisfies the future extensibility.

To match the foreach style of the original code, we used the iterator encapsulated CSV operation and completed the CSV key->value merge. In this way, the migration of code is almost negligible, and the bigger advantage is that the CSV file size is much larger than the serialized file size, and there may be concerns about performance.

After testing our existing template 2000, the speed is not more than 1 seconds, the number of future templates to increase to 20,000, the traverse speed is not more than 2s, also in an acceptable range. So the problem was solved.

Recently, our self-made Simple ORM Framework has exceeded the maximum value of memory, the reason is to put all the values of the database into an array of memory, so I held the lazy loaded spear, one hand holding the iterator shield, once again into the war to solve the problem.

