An example analysis of big data processing in PHP

encountered a problem with a previous version of a large number of data processing reports. I had previously solved this problem with the sync handler code, but it was very slow, which led me to have to extend the maximum script run time for 10-15 minutes. Is there a better way to handle a lot of data in a PHP site? Ideally I want to run it in the background and run as quickly as possible. This process involves processing thousands of financial data, and I use Laravel to reconstruct the site.

The best popular answer (from Spin81):

People are telling you to use queues and stuff like that, which is a good idea, but the problem doesn't seem to be on PHP. Laravel/oop is very powerful, but the process of generating the report you are talking about doesn't seem to be a problem for you. For different views, I would like to see the SQL query you used when you got this data. As other people say, if your form has thousands of lines, your report should not take 10-15 minutes to complete. In fact, if you're not doing anything wrong, you might be able to process thousands of records in a minute and complete the same report.

1. If you are doing thousands of queries, see if you can do only a few queries first. I've previously used a PHP function to reduce the 70,000 queries to more than 10 queries, so it's run time from a few minutes to a fraction of a second.

2. Run explain on your query to see if you are missing an index. I once made a query that increased the efficiency by an index by 4 orders of magnitude, which does not have any exaggerated ingredients. If you are using MySQL, you can learn this, and this "black magic" skill will make you and your little partner stunned.

3. If you are working on a SQL query and then get the results and get a lot of numbers together, see if you can call the group BY statement with functions such as SUM () and AVG (). With the general case, let the database handle as much computation as possible. I can give you a very important hint is: (at least in MySQL) the value of the Boolean expression is 0 or 1, if you are very creative, you can use SUM () and its small partners to do something very surprising.

4. Ok, let's end with a PHP tip: see if you've counted the same time-consuming numbers many times. For example, suppose that the cost of a 1000-bag potato is expensive, but you don't need to calculate that cost 500 times before you store the cost of a 1000-bag potato in an array or other similar place, so you don't have to toss the same thing over and over. This technique, called memorization, is used in reports like yours to have a miraculous effect.
Related recommendations:

PHP website Big Data Large traffic and high concurrency solution

JavaScript Basic tutorial Six Big data type usage detailed

Recommended articles for big Data volume testing