How PHP asynchronously generates an Excel file and saves it to the server

Let's talk about my current needs and dilemmas:

• 1. Export order This function believes that we are not unfamiliar with the large amount of data and complex business. Now the product needs to export one months of data, about 2w, the query interface calls is the Java-provided API.

• 2. Query 2000 data This interface requires a second, and the server time-out is 30 seconds, so time-out is unavoidable. 1w data per 1000, a single curl request, takes 60-78 seconds, Curl_multi request, 45.78-52 seconds.

• 3. The business logic is not completed at this time, the user's information needs to be returned in the results of the query database based on the UID, some fields also need to be judged, so you need to wait for rest results and processing logic.

• 4. The processed data results are then traversed and assembled into Excel cells, which takes up a lot of memory.

Now talk about my own thoughts:

• 1. The client initiates an "export" action, the service sends a query operation, and the generated Excel file is saved to the server side.

• 2. After the client issued the action, it is best to return immediately, do not wait for the results to repay the customer, directly to the customer, half an hour before the download.

• 3. Download the generated Excel file directly from the server.

My confusion:

How to make the customer initiate the request and return immediately, but also the server to do business logic, processing queries, field assembly and other processes? Please the great God help me to guide twos, thank you!

Add one point:
Originally want to use cron job, but things are more and more, the server is not very well-off, the above deployment of many services, these things in their own internal use, efficiency does not require very high.

Because the app side shows a lot of things are scheduled tasks, such as product refinement, too many, so, currently do not want to use cron job. and using spring's task in Java code performs many tasks.

Reply content:

Let's talk about my current needs and dilemmas:

• 1. Export order This function believes that we are not unfamiliar with the large amount of data and complex business. Now the product needs to export one months of data, about 2w, the query interface calls is the Java-provided API.

• 2. Query 2000 data This interface requires a second, and the server time-out is 30 seconds, so time-out is unavoidable. 1w data per 1000, a single curl request, takes 60-78 seconds, Curl_multi request, 45.78-52 seconds.

• 3. The business logic is not completed at this time, the user's information needs to be returned in the results of the query database based on the UID, some fields also need to be judged, so you need to wait for rest results and processing logic.

• 4. The processed data results are then traversed and assembled into Excel cells, which takes up a lot of memory.

Now talk about my own thoughts:

• 1. The client initiates an "export" action, the service sends a query operation, and the generated Excel file is saved to the server side.

• 2. After the client issued the action, it is best to return immediately, do not wait for the results to repay the customer, directly to the customer, half an hour before the download.

• 3. Download the generated Excel file directly from the server.

My confusion:

How to make the customer initiate the request and return immediately, but also the server to do business logic, processing queries, field assembly and other processes? Please the great God help me to guide twos, thank you!

Add one point:
Originally want to use cron job, but things are more and more, the server is not very well-off, the above deployment of many services, these things in their own internal use, efficiency does not require very high.

Because the app side shows a lot of things are scheduled tasks, such as product refinement, too many, so, currently do not want to use cron job. and using spring's task in Java code performs many tasks.

2, one of which is to use a script to run. Another is to receive a front-end request to return directly or let it time out, PHP can continue to execute the Display settings script set_time_limit (0), php-fpm words can be fastcgi_finish_request (), return to continue execution, Apache can set Ignore_user_abort ();

However, the latter way PHP-FPM will block the PHP receive request, it may lead to normal users can not access the site, because that is randomly assigned, if Nginx connected to that long process of the blocked.
It is recommended to do this in PHP script, use a queue, or use the shell to invoke a PHP script to execute

Can be exported through a queue
Can meet your needs for my export

Or you can choose to simply point it out with a Linux scheduled task

The foreground submits the export request, you add the exported request to the Database task table, and return the information to the client that is performing the export, and then the Linux scheduled task queries the task table to the new task and exports it, and it is not particularly complicated to modify the current task to complete after the export is completed.

Classmate, have you ever heard of cron job?

You can use the Distributed task queue
For example, Gearman, RABBITMQ asynchronous processing.

Why not use the inherently asynchronous NODEJS implementation?

