WEB development in progress. Download the Report on the server. How to Prevent timeout caused by excessive data volume

The server must generate a report because the data volume is too large. The download process may cause 504 errors. The download process has been optimized. Currently, in the solution, click Download Report at the front end to initiate an asynchronous request. After the data is processed in the background. Send the data to the downloader by email. Yes...

The server must generate a report because the data volume is too large. The download process may cause 504 errors.
The download process has been optimized.

Current solution

Click Download Report in the foreground to initiate an asynchronous request. After the data is processed in the background. Send the data to the downloader by email.

Is there any better way to solve this problem? Thank you.

Reply content: Question

The server must generate a report because the data volume is too large. The download process may cause 504 errors.
The download process has been optimized.

Current solution

Click Download Report in the foreground to initiate an asynchronous request. After the data is processed in the background. Send the data to the downloader by email.

Is there any better way to solve this problem? Thank you.

Thank you for your answers.
I have summarized it here, based on my actual situation, mainly

1. The foreground asynchronously submits a request and the background asynchronously combines the report. Then, notify the user by email. 2. PUSH the message to the user through the timer, and let the user download the message from the corresponding address.
2. The query time is accurate to seconds. Before the user requests, inform the user of the number of orders in this period. We can set a reasonable value to prompt the user (minimum change)
3. In the form of a task, the foreground submits a task to the background, creates a task in the background, and saves it to the database. The foreground can send a request to the background every 30 seconds to obtain the task status to prevent timeout. The corresponding front-end has a loading box (this can also be changed to put the file generation process on the server side, and then return a connection to the user, and the user can directly click the connection to download the file) (This is similar to the AJAX persistent connection mode)
4. Put the analysis process on the server. The server starts a scheduled task to analyze the data in the interval. Directly store production files on the server. A download list can be provided to users and directly provided to users for download.
   5. Optimize the process and adopt dubbo batch data requests. Adopt dubbo asynchronous request. Multi-threaded data processing. Merge multiple dubbo services (performance optimization)

The most direct method is to compress data (for example, compress data during export) and adjust the CGI timeout time. In this case, you can consider saving the CGI process (because even if the output result has been completed, the response buffer of fast_cgi is definitely much smaller than the output size according to the subject's description ), in the final output process, use other services (for example, directly use nginx.

In fact, for example, if the problem is solved from the product design perspective in my opinion, is real-time download really necessary?

The server must generate a report because the data volume is too large. The download process may cause 504 errors.

The reason for the timeout is not that it takes too long to generate a report, but because the report file is too large.

Three methods

1. Compress report files, such as tar, gzip, zip ......

2. Modify the timeout settings of the web server.

3. Place the exported file in another directory and download it using other tools, such as ftp, samba, sftp ......

Read data while writing data, such as paging read/write, so that no timeout will occur.

In another way, all your problems are timeout! Why does the timeout occur because the wait time is too long. Why wait ???

You can send a request, and the server executes the request, telling the user that the request is being executed, indicating that the request is successful. Then the user waits for receiving the email.

At this time, you can tell the user to wait, or add a Message notification to notify the user after the success