In web development. Download the report above the server. How to prevent a scenario where the data volume is too large to cause timeouts

Problem

The report needs to be generated by the server, because the amount of data is too large. Will often cause 504 errors during the download.
The download process has been optimized.

Current Solutions

The front desk clicks on the download report to initiate an asynchronous request. After the data is processed in the background. Send data to the downloader via email.

Is there any other better way to solve this problem? Thank you.

Reply content:

Problem

The report needs to be generated by the server, because the amount of data is too large. Will often cause 504 errors during the download.
The download process has been optimized.

Current Solutions

The front desk clicks on the download report to initiate an asynchronous request. After the data is processed in the background. Send data to the downloader via email.

Is there any other better way to solve this problem? Thank you.

Thank you for your answer.
Here I summed up my own, according to my own actual situation mainly

1. The foreground asynchronously submits a request to the background for the asynchronous composite report. then through 1. The way the message is notified to the user 2. Push the message to the user via the timer and let the user go to the corresponding address to download
2. The query time is accurate to seconds. Before the user requests, let the user know the number of orders in this time period. We can set a reasonable value to prompt the user (minimal changes)
3. Take the form of a task, the foreground submits a task to the background, creates a task in the background, and stores it in the database. The foreground can request the status of the task every 30s to the back desk to prevent timeouts. The corresponding foreground has a loading box (which can also be changed to put the file generation process on the server side, and then return to the user is a connection, the user can directly connect to download files) (This is similar to the use of Ajax Long connection mode)
4. Put the analysis process on the service side. The service has a timed task that analyzes the data during intervals. Direct production files are placed on the server side. can provide users with a download list directly to the user to download
   5. Optimize the process and take Dubbo bulk data requests. Take Dubbo asynchronous request. Multi-threaded processing of data. Multiple Dubbo Services Consolidation (performance optimization)

The most straightforward approach is to compress the data (for example, to compress on export) and adjust the CGI time-out. This can be considered to save the CGI process (because even if the output has been completed, according to the owner's instructions, the fast_cgi response buffer is certainly much smaller than the size of this output), in the final output of the process with other services (such as directly with Nginx).

In fact, for example in the problem of this from the perspective of product design to solve the problem in my opinion, the best solution, real-time download is really so necessary?

The report needs to be generated by the server, because the amount of data is too large. Will often cause 504 errors during the download.

As you mean, the reason for the timeout is not that it takes too long to produce a report, but because the report file is too large.

3 Ways of doing

1. Compress the report file, for example: Tar,gzip,zip ...

2. Modify the time-out settings for the Web server.

3. The exported file is placed in another directory and downloaded with other tools, for example: Ftp,samba,sftp ...

While reading data, writing data, such as paging and reading and writing, so that it does not time out

In a different way, all your problems are over-timed! Why timeout Because the wait time is too long. Then why wait???

You can make a request, the server side executes, tells the user, is executing, the expression succeeds. Then the user waits for the message to be accepted.

This time, you can tell the user to wait, or add a message to notify the user after success

