Web page implements bulk data import function

Scene

        I have a batch of tiled data placed in txt db http 1 hours or more, and waiting so long will cause , nginx Span style= "font-family: the song Body;" > or the browser side (typically 60 minutes) timeout. So what should we do to solve this problem?

Batch Processing

Batch solution means that the big data of the file is converted into blocks, such as 10W pieces , each processing 5K data, so that the time of each processing will be shortened, The user can also see the interactive process in real time without time-out to know the result.

1, the use of local read file data method, the data is passed to the server in batches, the service side takes over the data processing returns, the client obtains the results after the execution of the batch display to the user results.

2, Socket solution, the server and the client use the socket mechanism, the client will pass the file to the server one-time, the server accepts the data after the batch processing, each processing complete part will pass through the socket notifies the front-end portion of the processing result to complete.

In this way, the user must wait to see the results of the execution on the front end, and if the user shuts down, it will cause only some data to be uploaded or processed, but the user can see the running process directly.

Asynchronous process Processing

Asynchronous resolution process is mainly, one-time upload to the server, the server does not execute immediately after receiving the data, but placed in a db queue or asynchronous execution queue, when the execution is completed by callback in the db write execution process, The client only needs to query the results when the user is actively querying.

1. Asynchronously executes by using asynchronous functions , such as some asynchronous file read operations in node. JS, while synchronization prompts the user for file data being uploaded directly.

2, through the queue task mode, if this data upload more frequently, then the recommendation is to use the Queue task mode, the current side of the upload, the task is placed in the task queue, and then need to open a process specifically responsible for the task of reading and parsing the execution process, the completion of execution and then write the results.

This method users can not view the results in real-time, the server only through the size of the file to estimate the execution end time, the best advantage of this scheme is to ensure that the file can complete the upload and data import complete, the disadvantage is that users can not accurately know the end of the execution point of time.

Welcome to provide better solutions.

blog.lovedan.cn

Web page implements bulk data import function