Php interface-PHP Data Interface Design

Service A provides million log data per day, and B needs to write an interface to capture data of Service A (assuming that million of A can be simulated as retrieved from the database ), now we need to design an interface to ensure that the interface can quickly obtain million data, and that the sudden interruption of data acquisition can be interrupted... service A provides million log data per day, and B needs to write an interface to capture data of Service A (assuming that million of A can be simulated as retrieved from the database ), now, you need to design an interface to ensure that the interface can quickly obtain million data. when the data is suddenly interrupted, you can continue to obtain the data from the breakpoint, and at the same time ensure the interface security.

Reply content:

Service A provides million log data per day, and B needs to write an interface to capture data of Service A (assuming that million of A can be simulated as retrieved from the database ), now, you need to design an interface to ensure that the interface can quickly obtain million data. when the data is suddenly interrupted, you can continue to obtain the data from the breakpoint, and at the same time ensure the interface security.

I have done the same data statistics service,
A service is A server (Server)
B service, on the other server (Server B)

The final solution is to store the data of server A through A file,
Then use the script (Php curl) Or simplyrsyncThe command is synchronized to server B,

Server B then scans the file content andData Archiving,Deduplication,Warehouse receiving..

There is still a small amount of data. At first glance, I thought it was MB.

The solution agrees with @ sunwenzheng's proposal. Redis queue is used to solve the problem.

The solution is as follows:

1. Set up A queue service on server B (not set up on server A because your server A may be the master server, reducing its pressure)
2. server A immediately pushes the generated logs to the Redis queue of server B.
3. Server B polls the queue and inserts collected data into the database.

The advantage of this is that it ensures the sequence of log records and can control the validity of Log retrieval. More than million data records are supported.

Resumable data transfer is required for such a large amount of data. Fortunately, it is log data and the real-time performance should not be high.
You can consider letting service A regularly export data to A file, and then Service B directly downloads data through ftp/sftp. The ftp speed is faster enough, you can create an NFS shared file. (Resumable data transfer is supported)

Does the interface provided by the other Party have a time parameter or something? If yes, batch requests can be made, and the last time is recorded each time. The next request uses this time as the condition.
You can also write the received data to the redis queue, while another process reads data from the redis queue and writes it to the database in batches.
If you can use text, you can also use rsync for synchronization.

The API transmission efficiency of such big data is too low.
* Export the daily database directly to a file (if the data volume is too large, you can export one file or 10 million files per hour)
* Server B then obtains the data through Http (Http resumable data transfer is easy to implement)
* After obtaining the data, decompress the package and import it to the database.
The advantage of doing so is that the implementation is relatively simple and error-free.

You can consider using paging. For example, if I process million records at a time, the data on the next page will be processed. You can obtain the total number and then generate a queue task. Attackers can obtain data by relying on the server.