Which of the following is the optimal solution for loop reading with paging interface?

Currently, third-party interface data needs to be read. the data volume is large, and the third-party only uses the paging method. Currently, my solution is to call the curl read interface method in a loop, convert json data to an array and filter and store the data in the spliced array. However, the third-party interface does not... currently, third-party interface data needs to be read. the data volume is large, and the third-party only uses the paging method. Currently, my solution is to call the curl read interface method in a loop, then, convert json data into arrays and filter and store the data in the spliced array. However, the read may fail due to the instability of third-party interfaces. Is there a better solution?

Reply content:

Currently, third-party interface data needs to be read. the data volume is large, and the third-party only uses the paging method. Currently, my solution is to call the curl read interface method in a loop, then, convert json data into arrays and filter and store the data in the spliced array. However, the read may fail due to the instability of third-party interfaces. Is there a better solution?

If the data volume is large, you can use a scheduled script to pull the data.
When pulling data, sort it by a field such as auto-increment id to avoid paging data changes.
Then, when the script is executed or when the loop ends, write down the largest id, and then add the condition greater than this value to the next execution.

When the script is executed, if the interface call fails, you can try to pull the script several times again. if the script fails, stop the script execution, write down the id, and then give the warning information and manual intervention.
Amount, simple ideas

1. it is recommended that you create a grabbing source database, and the id or md5 should be good to ensure the uniqueness of a single data capture.
2. the interface is unstable. you can handle curl failure exceptions to ensure data capture is successful.
3. Prepare for repeated data capturing. based on 1, ensure that the data is not processed repeatedly.

The above is what kind of content is the optimal solution for loop reading with paging interface. For more information, see PHP Chinese network (www.php1.cn )!