Which of the following statements can I use to retrieve all data at one time and read data in batches by limit?

One-time read-out of all data and batch read of data by limit which is good at data collection, for example, the database has 0.1 million record URLs which are the URLs on the list page

1. read the 0.1 million URLs from the database at a time, and then perform foreach collection.
2. limit 10 items first. after foreach completes collection, read the database limit and repeat it.

Which method is generally used?

Reply to discussion (solution)

Is collection and analysis separated?
If one side is collected, one side is analyzed in batches.
If the separation is performed once, it depends on the machine's affordability.

If you? Save ?,? What? Not once? Outgoing

The reading speed of 0.1 million entries is very fast. how can I use limit? it is too troublesome and will not improve the efficiency.

What should I do with limit? it's not a database query. Can it improve the efficiency.

Generally, I recommend reading data in batches. If you must make statistics on a large dataset, the best way is to use the stored procedure on the server side for statistics and then return the statistical results. Although the current network speed and stability are greatly improved, what can be done on the server side is generally not wrong.

However, according to the scenario you described, if you need to collect 0.1 million URLs on the server side, it may be better to save a local cache for the database.

There is no problem with reading 0.1 million pieces of data at a time. The key is how efficient your collection program is.