Read all data at once and limit read data in batches which is good.
At the time of collection, such as the database has 100,000 records URL is the list page URL
1, one-time from the database read out the 100,000 URLs, and then foreach to collect
2, first limit10 bar, foreach collection, and then read the database limit, and constantly repeat
What kind of method is generally used?
Share to:
------Solution--------------------
Collection and analysis separation?
If one side of the collection side analysis, batch
If the separation is one time, of course, the machine can withstand
------Solution--------------------
If you are allowed, why not read it once?
------Solution--------------------
100,000 reads quickly, with limit is how to think out, more trouble, will not improve efficiency.
------Solution--------------------
What does this add limit to do, is not a database query. What efficiency can you improve?
------Solution--------------------
Generally I would suggest reading in batches. If it is important to count a large data set, the best way to do this is to return the statistical results after the server-side has counted the stored procedures. Although the network speed and stability are greatly improved at present, can be done on the server side on the server side do generally always a wrong approach.
But according to the scenario you describe, you need to collect the server-side 100,000 URLs, then I suggest you might as well save a cache of that database locally.
------Solution--------------------
100,000 data, one-time read, no problem, the key is how efficient your collection program.
