The method that comes to mind for improving the external sorting efficiency of a question

Today I read a question "how to improve external sorting efficiency ".

(1) Reduce the number of initial merge segments by half by using the replacement-selection method.

For example, if a group of sorting codes k1> k2>... kn, the buffer zone can accommodate M sorting codes. If in-memory sorting is used, N/m merging segments can be generated.

If you use the replacement-selection method, you can generate n/2 m merge segments.

(2) Reduce the number of merged shards by means of the number of losers, that is, reduce the height of the merged tree.

(3) Improve the concurrency of Io and CPU by using the cache

 

In particular, I would like to mention that in the replacement-selection method, if the buffer can hold M sorting codes, we can get a merge segment of 2 m. The book provides an analogy of a snow sweeper, in fact, you can see this:

In placement-selection, when a sorting code ki1 (the first keyword extracted from segment I) is extracted, A ki2 is read from the file, if ki2 <ki1 (below the ki1 threshold), ki2 is placed in the next merging segment. Otherwise, it is placed in the current merging segment. Therefore, the probability of both is 1/2, the expected record that can be selected in Section I is:

 

1 + 1/2 + 1/4 + 1/8 +... = 1/(1-1/2) = 2

 

If there are m segments (the buffer can hold m sort codes, so it can be regarded as M segments), the number of records that can be selected is 2 * m.

 

The method of proof is more intuitive than that of the anti-snow machine. Today I want to write it down for you to see.