Select Displace + loser tree to fix the external sort

Personally, the external ordering is our learning process in contact with a more important algorithm, it contains both the basic sorting algorithm, also examines the file IO and memory understanding, but also shows the most basic program optimization ideas, can say that the ability to write an external sort, it shows that the basic programming skills have passed the border. This article will conduct a detailed analysis of the entire external ordering process, and introduce two classical algorithms, and finally attach the complete program code.

1. Description of the problem

Because in the real world, sometimes it is necessary to sort a very large file, and the computer memory is limited, when the data can not be fully stored in memory, you can not use the normal sorting algorithm one time to complete the sorting, but must take advantage of disk space auxiliary for external sorting, That is, the use of a limited amount of memory each time the data is read into a sequence after a straight string to the disk, and finally a plurality of strings to merge until the final sorting, because in the process of merging, only the smallest data from each string to compare, and do not need the entire string in memory, This solves the problem of insufficient memory space. Then, the original problem can be decomposed into two sub-problems, one is how to generate a string, and the other is how to merge the string.

First, in terms of performance, because disk IO is hundreds of thousands of times times slower than memory reads, you must minimize the number of disk IO times. Consider the merger process, assuming that there are 8 straight strings, each merge two, then the first round merged into 4, the second turn into 2, until the third round to complete the merge, in this process for each data 3 Io, and if you can merge 8 straight strings, then only one round to complete, That is, only 1 Io is made for each data. Therefore, in order to improve the efficiency of the process, it is necessary to minimize the number of rounds in the merger, to achieve this, you can start from two angles, one is to reduce the number of even if (the length of each shun string as long as possible), the second is the use of multiple merge, for these two points, this article will be selected by the replacement algorithm

2. Select Displacement

The selection permutation algorithm is used to generate a straight string, under limited memory limit, it can generate approximately twice the memory size of the following string, the algorithm steps are as follows:

Suppose there is only one array in memory that can hold n integers

(1) First read n numbers from the input file to fill the array

(2) Building a minimum heap using existing data in an array

(3) Repeat the following steps until the size of the heap becomes 0:

A. Output the number A of the root node (that is, the minimum value in the current array)

B. read a second number B from the input file, if R is larger than the number a just output, place b at the root of the heap, and if B is not larger than a, move the last element of the heap to the root node, place B in the last position of the heap, and reduce the heap size by 1 (that is, the newly read data does not enter the heap)

C. Calling Siftdown at the root node to re-maintain the heap

(4) Replace an output file and go back to step (2)

Explanation: During the operation of the above algorithm, step (3) each from the minimum heap output a minimum value, from the input file is read into a data, if the number of new reads than the number of just output, you can belong to the current string, put it into the heap, otherwise it can only belong to the next straight string, need to put it outside the heap, in the process of running, The size of the heap is gradually reduced to 0, and a straight string is output, and the new number in the array can be used to construct a new heap so that the original large file is converted into a string of about 2N. As for why 2N, there is a more abstract analogy to prove:

Suppose there is a snow shovel on a roundabout, and the air is still evenly underground snow, then when the sled has been running along the runway, as long as the speed and snowfall rate is constant, the runway snow volume S is also constant, and the car after the minimum amount of snow, the largest amount of snow in front, as shown in Figure A. In this case, snow shovel for every lap of time, the amount of snow is X, the car shovel the amount of Y, then X, y to meet the s+x-y =s, that is × = Y, and because in the process of shoveling snow, the shovel off the original snow plus half of the snow, so y = S + x/2, so y = 2S, that is, shovel shovel off 2S Amount of snow. In the selection permutation, the size of the array is equal to S, the amount of snow shovel is equivalent to the output of the size of the string, that is, twice times the array large. This proves to be a bit abstract, but in practice as long as the numbers in the input file are randomly distributed, the resulting string size is probably twice times the size of the array used.

3. Loser Tree

In the process of multi-path merging, if there are K-string, every time there are K candidate values, to find out the minimum value, the common practice requires a K-1 comparison, and the use of the loser tree, then only the O (LOGK) comparison, its principle is like our usual group competition, a contestant after the group outlet, You only need to compete with other teams to qualify for the final title (maximum value), and not to compete with all other contestants.

The following figure is a 5-way merge process built in the loser tree, because to order from small to large, so in each comparison, the small for the win, the big for the defeat. Array B[0..4] Stores the number of reads in a string, L[0] stores the final winner (minimum) position, l[1..3] stores the loser's position in the middle of each race.

The current minimum value is 5 (b[4]), after the 5 output, if the newly read data is 11, first with the group before the loser B[3] comparison, and then compared with b[0], the result is defeated, then the Subscript 4 recorded in l[2], so that the winner B[0] to continue to compare with B[2], after winning will be subscript record to l[0 ], after 3 comparisons, a new minimum value of ten (B[0]) is obtained, as shown in the following figure

For the construction of the loser tree and the adjustment steps after each read into the new value, see the code below.

void Createlosertree (Run **runs, int n)
{for
    (int i = 0; i < n; i++)
    {
        ls[i] = 1;
    }
    for (int i = n-1; I >= 0; i--)
    {
        Adjust (runs, n, i);
    }
}
void Adjust (Run **runs, int n, int s)
{
    //first calculates according to s the corresponding LS in which subscript
    int t = (s + N)/2;    
    int tmp;

    while (t! = 0)
    {
        if (s = =-1) break
            ;
        if (ls[t] = = 1 | | runs[s]->buffer[runs[s]->idx] > 
                runs[ls[t]]->buffer[runs[ls[t]]->idx])
        {
            tmp = s;
            s = ls[t];
            LS[T] = tmp;
        }
        T/= 2;
    }
    Ls[0] = s;
}

http://shmilyaw-hotmail-com.iteye.com/blog/1776132

First, the winner Tree will be one of these forms:

A bit different from what we discussed earlier, where almost every leaf node in the heap corresponds to an input sequence. We let them form a completely binary tree. For example, after we make a selection of winners, the heap structure is as follows:

We can see that the final element at the top of the heap is the smallest, and there is a path from the leaf node to the root of the heap. What should we do if we finish processing the smallest element? The following figure illustrates this problem:

We found this problem by taking the following elements in the original sequence, then upward as the winner tree adjusts, the elements that meet the criteria are placed above, and then compared to the root. This will find the next smallest element. This keeps going through the loop. If a sequence is finished, we can use a value that adds an infinity at the end.

In general, this method is similar to the normal minimum heap adjustment, which means that the adjustment is not the same. We can also easily get the implementation of the object, where the detailed implementation code is no longer attached.