Master-level articles can always bring you closer to the nature of things. Recently saw the Pongba of mathematics of the United States: Fast row why so fast. An article in Mackay (translated here) mentions the essential reason for using information theory to explain the speed difference between a row and a heap. Look, there is a kind of inexplicable excitement inside. Although do not understand information theory, but Mackay (master after all is a master), the simplest explanation, but direct contact with the essence. Interested to see this article Mackay. Here, just a little sentiment of my own.
What is information entropy.
An event, its amount of information is directly related to its uncertainty. For example, we need to know a lot of information about a very, very uncertain thing, or something that we don't have a clue about. On the contrary, if we have a better understanding of something, we don't need too much information to figure it out. So, from this point of view, we can think that the measurement of the amount of information equals the amount of uncertainty. The measurement of this information is called entropy, and the greater the entropy, the more unpredictable the outcome of the event, and the more information the event will bring to us.
Everyone has tossed a coin. We think of the entropy of this event as 1bit, why we think of the result of a comparison as 1bit information.
Can explain this: a flip coin positive and negative each 1/2 probability. If the signal source is this, the amount of information contained is-1/2 * log (1/2) + -1/2 * LOG (1/2) = 1 bit.
Therefore, a 1/2, 1/2 probability (equal probability) distribution of the signal contains the amount of information is 1bit. Comparisons are such operations as CMP (A, b) as a flip coin. Returns 1 bit at a time. One would argue that, because sometimes the nature of the order, such as a>b, B>c a>c cause a third time to ask CMP (A, c), it is not possible to generate a random signal with a probability of 1/2 1/2. In fact, a good sorting algorithm is to avoid the use of CMP (A, B) CMP (b, c) to get A>b b>c after the comparison of a, C. The best sorting algorithm is to get 1bit of information from CMP each time.
However, if the coin has been tampered with, so that the positive probability is greater than the opposite probability (or reverse), then the entropy of the event will be <1bit. Because we already know something about the outcome, and obviously we have less information. More extreme, if the coin is two positives (or vice versa), then the entropy of this event will be 0 bit, because we know exactly what the result is. 1, ordering and information entropy
Why the information theory can be extended to (based on comparison) sort.
Entropy is an important concept in information science, which is used to express the uncertainty of things. The order is to make the sequence from disorderly to orderly process, disorder is uncertainty, so we can use the concept of information entropy to describe the disorder degree of sequence, and then study the efficiency of the ranking algorithm based on comparison.
So, you want to make the sort based on comparison faster, equivalent to the maximum information entropy for each comparison. So, let each comparison of the result probability equal (make each comparison of information entropy close to 1bit), this is the core idea of algorithm improvement.
Give a title: There are 5 numbers, and at least several comparisons are required to achieve the sorting of the data.
Set the sequence s length to n, then the series has a different arrangement of x = n!. If the sequence is regarded as the output of the source, and the probability of each permutation is the same, then according to the definition of information entropy of the discrete random variable, the average information of the sequence s is defined as: Hn = L (x) = log2 (x) =l og2n! (bit)
5 integers, resulting in a full array of 5!=120 species, from the information theory point of view, the amount of information is log2{120} = 6.90689059560852 bit, and two of the number of times, can produce a maximum of 1bit of information, so 7 comparisons is almost the limit. Of course this is not to say that you can get the sort results 7 times, but when you have tried countless times, you find that 7 times is the limit of its number. Below, we take a look at the use of each comparison to get the information entropy close to the idea of 1bit, to specific implementation, the appeal of the sequencing process:
set the original sequence consists of 5 elements (A, B, C, D, E), first sorting (A, B) and (C, D) respectively, and setting the result of the order to [a b] and [C D] respectively. ([x y] means sorted) (For other sorting results, the following analysis process is the same), obviously this step requires 2 comparisons. then compare a with C, if a < C, the sort result is [A C d], and the binary insertion method inserts e into [AC d] after 2 comparisons, finally inserts B into the sorted sequence, Since B > A is known, inserting B with the binary insertion method is a worst-case scenario that requires only 2 comparisons. If a > C, the result of the order is [c a b], the same with the binary insertion method through 2 comparisons to insert E, since C has been compared with D, so finally inserting D into the sorted sequence requires a maximum of 2 comparisons. comprehensive, the entire sorting process needs to be compared at most 7 times. The first step between A and B and C and D is obviously reduced by 1 bit information entropy, at this time there are 30 different permutations of the sequence, the remaining information entropy is 4. 9069 b it. A compared with C, the sequence has 15 kinds of permutations, the remaining information entropy is 3. 9069 b It, so this time the comparison also reduces the information entropy 1 bit. Insert E, if E < A, the sequence may appear 3 kinds of permutations, if E > A, there are 16 possible permutations, so the average information entropy is: So 2 times compared to reduce information entropy 1.9724 bit, the last 2 times compared to reduce 1.9345 bit. It can be seen that each comparison reduces information entropy close to 1 bit, so the ranking efficiency is highest. 2, bubbling, selecting, inserting, why so slow. First we look at three classical square complexity algorithms. Their efficiency is not high, because the algorithm will appear more and more probability of serious uneven comparison. 1, bubbling with the bubble sort, the whole sequence will become more and more orderly, there will be fewer bubbles in the reverse position. 2, select the sorting selection of the sorting choice, you will continue to get more and more large numbers, while in the future comparison to find a larger number of probability is also getting lower; 3, insert sort in the Insert sort, You always compare the new number with the number you've already lined up in order from big to small, and you can think of a new number that starts with all the previousThe largest of the number of the large probability is quite small. Inspired by this, we can naturally think of an improvement in the insertion sort: How to compare the number of digits in the previous processed numbers when dealing with a new number. The entropy of this comparison is obviously larger, the amount of information available is much larger and obviously more valuable. This is the binary lookup improvement of the insertion sort.
3, Fast platoon why so fast. In the fast sorting algorithm, the information entropy of the comparison is not gradually reduced because of the sorting algorithm, which is the root cause of the better sorting algorithm. With a careful review of the process of fast sorting algorithms, we can see that the probability of the two results of each comparison is entirely determined by the benchmark key chosen by the partitioning process: If the selected Datum key is just the median of the number set currently being processed, The uncertainty of the comparison results is maximum (the information entropy of the comparison is up to 1bit). This time, the speed of the comparison of the number of times, is equal to the need for sorting the total information entropy: log2n! (bit) This is precisely the time complexity of the fast row. 4, the fast row is not so fast. If the selected benchmark keyword is too large or too small, there will be comparisons resulting in uneven results, resulting in a significant reduction in the amount of information each time the comparison is averaged. For example: The selected main element is the minimum value, then the comparison with this main element gets the entropy of the information is 0bit. Because, every time the result is affirmative. Look at the case of the main element at random:
When the main element is randomly selected: we might as well make the axis element is pivot, the first comparison result is A1<pivot, then can prove that the second comparison A2 is also less than pivot probability is 2/3. This is easy to prove: if A2>pivot, then a1,a2,pivot the relationship between these three elements completely determine the--A1<PIVOT<A2, the remaining elements of the arrangement of the possibility we may wish to remember as P (do not need to be specifically calculated). And what if A2<pivot. Then the relationship between A1 and A2 is still uncertain, that is to say, there are two kinds of situations in this branch: A1<a2<pivot, and A2<a1<pivot. For either case, the remaining elements are likely to be arranged in P, so the remaining arrangement in this branch is 2P. So when A2<pivot, there are still 2/3 of possibilities to be checked.
Further, if the second step is really found a2<pivot words, the third step is more bad, imitate the above reasoning, a3<pivot probability will be 3/4.
This is the reason for the fast row is not so fast, it does not guarantee the probability of each comparison result is the same (1/2:1/2), therefore, each comparison of information entropy is not =1bit.
If you want to improve the fast platoon, try to focus on improving the information entropy of each comparison.
from:http://blog.csdn.net/cyh_24/article/details/8120045
