CRC32 algorithm conflict probability test and analysis

Recently, because a business needs to use the CRC32 algorithm, but the business can not tolerate duplicate values appear, so naturally want to understand the CRC32 algorithm collision probability (or called collision probability).

I think this problem should be analyzed by many people, the results of finding to find only a large number of mathematical formulas, I this kind of mathematics blind completely do not understand.

Finally find a picture, but see Foggy (original link: http://preshing.com/20110504/hash-collision-probabilities/):

Since the internet is not reliable, then to verify it yourself, write a PHP script is very simple, my 1th validation model is this:

Take 1 integer values as initial values, then increment 1000W times, calculate the value of CRC32 each time, output to a file, and then use sort Crc32.result | Uniq-c-d > crc32-collision.txt to output conflicting results

As a result, I was shocked:1000W No 1 conflicts ! Greatly unexpected, so try 2000W, or no conflict !! So try 100 million, this time there is a conflict, the number of collisions is about 240W!!

Although I did not understand the principle of the CRC32 algorithm, but vaguely feel that this conflict rate is not in line with the actual, so continue to find, finally Kung Fu, to find a detailed and complete test report (http://www.backplane.com/matt/crc64.html):

CRC16-CRC64 test results on 18.2M dataset

This test report is very detailed, basically solved our question, from this report can see,1820W data, the number of collisions is 38,638 , this is more in line with my understanding and expectations.

But the problem is still unresolved: Why is my test result so good ?

Because the CRC32 algorithm is universal, so there is no different language implementation mechanism of different problems, so I turned to the test model, the problem is here.

My test model: CRC32 (i++), the original value entered by the calculation model is just a continuous data in a range, not completely random .

So I modified it a little bit: CRC32 (MD5 (i++)), so that the original value of the input is completely random .

Re-test, the results come out, and the results of the complete test report above exactly the same!!

Summarize:

1) CRC32 in a completely random input situation, the conflict probability is still relatively high , especially to 100 million of the data volume, the conflict probability will be higher

2) CRC32 in the case of entering data for a contiguous segment, the conflict probability is very low , because the original value of two conflicts should theoretically be very far apart, in the case of entering only one piece of data, the original value of the conflict in this paragraph will be very small

CRC32 algorithm conflict probability test and analysis