How to use Python to detect duplicate images through a hash algorithm

This article mainly introduces how to use the hash algorithm to detect duplicate images in Python. this method is used by Iconfinder as anti-piracy technology. if you need it, refer to Iconfinder as an icon search engine, it provides exquisite icons for designers, developers and other creative workers. it hosts more than 0.34 million icons and is the largest paid Icon Library in the world. You can also upload and sell original works in the Iconfinder transaction section. Every month, thousands of icons are uploaded to Iconfinder, along with a large number of pirated images. In this article, Iconfinder engineer Silviu Tantos proposes a novel and clever image re-checking technology to prevent piracy.

We will launch a function to check whether the upload icon is repeated in the next few weeks. For example, if a user downloads an icon and tries to make a profit by uploading it (a similar case has occurred), we can use our method to check whether the icon already exists, and mark this account as fraudulent. A common method to check whether a file exists in a large number of files is to compute the hash value of each file in the dataset and store the hash value in the array library. When you want to find a specific file, first calculate the hash value of the file, and then find the hash value in the database.
Select a hash algorithm

The encryption hashing algorithm is a common hash algorithm. Standard libraries such as MD5, SHA1, and SHA256 can be found in any language. they are very effective for simple use cases.

For example, in Python, first import the hashlib module, and then call a function to generate the hash value of a string or file.

>>> import hashlib # Calculating the hash value of a string.>>> hashlib.md5('The quick brown fox jumps over the lazy dog').hexdigest()'9e107d9d372bb6826bd81d3542a419d6' # Loading an image file into memory and calculating it's hash value.>>> image_file = open('data/cat_grumpy_orig.png').read()>>> hashlib.md5(image_file).hexdigest()'3e1f6e9f2689d59b9ed28bcdab73455f'

This algorithm is very effective for uploaded files that have not been tampered with. if the input data changes slightly, the encryption hashing algorithm will lead to an avalanche effect, as a result, the hash value of the new file is completely different from that of the original file.

For example, in the following example, it adds an end to the end of a sentence.

# Original text.>>> hashlib.md5('The quick brown fox jumps over the lazy dog').hexdigest()'9e107d9d372bb6826bd81d3542a419d6' # Slight modification of the text.>>> hashlib.md5('The quick brown fox jumps over the lazy dog.').hexdigest()'e4d909c290d0fb1ca068ffaddf22cbd0'

If the image background color is changed, the image is cropped, rotated, or a certain pixel is modified, the image cannot be matched in the image hash library. It can be seen that the traditional hash algorithm is not practical. As you can see in the above example, the hash value 9 e0000d9d372bb6826bd81d3542a419d6 and e4d909c290d0fb1ca068ffaddf22cbd0 are almost different (except for several characters ).

For example, after you modify the color of the cat's nose in the image, the hash value of the image changes.

# Load the original image into memory and calculate it's hash value.>>> image_file = open('data/cat_grumpy_orig.png').read()>>> hashlib.md5(image_file).hexdigest()'3e1f6e9f2689d59b9ed28bcdab73455f' # Load the modified image into memory and calculate it's hash value.>>> image_file_modified = open('data/cat_grumpy_modif.png').read()>>> hashlib.md5(image_file_modified).hexdigest()'12d1b9409c3e8e0361c24beaee9c0ab1'

There are already many perceptual hashing algorithms. this paper will propose a new dhash (differential hashing) algorithm, which calculates the brightness difference between adjacent pixels and determines the relative gradient. For the preceding use cases, it is very effective to perceive the hash algorithm. The sensor hashing algorithm obtains a multimedia file fingerprint that can flexibly distinguish tiny differences between different files from various features of the file content.

DHash

Before going deep into the dHash algorithm, I would like to introduce some basic knowledge. A color image is composed of RGB primary colors and can be regarded as a color set of red, green, and blue primary colors. For example, you can use the Python Image Library (PIL) to load an image and print the pixel value.

Test image

>>> from PIL import Image>>> test_image = Image.open('data/test_image.jpg') # The image is an RGB image with a size of 8x8 pixels.>>> print 'Image Mode: %s' % test_image.modeImage Mode: RGB>>> print 'Width: %s px, Height: %s px' % (test_image.size[0], test_image.size[1])Width: 4 px, Height: 4 px # Get the pixel values from the image and print them into rows based on# the image's width.>>> width, height = test_image.size>>> pixels = list(test_image.getdata())>>> for col in xrange(width):...  print pixels[col:col+width]...[(255, 0, 0), (0, 255, 0), (0, 0, 255), (255, 255, 255)][(0, 0, 0), (212, 45, 45), (51, 92, 154), (130, 183, 47)][(206, 210, 198), (131, 78, 8), (131, 156, 180), (117, 155, 201)][(104, 133, 170), (215, 130, 20), (153, 155, 155), (104, 142, 191)]

Now let's go back to the dHash algorithm, which has four steps. This article describes each step in detail and verifies its effect on the original image and the modified image. The intensity values of the first three pixels are 255, the other two are 0, the three primary colors of the pure black pixels are 0, and the three primary colors of the pure white pixels are 255. Other color pixels are composed of three primary color values of different intensity.

1. grayscale images

Reduce the pixel value to a luminous intensity value through a grayscale image. For example, a white pixel (255, 255, 255) is 255, and a black pixel (, 0) is 0.

2. scale down an image to a common size

Scale down an image to a common base size, for example, the size of 9*8 pixels in a pixel value in width or height (you can see the reason for this size in step 3 ). This method removes the high frequency and detail parts of the image to obtain a sample with 72 intensity values. Since adjusting or stretching an image does not change its hash value, all images are normalized to this size.

3. compare neighboring pixels

After the previous two steps are completed, a list of intensity values is obtained to compare the adjacent pixels of each row of the binary value array.

>>> from PIL import Image>>> img = Image.open('data/cat_grumpy_orig_after_step_2.png')>>> width, height = img.size>>> pixels = list(img.getdata())>>> for col in xrange(width):...  print pixels[col:col+width]...[254, 254, 255, 253, 248, 254, 255, 254, 255][254, 255, 253, 248, 254, 255, 254, 255, 255][253, 248, 254, 255, 254, 255, 255, 255, 222][248, 254, 255, 254, 255, 255, 255, 222, 184][254, 255, 254, 255, 255, 255, 222, 184, 177][255, 254, 255, 255, 255, 222, 184, 177, 184][254, 255, 255, 255, 222, 184, 177, 184, 225][255, 255, 255, 222, 184, 177, 184, 225, 255]

Compare the first value 254 with the second value 254, compare the second value to the third value, and so on, so that each row gets eight Boolean values.

>>> difference = []>>> for row in xrange(height):...  for col in xrange(width):...   if col != width:...    difference.append(pixels[col+row] > pixels[(col+row)+1])...>>> for col in xrange(width-1):...  print difference[col:col+(width-1)]...[False, False, True, True, False, False, True, False][False, True, True, False, False, True, False, False][True, True, False, False, True, False, False, False][True, False, False, True, False, False, False, True][False, False, True, False, False, False, True, True][False, True, False, False, False, True, True, False][True, False, False, False, True, True, False, False][False, False, False, True, True, False, False, True]

4. convert to binary

To facilitate the storage and use of hash values, convert eight Boolean values into hexadecimal strings. True to 1, and False to 0.
Python implementation

The complete Python implementation algorithm is as follows:

def dhash(image, hash_size = 8):  # Grayscale and shrink the image in one step.  image = image.convert('L').resize(    (hash_size + 1, hash_size),    Image.ANTIALIAS,  )   pixels = list(image.getdata())   # Compare adjacent pixels.  difference = []  for row in xrange(hash_size):    for col in xrange(hash_size):      pixel_left = image.getpixel((col, row))      pixel_right = image.getpixel((col + 1, row))      difference.append(pixel_left > pixel_right)   # Convert the binary array to a hexadecimal string.  decimal_value = 0  hex_string = []  for index, value in enumerate(difference):    if value:      decimal_value += 2**(index % 8)    if (index % 8) == 7:      hex_string.append(hex(decimal_value)[2:].rjust(2, '0'))      decimal_value = 0   return ''.join(hex_string)

In the most common case, the image is slightly different, and the hash value may be the same, so we can directly compare it.

>>> From PIL import Image >>> from utility import dhash, hamming_distance >>> orig = Image. open ('data/cat_grumpy_orig.png ')> modif = Image. open ('data/cat_grumpy_modif.png ')> dhash (orig)' 4c8e3366c275650f '> dhash (modif)' 4c8e3366c275650f '> dhash (orig) = dhash (modif) True if one

The SQL database that stores the hash value. you can easily determine whether the hash value "4 c8e3366c275650f" exists:

SELECT pk, hash, file_path FROM image_hashes  WHERE hash = '4c8e3366c275650f';

Now, for some images with large differences, their hash values may be different. Therefore, you need to calculate the minimum number of characters to replace from one string to another, that is, the Hamming distance.
Wikipedia has some Python Sample code that calculates the Hamming distance between two strings. However, it can also be implemented directly based on the computation and query on the MySQL database.

SELECT pk, hash, BIT_COUNT(  CONV(hash, 16, 10) ^ CONV('4c8e3366c275650f', 16, 10)) as hamming_distance  FROM image_hashes  HAVING hamming_distance < 4  ORDER BY hamming_distance ASC;

Compare the queried value with the hash value in the database, and count different digits. BIT_COUNT can only operate on integers. Therefore, convert all hexadecimal hash values to decimal.

Conclusion

This article uses Python to implement the introduced algorithm. of course, you can use any programming language to implement the algorithm.

As mentioned in the introduction, the algorithm in this article will be applied to Iconfinder to prevent repeated submission of icons. as you can expect, there are more practical applications to perceive the hash algorithm. Because the hash values of images with similar features are similar, it can help the image recommendation system look for similar images.