[Simhash] Find the percentage of similarity between, given data

Simhash algorithm, introduced by Charikar and was patented by Google.

Simhash 5 steps:tokenize, Hash, weigh Values, Merge, dimensionality Reduction

• Tokenize

  • Tokenize your data, assign weights to each token, weights and tokenize function is depend on your business

• Hash (MD5, SHA1)

  • Calculate token ' s hash value and convert it to binary (101011)

• Weigh values

  • For each hash value, does hash*w, in this: (101011), (W,-W,W,-W,W,W)

• Merge

  • Add up tokens ' values, to merge to 1 hashes, for example, merge (4-4-4 4-4 4) and (5-5 5-5 5 5), results to (4+5-4+-5 -4+5 4+-5-4+5 4+5), which is (9-9 1-1 1)

• dimensionality Reduction

  • Finally, signs of elements V of corresponds to the bits of the final fingerprint, for example (9-9 1-1 1), (1 0 1 0 1), we get 10101 as the fingerprint.

How do I use Simhash fingerprints?

Hamming distance can be used to find the similarity between both given data, calculate the Hamming distance between 2 Finge Rprints.

Based on my experience, for the bit simhash values, with elaborate weight values, distance of similar data

Often differ appreciably in magnitude from those unsimilar data.

How to Calculate:xor, only two bits does not have the same result is 1, otherwise 0, two binary value "XOR" after the number of 1 is the Hamming distance.

Simhash 0.1.0:python Package Index

[Simhash] Find the percentage of similarity between, given data