Several distance formulas for text similarity calculation (Euclidean distance, cosine similarity, jaccard distance, editing distance)

This paper mainly discusses some distance formulas of text similarity calculation, including: Euclidean distance, cosine similarity, jaccard distance, editing distance.

Distance calculations can be used in many scenarios, such as clustering, K-nearest neighbors, machine learning features, text similarity, and so on. Here's a look at the following:

Suppose two text x= (x1, x2, x3,... xn) and y= (Y1, y2, y3, ..., yn), whose vectors are represented by: VEC (X) = (v1, v2, v3, ... vp), VEC (Y) = (L1, L2, L3, ... LP)

1. European distance

European distance, also known as Euclidean distance, is the most common distance measure, which measures the absolute distance between two points in a multidimensional space.

The calculation formula is as follows:

2. Cosine similarity

Cosine similarity uses the cosine of the angle of the two vectors in the vector space to measure the similarity between the two texts, and the cosine similarity pays more attention to the difference in the direction of the two vectors than the distance measure, in general, after the vector representation of two text is obtained by embedding. You can use the cosine similarity to calculate the similarity between two texts.

The calculation formula is as follows:

3.Jaccard away

Jaccard distance is used to calculate the similarity between two individuals with symbolic or boolean measurements, since the characteristic attributes of an individual are measured by symbols or Boolean values, only the number of common features included can be counted.

The calculation formula is as follows:

4. Editing distance

The editing distance is primarily used to calculate the similarity of two strings, which is defined as follows:
With strings A and b,b as pattern strings, you are given the following actions: Remove a character from a string, insert a character from a string, and replace a character from a string. With the above three operations, the minimum number of operands required to edit string A to pattern string B is called the minimum editing distance of A and B, as an ED (b).

The algorithm for solving the shortest editing distance is described as follows:
A two-dimensional array ed[i][j] represents the minimum number of operands required for the first I character of String A to be edited into the first J characters of String B. The recursive formula for ED[I][J] is:
⑴ed[i][0]=i,ed[0][j]=j, of which 0≤i≤a.len,0≤j≤b.len;
⑵ if A[I]=B[J], then ed[i][j]=ed[i-1][j-1];
⑶ if A[I]≠B[J], then Ed[i][j]=min (Ed[i-1][j-1],ed[i][j-1],ed[i-1][j]) +1.

The smaller the editing distance, the more similar the two strings are. Conversely, the less similar.

The above is only the distance calculation formula common four kinds, there are many kinds of distance calculation formula, if used later will continue to add.