Mahout Series: Similarity degree

There are many similarity implementations in the Mahout recommendation system that compute the similarity between the user or item. For data sources with different data volumes and data types, different similarity calculation methods are needed to improve the recommended performance, and a large number of components for computing similarity are provided in mahout, and these components implement different computing parties of similar degree respectively.

User Similarity:

Item Similarity:

Pearson correlation Degree

Class Name: Pearsoncorrelationsimilarity

Principle: A statistic used to reflect the linear correlation of two variables

Scope: [ -1,1], the greater the absolute value, the stronger the correlation, negative correlation for the recommendation of small significance.

Note: 1, do not consider the number of overlapping, 2, if there is only one overlap, can not calculate the similarity (the calculation process is divided into n-1) 3, if the overlapping values are equal, can not calculate the similarity (standard deviation is 0, do divisor).

This similarity is not the best choice, nor is it the worst choice, just because it is easy to understand and is often mentioned in early studies. The use of Pearson linear correlation coefficients must assume that the data is obtained from the normal distribution in pairs, and that the data must be equal spacing at least in the logical category. In Mahout, an extension is provided for Pearson correlation calculation, and the overlapping number is also the influence factor of calculating similarity by adding a parameter of an enumeration type (Weighting).

Euclidean distance similarity degree

Class Name: Euclideandistancesimilarity

Principle: Using Euclidean distance D to define similarity S,s=1/(1+D).

Scope: [0,1], the bigger the value, the smaller the D, the closer the distance, the greater the similarity.

Note: As with Pearson's similarity, the similarity does not take into account the effect of overlapping numbers on the result, and similarly, mahout by adding an enumeration type (Weighting) parameter to make the overlapping number also the influence factor of the computational similarity.

Cosine similarity degree

Class Name: Pearsoncorrelationsimilarity and Uncenteredcosinesimilarity

Principle: The cosine of the angle between two points of multidimensional space and the set point.

Range: [ -1,1] The larger the value, the greater the angle, the farther the two points are, the smaller the similarity.

Description: In mathematical expression, if the properties of two items are data-centric , the cosine similarity and Pearson similarity are the same, in Mahout, the data-centric process is realized, so Pearson similarity value is also the cosine similarity after data centrality. In addition, in the new version, Mahout provides the Uncenteredcosinesimilarity class as the cosine similarity for the calculation of the data that is not centralized.