Mahout similarity Algorithm (ii) __ algorithm

In reality, the recommended systems are generally based on the collaborative filtering algorithm, such algorithms usually need to calculate the user and user or project and project similarity, for data and data types of different data sources, need different similarity calculation method to improve the recommended performance, The mahout provides a large number of components for computing similarity, and these components implement different methods of similarity calculation respectively. The following figure is used to implement the relationship between components of the similarity calculation:

Figure 1, Project similarity computing component

Figure 2, User similarity computing component

The following is an introduction to several key similarity calculation methods:

Pearson correlation Degree

Class Name: Pearsoncorrelationsimilarity

Principle: A statistic used to reflect the linear correlation of two variables

Scope: [ -1,1], the greater the absolute value, the stronger the correlation, negative correlation for the recommendation of small significance.

Note: 1, do not consider the number of overlapping, 2, if there is only one overlap, can not calculate the similarity (the calculation process is divided into n-1) 3, if the overlapping values are equal, can not calculate the similarity (standard deviation is 0, do divisor).

This similarity is not the best choice, nor is it the worst choice, just because it is easy to understand and is often mentioned in early studies. The use of Pearson linear correlation coefficients must assume that the data is obtained from the normal distribution in pairs, and that the data must be equal spacing at least in the logical category. In Mahout, an extension is provided for Pearson correlation calculation, and the overlapping number is also the influence factor of calculating similarity by adding a parameter of an enumeration type (Weighting).

Euclidean distance similarity degree

Class Name: Euclideandistancesimilarity

Principle: Using Euclidean distance D to define similarity S,s=1/(1+D).

Scope: [0,1], the bigger the value, the smaller the D, the closer the distance, the greater the similarity.

Note: As with Pearson's similarity, the similarity does not take into account the effect of overlapping numbers on the result, and similarly, mahout by adding an enumeration type (Weighting) parameter to make the overlapping number also the influence factor of the computational similarity.

Cosine similarity degree

Class Name: Pearsoncorrelationsimilarity and Uncenteredcosinesimilarity

Principle: The cosine of the angle between two points of multidimensional space and the set point.

Range: [ -1,1] The larger the value, the greater the angle, the farther the two points are, the smaller the similarity.

Description: In mathematical expression, if the properties of two items are data-centric, the cosine similarity and Pearson similarity are the same, in Mahout, the data-centric process is realized, so Pearson similarity value is also the cosine similarity after data centrality. In addition, in the new version, Mahout provides the Uncenteredcosinesimilarity class as the cosine similarity for the calculation of the data that is not centralized.

Spearman rank correlation coefficient

Class Name: Spearmancorrelationsimilarity

Principle: The Spearman rank correlation coefficient is usually considered to be the Pearson linear correlation coefficient between the permutation variables.

Range: { -1.0,1.0}, 1.0 when consistent, and 1.0 when inconsistent.

Description: The calculation is very slow and has a large number of sorts. It is not suitable to use the Spearman rank correlation coefficient as the similarity measure for the data set in the recommended system.

Manhattan Distance

Class Name: Cityblocksimilarity

Principle: The realization of Manhattan distance, similar to the European distance, are used for multidimensional data space distance measurement

Scope: [0,1], the same as the European distance, the smaller the value, indicating the greater the distance value, the greater the similarity.

Description: Less computation than Euclidean distance, relatively high performance.

Tanimoto coefficient

Class Name: Tanimotocoefficientsimilarity

Principle: Also known as the generalized Jaccard coefficient, is the expansion of the Jaccard coefficient, the equation is

Scope: [0,1], when the total overlap is 1, when there is no overlap, 0, the closer 1 indicates the more similar.

Description: Handle the preference data without scoring.

Logarithmic likelihood similarity degree

Class Name: Loglikelihoodsimilarity

Principle: The number of overlapping, the number of not overlapping, there is no number of

Scope: Specific can go to Baidu Library to find the paper "accurate Methods for the Statistics of surprise and coincidence"

Note: Processing of the preference data without scoring is more intelligent than the calculation method of Tanimoto coefficients.