In reality, the recommended systems are generally based on the collaborative filtering algorithm, such algorithms usually need to calculate the user and user or project and project similarity, for data and data types of different data sources, need different similarity calculation method to improve the recommended performance, The mahout provides a large number of components for computing similarity, and these components implement different methods of similarity calculation respectively. The following figure is used to implement the relationship between components of the similarity calculation:
Figure 1, Project similarity computing component
Figure 2, User similarity computing component
The following is an introduction to several key similarity calculation methods:
Pearson correlation Degree
Class Name: Pearsoncorrelationsimilarity
Principle: A statistic used to reflect the linear correlation of two variables
Scope: [ -1,1], the greater the absolute value, the stronger the correlation, negative correlation for the recommendation of small significance.
Note: 1, do not consider the number of overlapping, 2, if there is only one overlap, can not calculate the similarity (the calculation process is divided into n-1) 3, if the overlapping values are equal, can not calculate the similarity (standard deviation is 0, do divisor).
This similarity is not the best choice, nor is it the worst choice, just because it is easy to understand and is often mentioned in early studies. The use of Pearson linear correlation coefficients must assume that the data is obtained from the normal distribution in pairs, and that the data must be equal spacing at least in the logical category. In Mahout, an extension is provided for Pearson correlation calculation, and the overlapping number is also the influence factor of calculating similarity by adding a parameter of an enumeration type (Weighting).
Euclidean distance similarity degree
Class Name: Euclideandistancesimilarity
Principle: Using Euclidean distance D to define similarity S,s=1/(1+D).
Scope: [0,1], the bigger the value, the smaller the D, the closer the distance, the greater the similarity.
Note: As with Pearson's similarity, the similarity does not take into account the effect of overlapping numbers on the result, and similarly, mahout by adding an enumeration type (Weighting) parameter to make the overlapping number also the influence factor of the computational similarity.
Cosine similarity degree
Class Name: Pearsoncorrelationsimilarity and Uncenteredcosinesimilarity
Principle: The cosine of the angle between two points of multidimensional space and the set point.
Range: [ -1,1] The larger the value, the greater the angle, the farther the two points are, the smaller the similarity.
Description: In mathematical expression, if the properties of two items are data-centric, the cosine similarity and Pearson similarity are the same, in Mahout, the data-centric process is realized, so Pearson similarity value is also the cosine similarity after data centrality. In addition, in the new version, Mahout provides the Uncenteredcosinesimilarity class as the cosine similarity for the calculation of the data that is not centralized.
Spearman rank correlation coefficient
Class Name: Spearmancorrelationsimilarity
Principle: The Spearman rank correlation coefficient is usually considered to be the Pearson linear correlation coefficient between the permutation variables.
Range: { -1.0,1.0}, 1.0 when consistent, and 1.0 when inconsistent.
Description: The calculation is very slow and has a large number of sorts. It is not suitable to use the Spearman rank correlation coefficient as the similarity measure for the data set in the recommended system.
Manhattan Distance
Class Name: Cityblocksimilarity
Principle: The realization of Manhattan distance, similar to the European distance, are used for multidimensional data space distance measurement
Scope: [0,1], the same as the European distance, the smaller the value, indicating the greater the distance value, the greater the similarity.
Description: Less computation than Euclidean distance, relatively high performance.
Tanimoto coefficient
Class Name: Tanimotocoefficientsimilarity
Principle: Also known as the generalized Jaccard coefficient, is the expansion of the Jaccard coefficient, the equation is
Scope: [0,1], when the total overlap is 1, when there is no overlap, 0, the closer 1 indicates the more similar.
Description: Handle the preference data without scoring.
Logarithmic likelihood similarity degree
Class Name: Loglikelihoodsimilarity
Principle: The number of overlapping, the number of not overlapping, there is no number of
Scope: Specific can go to Baidu Library to find the paper "accurate Methods for the Statistics of surprise and coincidence"
Note: Processing of the preference data without scoring is more intelligent than the calculation method of Tanimoto coefficients.