(Data mining-Getting Started-2) how similarity is measured

Main content:

1. Distance measurement

2. Pearson Relationship coefficient

3. Similarity of cosine

4, the choice of methods

1. Distance measurement

Distance measurement is the simplest way to measure similarity, and the formula is as follows:

When R=1, for the Manhattan Distance (Manhattan distance);

When r=2, for Euclidean distance (Euclidean distance);

Advantages: Simple

Disadvantage: The measurement results are inaccurate when certain attributes or features are missing from the data

Code:

defMinskowski (rating1,rating2,r): Distance=0 commonratings=flase forKeyinchrating1: forKeyinchrating2:distance+=pow (ABS (rating1[key]-Rating2[key]), R) commonratings=Trueifcommonratings:returnPow (distance,1/R)Else:        return0#indicates no ratings in common

2. Pearson relationship coefficient (Pearson Correlation coefficient)

Sometimes, everyone's judgment is different, for example, when the item is scored (1-5), some people have a scoring range of 4-5, and some people are 1-5, and they actually evaluate the same result.

But if measured by the distance above, then the similarity of these two people will be very different, so a similar normalization method needs to be solved.

The formula is as follows:

R Range is -1~1

In the actual code implementation, can also be written in the following form, this kind of just need to traverse through the data.

Code:

defPearson (Rating1, rating2): Sum_xy=0 sum_x=0 sum_y=0 sum_x2=0 Sum_y2=0 N=0 forKeyinchrating1:ifKeyinchrating2:n+ = 1x=Rating1[key] y=Rating2[key] Sum_xy+ = x *y sum_x+=x sum_y+=y sum_x2+ = POW (x, 2) Sum_y2+ = Pow (y, 2)    #Now compute Denominatordenominator = sqrt (Sum_x2-pow (sum_x, 2)/N) * sqrt (Sum_y2-pow (sum_y, 2)/N)ifDenominator = =0:return0Else:        return(Sum_xy-(sum_x * sum_y)/N)/Denominator

3. Similarity of cosine

As mentioned in distance measurement, feature spaces are often sparse, and inaccurate results can occur if the similarity is measured by distance.

Therefore, the cosine similarity calculation solves this problem because it ignores the calculation of the value of 0, and the formula is as follows:

Code:

defcosine (rating1, rating2): Sum_xy=0 sum_x2=0; Sum_y2=0 forKeyinchrating1:sum_x+ = Rating1[key] *Rating1[key] forKeyinchrating2:sum_y+ = Rating2[key] *Rating2[key] forKeyinchrating1: forKeyinchRating2:sum_xy+ = Rating1[key] *Rating2[key] Denominator= sqrt (sum_x2) *sqrt (sum_y2)ifDenominator = =0:return0Else:        returnSum_xy/denominator

4, the choice of methods

If the data is dense dense, then the distance measurement can be used;

If the data is sparse sparse, then it can be measured with cosine;

If the data scale is inconsistent, then the Pearson metric can be used.

(Data mining-Getting Started-2) how similarity is measured