[Recommendation System] collaborative filtering-data cleaning under highly sparse data (Pearson correlation coefficient)

Similarity between vectors

There are many ways to measure the similarity between vectors. You can use the reciprocal of distance (various distances), vector angle, Pearson correlation coefficient, and so on.

Pearson correlation coefficient calculation formula is as follows:

The numerator is the covariance, And the numerator is the product of the standard deviation of two variables. Obviously, the standard deviation of X and Y cannot be 0.

Because, the Pearson correlation coefficient calculation formula can also be written:

When the linear relationship between two variables is enhanced, the correlation coefficient tends to 1 or-1.

User rating Prediction

The basic principles of user scoring prediction are:

Step 1. If user I does not comment too much on Project J, find the K neighbors most similar to user I (using vector similarity measurement)

Step2. then, the weighted average of the scores of the K neighbors on the Project J is used to predict the user I's score on the Project J.

	Iterm1	............	Itemn
User1	R11		R1n
......		Rij
Userm	Rm1		Rmn

User rating data matrix

Step1.

The user rating matrix is a highly sparse matrix, that is, the user does not score many projects. The traditional vector similarity measurement method is used to measure the similarity between two users in a highly sparse manner. A simple solution is to make all the unrated items equal to the average score of the user.

A better way to measure the similarity between user I and user J is:

1. Set User I to participate in the scoring project set to II, Set User J to participate in the scoring project set to IJ, and find their Union set.

2. items in which user I is not rated are re-evaluated for each item in user I.

. Retrieve the two columns of the scoring matrix and calculate the similarity between the two columns .. Find the most similar set of V projects.

2.2.

3. In this way, both user I and j have a non-zero score. In this case, their similarity is calculated.

Step2.

Indicates the average rating of user U for all the items that have been scored.

Reference:

Http://zh.wikipedia.org/wiki/%E7%9A% AE %E5%B0%94%E9%80%8A%E7%A7%AF%E7%9F%A9%E7%9B%B8%E5%85%B3%E7%B3%BB%E6%95%B0

Http://www.cnblogs.com/zhangchaoyang/articles/2664366.html