Massive recommendation system: mapreduce Method

1. Motivation

2. mapreduce

Mapreduce is a data-intensive parallel computing framework.

The data to be processed is stored in the HDFS in the Cluster machine File System in blocks and saved as key-value pairs.

When a task is started, the system assigns the computing task to the machine that stores the data.

Mapreduce computing tasks can be divided into two stages:

Mapreduce Data Flow Diagram

?

3. scalable similarity-based neighborhood

3.1 User-based recommendation

Note:

: User's score on the product

: Obtains the similarity between user U and other users.

Finally, for a commodity I, the user's prediction score is obtained based on the scores of other users and the user similarity weighting.

This is called the user-based method. The algorithm is based on the similarity between users.

3.2 item-based recommendation

Note:

Obtains the similarity Matrix of an item. Finally, the user U's score on item I is equal to the weighted sum of the user's score on the item and the item I similarity.

Therefore, the core of the Recommendation System is to calculate the similarity matrix.

3.3 scalable Calculation Method

Traditional similarity calculation is based on standard matrix multiplication.

Disadvantages:

1. In each map task, to initialize the scoring matrix A, map the input item and each column of A as a point multiplication. When matrix A is large, memory consumption is huge.

2. The complexity of the traditional calculation method is directly proportional to the square of the number of items. Furthermore, the sparse nature of user scores cannot be used.

Improvement Method

Column vector, which is the scoring vector of user U.

Specific Method:

First, the product of each user's scoring vector.

Then, the similarity matrix is obtained by adding these products. In this way, data can be divided by row vectors of.

Mapreduce computing framework

?

?

?

References:

[1]? Scalable similarity-based neighborhood methods with mapreduce?