Personalized recommended technology what are the challenges?

Personalized recommendations are often misunderstood as the two concepts of market segmentation and precision marketing. Although there is some connection between them, they are far different in essence. This article not only clearly describes the personalized recommendation technology, but also lists the top ten challenges it faces.

Many people know personal recommendation, there are many misunderstandings on the understanding. Some people think that personalized recommendation is the market segments and precision marketing, but in fact the market segments and precision marketing tend to be divided into potential users of many groups, which is compared with the statistics based on the overall compared to have made great strides, but From the "tailored information service for every user" is still a big gap, only personalization can achieve the dream of Amazon CEO Jeff Bezos "If I have one million users, I will do for them one million Amazon website".

Therefore, the market segmentation is only the initial stage, and personalized recommendation is the ultimate breakdown of the market. Others think that personalized recommendation is equivalent to collaborative filtering, which may be because collaborative filtering appears earlier and is easier for the general public to understand, but in fact collaborative filtering is only an early member of many personalized recommendation technologies, far from being able to represent Personalized recommendation technology.

As personalized business applications extend to all aspects of user life information flow, personalized recommendation technology is also evolving with each passing day. Similar to the earlier technology such as collaborative filtering has been unable to meet the requirements of the new environment, can not solve emerging problems. What's exciting is that over the past decade, we have witnessed numerous top-level experts and scholars engaged in the innovation of recommended methods and technologies. Today, some people think that the study of personalized recommendation technology has entered a very mature stage, there is no particularly exciting issues and achievements. In fact, on the contrary, personalized recommendation technology now faces great challenges, and we have only seen the tip of the iceberg where personalized recommendation technical problems are exposed.

This article will list the top ten challenges (personal views only) of personalized recommendation techniques, some of which are long-standing issues that have been recognized many years ago but have not been resolved. In fact, some of the challenges can not be completely solved, only improvement programs can be proposed, and some are the focus issues raised in recent studies.

Data sparsity

Now the system is recommended to increase the scale of users and products (including music, web pages, documents and other items) the number of millions, and there is very little overlap between the choice of users. If we measure the sparsity of the system in terms of the proportion of the existing choices between users and commodities in all possible selection relations, the sparsity of our most frequently studied MovieLens dataset is 4.5%, Netflix is ​​1.2% and Bibsonomy Is 0.35%, Delicious is 0.046%, these are actually very close data.

Think about 800 million Taobao goods, on average, a user can browse 800? Estimates can not, so the sparseness should be on the order of one millionth or less. The data is very sparse, making the vast majority of algorithms based on association analysis (such as collaborative filtering) ineffective. In essence, this problem can not be completely overcome. There are many ways to solve this problem, for example, diffusion algorithm can be from the original first-order association (two users have similar scores or co-purchase of goods) to second-order or even higher-order association (assuming relevance or similarity Sex itself can be spread), you can also add some default scoring, thereby increasing the similarity of the resolution. In general, the larger the data, the more sparse it is. Now consider the algorithm capable of handling sparse data (such as diffusion, iterative optimization, transfer similarity, etc.) more valuable.

Cold start problem

Because new users rarely have access to behavioral information, it is difficult to give precise recommendations. In turn, because of the small number of new products being selected, it is difficult to find a suitable way to recommend to users. One approach is to use text messages to assist in recommendation, or to register and ask about some of the user's attribute information, such as age, city of residence, education, gender and occupation.

Recently, a widely used labeling system offers a possible solution to the problem of cold start, because the label can be regarded as the extraction of the product content, but also reflects the user's personal preferences. Take the movie "Peach-sister" as an example. Some people label "ethics" while others tagged "Andy Lau". The two people watch the movie but the points of interest may not be the same. Of course, the use of labels can only improve the accuracy of the recommendation of users with a small number of acts, for pure cold-start users, there is no help, because these people have not played any label.

Interestingly, recent research shows that new users are more likely to choose particularly popular items. In any case, this is good news, indicating that the use of hot list can get good results. Cold-start issues can also be overcome by cross-recommendation of multidimensional data, which is far more accurate and more versatile than the hot list, as we will see later.

Big data processing and incremental computing issues

Because data is sparse and most data has millions of users and products, how to deal with the data quickly and efficiently becomes an immediate issue. The complexity of the algorithm in time and space, especially the former, has received unprecedented attention. An efficient algorithm, either with low complexity, or well parallelized, or both. The local diffusion algorithm has obvious advantages in both aspects.

Another solution is to design incremental algorithms. In other words, when new users, new products and new connections are generated, the result of the algorithm does not need to be recalculated over the entire data set. Instead, it only needs to consider the added information of the nodes and the connected edges, The results of the perturbation, quickly get new results. In general, with the increase of the amount of information added, the error of this algorithm will accumulate and become larger, and eventually need to be recalculated with global data every time.

A particularly difficult challenge is how to devise an algorithm that will ensure that errors do not accumulate, ie the difference between the result and the recalculation using the entire data will not increase monotonically. We call this algorithm adaptive algorithm, which is an enhanced version of the incremental algorithm, its design requirements and more difficult.

The industry is already applying incremental algorithms. For example, some algorithms in the percentage point technology recommendation engine employ incremental techniques that allow users to update their recommendation list every time they browse a collection or purchase a product. Of course, just part of the engine algorithm to achieve incremental technology, did not meet all the algorithms are able to adaptive learning degree, there is still a long way to go.

Diversity and accuracy of the dilemma

If you want to recommend your favorite products to users, the most "safe" way is to give him a particularly popular or particularly high-scoring products, because these products are more likely to be like (at least Bezos will think so), to the bad that Difficult to particularly hate. However, this recommendation is not necessarily a good user experience, as users are likely to already know about these hot or popular products, so the amount of information they get is small and users will not consider it a "personal" recommendation.

In fact, Mcnee et al. Have warned that blindly adhering to the accuracy metrics may hurt the recommendation system, as this may result in users getting "precise referrals" with a volume of 0 and narrowing the field of view. Narrowing the user's perspective is a major drawback of the collaborative filtering algorithm. At the same time, businesses that apply personalized recommendation technology also want more categories to appear in the recommendations, inspiring new shopping needs.

Unfortunately, there is a conflict between recommending a wide range of products and novelty products, and the accuracy of recommendations, as the former is at risk - recommending something that no one has seen or scored lower, is likely to be hated by the user, and more effective difference. In many cases, this is a dilemma that can only be improved by sacrificing diversity, or sacrificing accuracy to increase diversity. One way to do this is to work on the recommendation list directly to increase its diversity. Although this method is effective in application, but without any theoretical basis and graceful at all, can only be regarded as a practical tricks.

We found that the two algorithms, which combine sophistication with high precision and diversity, can improve the diversity and accuracy of the algorithm without sacrificing either. Unfortunately, we have not been able to provide a clear explanation and insight on this result. The intricacies of diversity and precision and the ensuing competition behind it have so far been a daunting challenge.

Recommended system vulnerabilities

Recommender systems can drive significant economic benefits in the field of e-commerce, leading to misconduct users who provide false or malicious behavior to intentionally increase or suppress the likelihood that certain products will be recommended. Therefore, whether or not an algorithm can maintain the robustness against malicious attacks to a certain degree becomes a feature that needs to be seriously considered. Taking the simplest association rule mining algorithm as an example, Apriori algorithm is much more robust than k-nearest neighbor algorithm.

There are some techniques specifically designed to improve the robustness of recommender systems in the face of malicious attacks. For example, by analyzing the difference between the scoring behavior patterns of real users and suspected malicious users, malicious behavior can be judged in advance to prevent it from entering the system or minimize the influence of malicious users.

Generally speaking, there are relatively few studies in this area and systematic analysis is still lacking. Instead, it is an endless stream of attack tactics with the feeling of being "one foot tall and one magical high". Burke et al.'s 2011 study "Robust Collaborative Recommendation" analyzed four broad categories and eight different attack strategies.

Mining and Utilization of User Behavior Patterns

Digging deeper into the user's behavior patterns is expected to improve recommendations or make recommendations in more complex scenarios. For example, new users and old users have very different selection modes: in general, new users tend to choose popular products, while old users pay more attention to niche products; new users choose products with higher similarity, The selection of goods is more diverse.

Some hybrid algorithms can adjust the diversity and popularity of recommended results with one parameter. In this case, different parameters can be considered for different users (from individualizing algorithm results to personalization of the algorithm itself) or even allowing the user to move a slider to adjust this parameter - when the user wants to see a hot product, Algorithms provide popular recommendations; algorithms can also provide popular recommendations when users want to find cool products. Spatiotemporal statistics of user behavior can also be used to enhance recommendations or to design application-specific scenarios.

The user's choice may have both long-term interests and short-term interests. By separating these two effects, the accuracy of the recommendation can be significantly improved. In fact, given the user interest decreasing exponentially over time, improved recommendations can also be obtained.

Now that the mobile Internet has become more and more popular. At the same time, more and more mobile phones are embedded in the GPS. Therefore, the location-based service has become a topic of widespread concern in academia and industry. The recommendation based on location information may become a research hotspot and an important application scenario for personalized recommendation. To solve this problem, it is necessary to have a deep understanding of the user's mobile mode (including predicting the user's mobile locus and determining whether the user has the current location May conduct food shopping activities, etc.), but also have a quantitative approach to define the similarity between users and places. In addition, different users scoring mode is very different, the user behavior patterns for different products is not the same (imagine a download of a song online and buy a house difference), these can be used to improve the recommended effect.

Recommended system performance evaluation

The concept of a recommendation system has been proposed for decades, but how to evaluate a recommendation system remains a big issue. Common assessment indicators can be divided into four categories, namely, accuracy, diversity, novelty and coverage. There are different indicators for each category. For example, the accuracy index can be divided into four categories, namely, the accuracy of prediction score, the correlation of prediction score, the classification accuracy and the sorting accuracy. Take classification accuracy as an example, it also includes accuracy rate, recall rate, rate of improvement of accuracy rate, rate of increase of recall rate, F1 index and AUC value.

Zhu Yu Xiao and Lv Linyuan authored a "review of recommended systematic reviews," a paper that almost summarizes all of the recommended system indicators that have appeared in the literature. These indicators are based on the data itself and can be considered as the first level. In fact, in the real application, what is more important is the other two levels of evaluation. The second level is a key performance indicator for business applications such as conversion rate, purchase rate, customer price, number of items purchased, etc. that are affected by the recommendation. The third level is the user's real experience.

The vast majority of studies focus only on the first level of evaluation and the industry is really interested in the second level of evaluation (for example, which one or which combination of indicators results in higher customer prices) The third level is the most difficult, no one can know, only through the second level to estimate. Therefore, how to establish the relationship between the first level and the second level of indicators has become the key. This step opened up, the barrier between theory and application pass more than half.

User interface and user experience

So far, this is an academic issue, rather than real application. Some ten years ago, some scholars pointed out that the interpretability of the recommended results, the user experience has a crucial impact - users want to know how this recommendation came. In this sense, collaborative filtering has obvious advantages.

Product-based collaborative filtering allows Amazon to tell a user what to recommend a book to when sending a referral email because the user has previously purchased some books. In contrast, matrix decomposition or integrated learning algorithms make it difficult to explain to users the origins of recommended results. Users prefer a recommendation from a friend rather than a system, as will be mentioned later in detail.

In addition, the recommendation list often contains many items, these recommendations are best divided into many categories, different categories tend to come from different recommendations. For example, seen also seen (browse this product has also been viewed by customers), bought also bought (bought this product customers also bought goods), read the final purchase (Browse this product Customer final purchase of goods), personalized hot list (personalized popular recommended) and guess you like (personalized popular product recommendation) and so on.

Of course, how to present the recommendation better is a problem that is difficult to set up a theoretical model and quantify. The guidelines for user interface design may vary widely for different recommendations. For example, in the home page, category page, specific product page, shopping cart page should be placed separately what the recommendation column? Different recommendation column on the page where to optimize the user experience? Under what circumstances should allow users to choose their own recommended personality The degree of the degree? Empirical research based on user behavior can answer some of the questions, at the same time need to establish a system that can do A / B test, or can accumulate important experimental data.

Multidimensional data cross utilization

At present, a popular concept of network science research is the structure and dynamics of networks with interaction. The interaction between the Internet and the Internet can be divided into three categories. The first is dependencies. For example, power networks and the Internet, in the event of a major blackout, the local autonomous systems and routers will be affected, leading to partial network interruption. The second type is cooperation, such as a person's trip, can be seen as a network of aviation, rail network and road network cooperation. The third category is overlap, mainly for social networks, which is our most concern.

Almost every of us is involved in more than one large-scale social network, both Sina Weibo account, RenRen registered users, or mobile phone users, then you have been in three huge social network at the same time. At the same time, you may also often online shopping in Taobao, Jingdong, wheat bags, Shop No. 1, Kuba and other websites, then you become a huge user - a member of the product map.

Imagine being able to integrate these network data, especially knowing the identity of each node (without having to know the true identity, just knowing that some of the nodes in different networks are the same person) Socio-economic value. For example, you may have been on Sina weibo, a lot of data mining people who microblogging, and share a lot of algorithm learning experience and problems. And when you first dangdang online book, the home page to recommend to you the latest data mining monographs with a discount, will you mind? Data mining in social relations overlap, or multidimensional data mining, is the real solution to the system The ultimate magic weapon for internal cold-start problems - as long as the user has activity on other systems outside the system.

Simply from the personalized recommendation of the goods, you can use the user to browse the purchase history of other e-commerce sites to improve the accuracy of the target e-commerce recommendations - of course, each e-commerce is both a payer and a profit. Overall, everyone can achieve a win-win situation by increasing user experience and click depth. At the same time, we can use the activities of Weibo and other social networks to improve the accuracy of product recommendation, and in turn, we can make use of the product browsing history to improve the recommendation accuracy of weibo concern. Recommend a regular professional badminton player and browse various professional badminton equipment users attention Badminton professional players and amateur coach's success rate should be high, and will not fall into the "always in one circle inside and outside the recommended" problems. From the perspective of machine learning, the "migration learning" algorithm proposed by Yang Qiang et al. Is expected to be used to solve this cross-neighborhood recommendation.

We analyzed the real data of percentile technology service customers and found that a significant proportion of users have the habit of cross-shopping (browsing and buying behavior among multiple independent B2C providers). Even considering only two points, for example, the use of wheat bags to browse the purchase of data for the shoe network users personalized recommendation (these users are in the shoe network without any history of new users, but in wheat bags have a browse Purchase behavior) can significantly improve the accuracy of the recommendation (ten times higher than a random recommendation of a fully cold-start). If you use three or more external e-commerce data, the accuracy of the recommendation can be significantly higher than the hot list (note that the hot list no personalization at all), especially in the group website performance is very good.

Although the research on multidimensional data mining has just started, we can fully believe that this will be a focus and a challenge both in academic and application.

Social recommendation

Long ago, researchers found that users prefer a friend's recommendation rather than a "calculated recommendation." Social influence is considered more important than the similarity of historical behavior. For example, through the analysis of social relations, can greatly improve the accuracy of scientific research literature to online shopping product recommendation. Social recommendation from friends has two effects: First, increase sales (including download, read ...), and second, improve the user's rating after the sale.

The effect of social recommendation is not entirely positive either. For example, Leskovec et al. Cite a counter-example in the essay "The Dynamics of Viral Marketing." Friend referrals have had little or no effect on book sales growth, sometimes with negative effects.

In terms of social recommendation, the best done in the country is the watercress network, which is highly regarded by its friends. Recently there is evidence that friend recommendation is also a very important driving force for Taobao sales.

The main challenges in the direction of social recommendation can be divided into three categories: one is how to use social relations to improve the accuracy of recommendation; the other is how to establish a better mechanism to promote social recommendation; and the other is how to introduce social trust into recommendation system . The effect of social recommendation may come from the social influence of word-of-mouth communication, or it may be because friends already have similar interests or are more likely to become friends. Quantifying the differences between these different potential factors is also one of the hot topics in academic research.

Related Reading

Some of my friends may not be very understanding of personalized recommendations, it is recommended to read the review of "Progress in Science and Technology", get a lot of information quickly, to understand the profile of personalized recommendation research. With this foundation, if you want to understand the application-level algorithms and techniques, recommended reading Liang, Chen Yi and Wang Yi co-author of "recommended system practice" book.

At the same time, it is suggested to pay attention to the recently published "Recommender Systems Handbook: A Complete Guide for Scientists and Practioners" and "Recommender Systems: An Introduction." The former is actually a summary of many articles, write the author of these articles are technical experts, the article is of good quality. The article "Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions" by Adomavicius and Tuzhilin is particularly influential, not only systematically reviews the full picture of the proposed systems research, but also presents some interesting The open question. This year in the "Physical Report" there is a large overview of the recommendation system, which should be the most comprehensive nowadays. What we emphasize is not only the algorithm, but also many phenomena and ideas, and it is recommended for everyone to read it.

Author Zhou Tao, director of the Internet Science Center of University of Electronic Science and Technology, Professor, doctoral tutor, Alibaba Business School Qiantang Distinguished Professor, Beijing Institute of Computing Sciences visiting professor. Graduated from the University of Friborg, Switzerland, Ph.D.