Pick one: Reason and system design

Personal usually more like to read some news information, such as science and technology Huxiu, 36KR, sports news, etc., to the relevant app is also practical, today's headlines do very well. There are so many people around. However, after a period of time, it is found that many apps have the following features:

1. The information is numerous and miscellaneous, even if I subscribe or focus on certain categories only. The push message first is too much and second is irrelevant.

Too much information I can not consume, irrelevant information I am more disgusted.

2. Today's app claims to be able to make accurate and personalized recommendations, and the headlines do just fine. But the feeling can not capture the user's interest change in time. The recommended results are also small. Surprise is not enough.

3. There is a lot of repetition in the news information of the aggregation class. And much more than simply grabbing and presenting, there's not much improvement in the way and experience of reading.

The above is probably a place to feel some inconvenience after use. Before doing a period of recommendation and text processing related things, coupled with some of their own ideas, want to achieve a simple system, take their own experiment to try, or verify their own ideas, in response to the above questions, the personal idea is 1. Every day to show users a certain amount of valuable news, that is, to limit the number of users to push the news. The relevance aspect needs to be modeled on the user's characteristics, the expected effect is not obvious, only can be controlled by some strategies, such as the hottest and related combination, an event or a category to show a news and other strategies to achieve. 2. Update the user's feature weights in a timely manner for the user's behavior. And make the change a little bit more real time. 3. Very many people read the article is only to see the main idea of the article, very little read through the full text, assuming that the article can be summarized. It should be better for the app class. But now there seems to be no good way to digest Chinese. Just keep on trying to improve. I will use the abstract algorithm introduced in the previous article to experiment, combined with Chinese lexical and semantic to do some attempts.

These are purely personal views and opinions. There must be something wrong with this idea of being able to communicate with each other.

The development work has been done for some now. has been using Java to do web-related services and design. The general cloud server running Java is expensive. Therefore, Python is used to carry out the related development work. The simple design of the system is as follows:

The system is mainly divided into online service, OffLine service, in which the online section mainly carries out the following operations:

a). Fetcher uses UA and PA to get the recommended news data, first to request the relevant data for Redis calculation. Then to MySQL to get the data, now assume that MySQL can meet a certain amount of concurrent requests, in the future can be considered according to the data type in front of MySQL to add a layer of cache.

b). Updater is primarily based on user behavior to update the UA weights in the cache, so that the next time the user can be based on the latest behavior of the recommended display.

Offline Part is mainly responsible for the processing of the offline logic, mainly including the fetch data cleaning, feature extraction, digest, warehousing and other operations, in order to understand the decoupling, using MQ to store the captured data.

At the moment, the Tornado framework is used to provide Web services, and Redis acts as a cache store for data, and MySQL as the underlying data store. RABBITMQ as a message queue, Jieba word breaker for Chinese word segmentation, Redis + MySQL has now been implemented. The design and implementation of the Web's main remaining pages, feature extraction and summary are ongoing. Because things are more, the last thing that can be achieved is very different from what is said in the article. Next, we will talk about the implementation process and effect of some ideas, depending on the progress and work. Suppose you are interested in being able to communicate together.

Pick one: Reason and system design