Topk-Weibo today's hot Topic

Outline

• TopK on single node
• TopK on multiple nodes
• Realtime TopK with Low QPS
• Realtime TopK with high QPS
• Approx TopK
• Mapreduce

One, TopK on single node

TopK series problems from several algorithms about TopK

1. Give you an unordered array of integers that asks for the TOPK (Order by Value)

Title Address: http://www.lintcode.com/zh-cn/problem/top-k-largest-numbers/

Data structure: Priority queue (MINHEAP) (if not data flow, use quickselect more efficiently)

Complexity of Time: O (NLOGK)

Space complexity: O (k)

2. Give you a list of Weibo topics and ask for TOPK (Order by Frequency)

Title Address: http://www.lintcode.com/zh-cn/problem/top-k-frequent-words/

Analysis: It is necessary to solve TOPK in terms of the frequency of the string appearing, and naturally it cannot be implemented directly using a priorityqueue as it was just before. But the fundamentals are still the same.

Using a hashmap,hashmap<string, integer> indicates the frequency at which a String appears. Then in the PQ is stored in our custom a data structure pair, pair contains string and frequency two variables, custom a comparator in ascending order by the frequency is possible.

Data structure: HashMap Priorityqueue

Time complexity: O (n + nlog (k)) O (NLOGK)

Space complexity: O (|n| + k) where |n| represents the number of unique strings

Second, TopK on multiple nodes

1. Now suppose such a scenario: give you a set of 10T files, the file content is 10million users of the day's search records, micro-bo today's topic hot search?

This scene can no longer use single node because the file is too large, the machine cannot be processed, on the other hand, processing speed is too slow

At this time, we should adopt the idea of &

Overview as follows:

• Split into small files
• distributed to different machines for processing
• Each machine gets TOPK, respectively.
• Combine these topk to get the total TOPK

　　

Note Here a more critical place: How to split the file?

One way of thinking is to split the order of the files, which is problematic, because if a string is more scattered, and the total number of times is able to enter the TOPK, but on Slavenode, the Slavenode may not be selected TOPK, which led to the error.

So, we use Divide by hash value here. So the same string is assigned to the same slavenode processing.

2. Suppose scenario Two: There are n machines, each machine each store word files, to find the frequency of all words TOPK

In the same way, if the topk of individual machines is merged, the problem will arise.

We need rehash! here.

Iii. Realtime TopK with low QPS

What is the real-time data about the offline of the situation?

Topk-Weibo today's hot Topic