Redis 3 ways to achieve a unique count sharing _redis

A unique count is a very common feature of a Web site system, such as the number of visitors to the site who need to count the unique visitor (that is, UV). Counting problems are common, but they can be very complex to solve: first, the amount of the need to count can be very large, such as large sites have millions of of people visit each day, the amount of data is quite large; the second is to expand the count of dimensions, such as in addition to the need for daily UV, but also want to know the weekly or monthly UV, so that the calculation

In a relational database storage system, the only way to achieve a unique count is select COUNT (distinct <item_id>), which is very simple, but if the volume of data is large, the execution of this statement is slow. Another problem with relational databases is the low performance of the insert data.

Redis to solve this kind of counting problem is handy, compared to relational database faster, consumes less resources, and even provides 3 different methods.

1. Based on set

Redis's set is used to hold a unique collection of data, which allows you to quickly determine whether an element exists in a collection, or to quickly compute the number of elements of a collection, and to combine the collection into a new collection. The following commands are involved:

Copy Code code as follows:

Sismember Key member # to determine if member exists
Sadd Key member # to the collection
SCard Key # Gets the number of elements in the collection

Based on the set method is simple and effective, accurate count, wide application, easy to understand, its disadvantage is that the consumption of resources is relatively large (of course, compared to the relational database is much less), if the number of elements (such as hundreds of billions of counts), memory is very scary.

2. Based on bit

The Redis bit can be used to implement a count that is higher than set memory compression, and it stores an element for information through a bit 1 or 0来. For example, the site's unique visitors count, you can take user_id as a bit offset offsets, set to 1 for access, use 1 MB of space can store more than 8 million users of the day access count. The following commands are involved:

Copy Code code as follows:

Setbit key offset Value # set bit information
Getbit key Offset # get bit information
Bitcount key [Start end] # count
Bitop operation Destkey Key [key ...] # Bitmap Merge

The method based on bit is much less than set space consumption, but it requires that the element can be simply mapped to a bit offset, the applicable surface is a lot narrower, and the space it consumes depends on the maximum offset, regardless of the count value, if the maximum offset is large, memory consumption is considerable.

3. Based on Hyperloglog

It is difficult to achieve a unique count of very large amounts of data, but if it is only approximate, there are many efficient algorithms in computational science, of which Hyperloglog counting is one of the most famous algorithms that can only use memory around K to achieve the only count of billions, And the error control is about 1%. The following commands are involved:

Copy Code code as follows:

Pfadd key element [element ...] # Add elements
Pfcount key [Key ...] # count

This counting method is really amazing, I do not understand thoroughly, interested in in-depth research related articles.

The three unique counting methods provided by Redis have advantages and disadvantages, which can meet the requirement of counting in different situations.