Use Redis bitmap to count active users, retained

Use Redis bitmap to count active users, retained

Spool's developer blog describes Spool's use of Redis's bitmaps-related operations to make website active user statistics work.

Original: http://blog.getspool.com/2011/11/29/fast-easy-realtime-metrics-using-redis-bitmaps/

Redis supports BITS-based bitwise operations on value of type string. By placing a user's ID on one of the value, and by placing a bit on the corresponding bit of the active user, the information of all active users can be recorded with a value. If not, the bitmap in the 9 bits are set to 1, indicating that the corresponding user on these 9 bits is today's active user. The 15th digit represents a user with a UID of 15, and the first bit represents the user with a UID of 0. (If your UID is not starting from 1, such as starting from 100000, you can actually use the UID minus the initial value to indicate its number of digits, such as 1000000 users corresponding to the first bit of bitmap)

　　

The specific code resembles the following:

Redis.setbit (PLAY:YYYY-MM-DD, user_id, 1)

The complexity of such a record is O (1), which is very fast in Redis.

Instead, we separate daily active user state records by swapping a different key each day. And it can calculate the N-day active user by some and or operation, and the statistic data such as the active user of N-day is connected.

For example, the first line represents the active user situation for Monday, the second row represents Tuesday, and so on. As a sample we can draw a list of users who have been active in n days by taking a set operation on an active user record for n days.

　　

The table below represents the time spent on the day, week, and one months of statistics.

　　

Here is a concrete snippet of Java code:

Figure out the number of active users per day

Import Redis.clients.jedis.Jedis;
Import Java.util.BitSet;
...
Jedis Redis = new Jedis ("localhost");
...
public int Uniquecount (string action, string date) {
String key = action + ":" + date;
BitSet users = bitset.valueof (Redis.get (Key.getbytes ()));
return users.cardinality ();
}

Calculate the number of active users within a few (one day active even if, so is the collection)

Import Redis.clients.jedis.Jedis;
Import Java.util.BitSet;
...
Jedis Redis = new Jedis ("localhost");
...
public int Uniquecount (string action, string ... dates) {
BitSet all = new BitSet ();
for (String date:dates) {
String key = action + ":" + date;
BitSet users = bitset.valueof (Redis.get (Key.getbytes ()));
All.or (users);
}
return all.cardinality ();
}

There are a lot of specific uses, such as you can also have a unique terminal user to remember a bitmap, so that you can count the different end-user activity. Some students will say that using set can also achieve the same effect. However, using set is much larger in memory usage.

==========================================================================

After reading this article, I tested it:
redis> setbit bit 10086 1
(integer) 0
redis> getbit bit 10086
(integer) 1

For setbit operations that use large offset, the first memory allocation may cause the Redis server to be blocked. Because Redis needs to produce a very long binary series.
Problem:

If active users are millions, using Redis bitmap is a good deal.

If there are few active users, and the user ID is more than 10-bit int. That would be a waste of memory. It's better to use set set. Then ask for the intersection to be able.

We can calculate the memory: offset = 999 999 999 = "Required memory 999 999 999/8/1024/1024 = around 119M."

If there are many dimensions to the statistics, and there are thousands of dimensions combined, it is not a good way to use it. We can learn from bitmap using another way to statistically active retention:

Retained indicators:
Registration of the next day,
2nd registration ...
N days registered and retained,
For example, yesterday registered 1000 users, in today there are 300 users and logged in, then the corresponding to yesterday's registered retention is 30%;
In general, these metrics depend on the core variable-the user access time.
Then we can use bitmap to record user access time:

If we are counting time starting from 2013, then 2013-01-01 is the first bit ... And so on
The last day of 2013 is the No. 365 bit of bit.

This allows us to record whether the user is logged on all days.
Then we calculate the retention:
Retention calculation:
1) Calculate the time of day, corresponding bit bit, now he is July 01, bit bit is 182.
2) retention on the following day:
View bit (182-1) = 181 bit exists, if present, number of retention +1
N Days retained:
Check if bit (182-n) is present, if present, n-Day retention number +1
Let's try to estimate the space occupied. 365bit bits a year. 10 million user, occupied space = 10 million * 365BIT/8/1024/1024 = 430M

Use Redis bitmap to count active users, retained