The use of bitmap in Redis

In Redis, we often use commands such as set and get. Have you ever noticed that there are several similar commands, setbit and getbit? what are they used? What is BitMap? it uses a bit to represent the value or state of an element. The key is the corresponding element...

In Redis, we often use commands such as set and get. Have you ever noticed that there are several similar commands, setbit and getbit? what are they used?

What is BitMap?

A bit is used to indicate the value or state of an element. The key is the corresponding element. We know that 8 bits can form a Byte, so bitmap itself greatly saves storage space.

BitMap in Redis

Redis has been added since version 2.2.0.setbit,getbit,bitcountAnd other bitmap commands. Although it is a new command, there is no new data type, becausesetbitWait for the commandset.

Setbit commands

CommandSETBIT key offset value
ComplexityO(1)
Set or clear the bit value of the key (string) at the offset (only 0 or 1 ).

Space occupation and the time required for the first space allocation

On a 2010 MacBook Pro, the offset is 2 ^ 32-1 (512 MB allocated ~ 300 ms, offset is 2 ^ 30-1 (128 MB allocated) needs ~ 80 ms, offset is 2 ^ 28-1 (32 MB allocated) needs ~ 30 ms. if the offset is 2 ^ 26-1 (8 MB allocated), 8 ms is required. <来自官方文档>
The formula for calculating the approximate space usage is: ($offset/8/1024/1024)MB

Scenario 1: user sign-in

Many websites provide the sign-in function (data landing is not considered here), and we need to display the sign-in status in the last month. what should we do if we use bitmap? The code is not clear!

 Connect ('2017. 0.0.1 '); // User uid $ uid = 1; // record uid key $ cacheKey = sprintf ("sign _ % d", $ uid ); // start date with the sign-in function $ startDate = '2017-01-01 '; // today's date $ todayDate = '2017-01-21 '; // calculate the offset $ startTime = strtotime ($ startDate); $ todayTime = strtotime ($ todayDate); $ offset = floor ($ todayTime-$ startTime)/86400 ); echo "Today is day {$ offset ". PHP_EOL; // sign in // how much space does a user occupy in a year? About 365/8 = 45.625 bytes, so small. Is there any Wood stunned? $ Redis-> setBit ($ cacheKey, $ offset, 1); // query the sign-in status $ bitStatus = $ redis-> getBit ($ cacheKey, $ offset ); echo 1 = $ bitStatus? 'Signed in Today ':' No sign in Yet '; echo PHP_EOL; // calculates the total number of sign-in times echo $ redis-> bitCount ($ cacheKey ). PHP_EOL;/*** calculate the number of sign-in times in a certain period of time * unfortunately, bitCount provides the start and end parameters, but this is the position of the string, instead of matching the "bit" position * We are lucky to get the value through the get command and parse it ourselves. And this value is not too big. after one year, a user only needs 45 bytes * set a small target for our website and run it for 30 years, so a total of 1.31KB is required (I will ask you if you are embarrassed ?) * // This is an incorrect calculation method echo $ redis-> bitCount ($ cacheKey, 0, 20). PHP_EOL;

Scenario 2: Count active users

Use the time as the cacheKey, and the user ID is offset. if the user is active on the current day, set it to 1.
So I should calculate the number of active users in a certain day, month, or year (as agreed, online only one day in the statistical period is called active). please use the next redis command.
CommandBITOP operation destkey key [key ...]
Note: bitwise operations are performed on one or more string keys that store binary digits, and the results are saved to destkey.
Note: BITOP supports any of the four operations: AND, OR, NOT, and xor.

// The active user corresponding to the date's hour $ data = array (interval '2017-01-10 '=> array (2017 ), comment '2017-01-11 '=> array (2017, 2017,), comment '2017-01-12' => array ), comment '2017-01-13 '=> array (2017, 2017), comment '2017-01-14' => array () comment )); reschedule // Set the active status in batches before ACH ($ data as $ date => $ uids) {region $ cacheKey = sprintf ("stat _ % s", $ date ); using foreach ($ uids as $ uid) {response $ redis-> setBit ($ cacheKey, $ uid, 1 ); response} comment $ redis-> bitOp ('and', 'Stat', 'stat '). PHP_EOL; // total active users: 6. echo "total active users :". $ redis-> bitCount ('Stat '). PHP_EOL; 
 $ redis-> bitOp ('and', 'stat1', 'stat '). PHP_EOL; // total active users: 2. echo "total active users :". $ redis-> bitCount ('stat1 '). PHP_EOL; 
 $ redis-> bitOp ('and', 'stat2', 'stat _ ', 'stat '). PHP_EOL; // total active users: 8. echo "total active users :". $ redis-> bitCount ('stat2 '). PHP_EOL;

Assume that the current site has 50000000/8 million users, the data per day is about 1024/1024/= 6 MB

Scenario 3: online user status

A project was developed some time ago, and the other party provided me with an interface to query whether the current user is online. I don't know how the other party does it. I thought about it myself. using bitmap is a space-saving and efficient method. I only need a key and the user ID is offset, if it is set to 1 online and 0 if it is not online, users only need 6 MB of space as in the preceding scenario.

// Set the online status in batches $ uids = range (1, 500000); interval foreach ($ uids as $ uid) {region $ redis-> setBit ('online', $ uid, $ uid % 2); identifier} percent // obtain the status values $ uids = range (1, 500000); percent $ startTime = microtime (true ); response foreach ($ uids as $ uid) {response echo $ redis-> getBit ('online', $ uid ). PHP_EOL; timeout} timeout $ endTime = microtime (true); then // on my computer, it takes 25 seconds to obtain the status of 50 million users. echo "total :". ($ endTime-$ startTime ). "s"; 
/** 
 * for batch acquisition, the above is a low efficiency method. actually, you can get the value through get, and then calculate the token by yourself * The specific calculation method can be written another day. the code you wrote earlier cannot be found... Optional */

In fact, BitMap can be used in many scenarios (of course, it will be subject to some restrictions) and its thinking can continue to spread ~ You are welcome to leave a message to discuss it ~

In Redis, we often use commands such as set and get. Have you ever noticed that there are several similar commands, setbit and getbit? what are they used?

What is BitMap?

A bit is used to indicate the value or state of an element. The key is the corresponding element. We know that 8 bits can form a Byte, so bitmap itself greatly saves storage space.

BitMap in Redis

Redis has been added since version 2.2.0.setbit,getbit,bitcountAnd other bitmap commands. Although it is a new command, there is no new data type, becausesetbitWait for the commandset.

Setbit commands

CommandSETBIT key offset value
ComplexityO(1)
Set or clear the bit value of the key (string) at the offset (only 0 or 1 ).

Space occupation and the time required for the first space allocation

On a 2010 MacBook Pro, the offset is 2 ^ 32-1 (512 MB allocated ~ 300 ms, offset is 2 ^ 30-1 (128 MB allocated) needs ~ 80 ms, offset is 2 ^ 28-1 (32 MB allocated) needs ~ 30 ms. if the offset is 2 ^ 26-1 (8 MB allocated), 8 ms is required. <来自官方文档>
The formula for calculating the approximate space usage is: ($offset/8/1024/1024)MB

Scenario 1: user sign-in

Many websites provide the sign-in function (data landing is not considered here), and we need to display the sign-in status in the last month. what should we do if we use bitmap? The code is not clear!

 Connect ('2017. 0.0.1 '); // User uid $ uid = 1; // record uid key $ cacheKey = sprintf ("sign _ % d", $ uid ); // start date with the sign-in function $ startDate = '2017-01-01 '; // today's date $ todayDate = '2017-01-21 '; // calculate the offset $ startTime = strtotime ($ startDate); $ todayTime = strtotime ($ todayDate); $ offset = floor ($ todayTime-$ startTime)/86400 ); echo "Today is day {$ offset ". PHP_EOL; // sign in // how much space does a user occupy in a year? About 365/8 = 45.625 bytes, so small. Is there any Wood stunned? $ Redis-> setBit ($ cacheKey, $ offset, 1); // query the sign-in status $ bitStatus = $ redis-> getBit ($ cacheKey, $ offset ); echo 1 = $ bitStatus? 'Signed in Today ':' No sign in Yet '; echo PHP_EOL; // calculates the total number of sign-in times echo $ redis-> bitCount ($ cacheKey ). PHP_EOL;/*** calculate the number of sign-in times in a certain period of time * unfortunately, bitCount provides the start and end parameters, but this is the position of the string, instead of matching the "bit" position * We are lucky to get the value through the get command and parse it ourselves. And this value is not too big. after one year, a user only needs 45 bytes * set a small target for our website and run it for 30 years, so a total of 1.31KB is required (I will ask you if you are embarrassed ?) * // This is an incorrect calculation method echo $ redis-> bitCount ($ cacheKey, 0, 20). PHP_EOL;

Scenario 2: Count active users

Use the time as the cacheKey, and the user ID is offset. if the user is active on the current day, set it to 1.
So I should calculate the number of active users in a certain day, month, or year (as agreed, online only one day in the statistical period is called active). please use the next redis command.
CommandBITOP operation destkey key [key ...]
Note: bitwise operations are performed on one or more string keys that store binary digits, and the results are saved to destkey.
Note: BITOP supports any of the four operations: AND, OR, NOT, and xor.

// The active user corresponding to the date's hour $ data = array (interval '2017-01-10 '=> array (2017 ), comment '2017-01-11 '=> array (2017, 2017,), comment '2017-01-12' => array ), comment '2017-01-13 '=> array (2017, 2017), comment '2017-01-14' => array () comment )); reschedule // Set the active status in batches before ACH ($ data as $ date => $ uids) {region $ cacheKey = sprintf ("stat _ % s", $ date ); using foreach ($ uids as $ uid) {response $ redis-> setBit ($ cacheKey, $ uid, 1 ); response} comment $ redis-> bitOp ('and', 'Stat', 'stat '). PHP_EOL; // total active users: 6. echo "total active users :". $ redis-> bitCount ('Stat '). PHP_EOL; 
 $ redis-> bitOp ('and', 'stat1', 'stat '). PHP_EOL; // total active users: 2. echo "total active users :". $ redis-> bitCount ('stat1 '). PHP_EOL; 
 $ redis-> bitOp ('and', 'stat2', 'stat _ ', 'stat '). PHP_EOL; // total active users: 8. echo "total active users :". $ redis-> bitCount ('stat2 '). PHP_EOL;

Assume that the current site has 50000000/8 million users, the data per day is about 1024/1024/= 6 MB

Scenario 3: online user status

A project was developed some time ago, and the other party provided me with an interface to query whether the current user is online. I don't know how the other party does it. I thought about it myself. using bitmap is a space-saving and efficient method. I only need a key and the user ID is offset, if it is set to 1 online and 0 if it is not online, users only need 6 MB of space as in the preceding scenario.

// Set the online status in batches $ uids = range (1, 500000); interval foreach ($ uids as $ uid) {region $ redis-> setBit ('online', $ uid, $ uid % 2); identifier} percent // obtain the status values $ uids = range (1, 500000); percent $ startTime = microtime (true ); response foreach ($ uids as $ uid) {response echo $ redis-> getBit ('online', $ uid ). PHP_EOL; timeout} timeout $ endTime = microtime (true); then // on my computer, it takes 25 seconds to obtain the status of 50 million users. echo "total :". ($ endTime-$ startTime ). "s"; 
/** 
 * for batch acquisition, the above is a low efficiency method. actually, you can get the value through get, and then calculate the token by yourself * The specific calculation method can be written another day. the code you wrote earlier cannot be found... Optional */

In fact, BitMap can be used in many scenarios (of course, it will be subject to some restrictions) and its thinking can continue to spread ~ You are welcome to leave a message to discuss it ~

For more information about the usage of bitmap in apsaradb for Redis, refer to PHP!