Redis3.0 cluster CRC16 algorithm PHP client Implementation method (PHP obtains the Redis partition slot of Redis data in the redis3.0 cluster and obtains the Redis server address of the partition based on the partition slot)

Redis3.0 cluster CRC16 algorithm PHP client Implementation method (PHP obtains the Redis partition slot of Redis data in the redis3.0 cluster and obtains the Redis server address of the partition based on the partition slot)

Data partitioning

Redis clusters store data on multiple nodes, that is, different partitions are stored on different nodes, and each node can store multiple partitions. Each partition is also referred to as a "hash slot" in Redis, with a total of 16,384 partitions planned in the Redis cluster.

For example: when there are 3 nodes in a cluster, Node A will contain 0-5460 partitions, Node B will contain 5461-10922 partitions, and node C will contain 10923-16383 partitions.

each key will be stored in a unique partition, each partition is actually a set of keys, the corresponding relationship is: Key CRC16 check code%16384=hash Slot (partition tag). It is visible that Redis does not use a consistent hash like memecache. The community says the key distribution with this rule is fairly uniform, as evidenced by our testing.

It is quite easy to add or remove a node from a Redis cluster. For example, when you add a new node D, you only need to move some partitions from a, B, and C nodes to D. Similarly, when you remove a, you only need to move the partitions of the genus A to B and C, and so on when a becomes empty.

partition movement between nodes does not need to stop the service, so adding nodes, removing nodes, or changing the number of partitions on a node does not require stopping the Cluster service.

when a client accesses a cluster, it can theoretically access any node in the cluster. At this point, the data being accessed may not exist in the node being accessed, but the Access node can automatically know the target node, redirect the client's access, and return the destination node address to the client, and the client initiates the access request again. Of course, a good client tool should cache the data partition and node correspondence, and update it automatically when the corresponding relationship changes. Currently, several client tools, including Jedis, have implemented this functionality.

CRC concept

CRC fundamentals do not understand, please go to Wikipedia: Cyclic redundancy test code

Different CRC algorithms, such as CRC-1, CRC-8, CRC-16, are usually distinguished based on the number of bits of the CRC checksum (which is also equal to the generation of the highest power of the polynomial "G (x)"). In the case of the same power, different standards have different CRC algorithms. For example, g (x) the highest power is 16 when there are: Crc-16-ccitt, CRC-16-IBM and so on. Redis uses the Crc-16-ccitt standard, which is g (x):x ^{+ x + x5} + 1.

The usual way to characterize G (x) is to convert the polynomial to binary: 1 0001 0000 0010 0001. expressed in hexadecimal as: 0x11021. The number of storage space is 17 bits (2 bytes + 1 bits, the C language is the actual storage is 3 bytes), in fact, at the time of modulo two division, the divisor of the highest bit 1 and the divisor of the highest bit 1 is always aligned, its XOR result, total 0, it can be omitted, then g (x) = 0x1021 (2 bytes Saves a single byte of space. Extract the original from the Sky Star, please indicate the author's origin

Source

crc16.c files in the src directory of Redis:

static const uint16_t crc16tab[256]= {0x0000,0x1021,0x2042,0x3063,0x4084,0x50a5,0x60c6,0x70e7, 0x8108,0x9129,0xa14a , 0xb16b,0xc18c,0xd1ad,0xe1ce,0xf1ef, 0X1231,0X0210,0X3273,0X2252,0X52B5,0X4294,0X72F7,0X62D6, 0x9339,0x8318, 0xb37b,0xa35a,0xd3bd,0xc39c,0xf3ff,0xe3de, 0x2462,0x3443,0x0420,0x1401,0x64e6,0x74c7,0x44a4,0x5485, 0xa56a,    0xb54b,0x8528,0x9509,0xe5ee,0xf5cf,0xc5ac,0xd58d, 0X3653,0X2672,0X1611,0X0630,0X76D7,0X66F6,0X5695,0X46B4, 0XB75B,0XA77A,0X9719,0X8738,0XF7DF,0XE7FE,0XD79D,0XC7BC, 0x48c4,0x58e5,0x6886,0x78a7,0x0840,0x1861,0x2802, 0x3823, 0xc9cc,0xd9ed,0xe98e,0xf9af,0x8948,0x9969,0xa90a,0xb92b, 0x5af5,0x4ad4,0x7ab7,0x6a96,0x1a71,0x0a50, 0x3a33,0x2a12, 0xdbfd,0xcbdc,0xfbbf,0xeb9e,0x9b79,0x8b58,0xbb3b,0xab1a, 0X6CA6,0X7C87,0X4CE4,0X5CC5,0X2C22, 0x3c03,0x0c60,0x1c41, 0xedae,0xfd8f,0xcdec,0xddcd,0xad2a,0xbd0b,0x8d68,0x9d49, 0X7E97,0X6EB6,0X5ED5,0X4EF4,  0x3e13,0x2e32,0x1e51,0x0e70, 0xff9f,0xefbe,0xdfdd,0xcffc,0xbf1b,0xaf3a,0x9f59,0x8f78,  0x9188,0x81a9,0xb1ca,0xa1eb,0xd10c,0xc12d,0xf14e,0xe16f, 0x1080,0x00a1,0x30c2,0x20e3,0x5004,0x4025,0x7046, 0x6067, 0x83b9,0x9398,0xa3fb,0xb3da,0xc33d,0xd31c,0xe37f,0xf35e, 0x02b1,0x1290,0x22f3,0x32d2,0x4235,0x5214, 0x6277,0x7256, 0xb5ea,0xa5cb,0x95a8,0x8589,0xf56e,0xe54f,0xd52c,0xc50d, 0x34e2,0x24c3,0x14a0,0x0481,0x7466, 0x6447,0x5424,0x4405, 0xa7db,0xb7fa,0x8799,0x97b8,0xe75f,0xf77e,0xc71d,0xd73c, 0x26d3,0x36f2,0x0691,0x16b0, 0x6657,0x7676,0x4615,0x5634, 0xd94c,0xc96d,0xf90e,0xe92f,0x99c8,0x89e9,0xb98a,0xa9ab, 0x5844,0x4865,0x7806, 0X6827,0X18C0,0X08E1,0X3882,0X28A3, 0xcb7d,0xdb5c,0xeb3f,0xfb1e,0x8bf9,0x9bd8,0xabbb,0xbb9a, 0x4a75,0x5a54, 0x6a37,0x7a16,0x0af1,0x1ad0,0x2ab3,0x3a92, 0XFD2E,0XED0F,0XDD6C,0XCD4D,0XBDAA,0XAD8B,0X9DE8,0X8DC9, 0x7c26,    0X6C07,0X5C64,0X4C45,0X3CA2,0X2C83,0X1CE0,0X0CC1, 0XEF1F,0XFF3E,0XCF5D,0XDF7C,0XAF9B,0XBFBA,0X8FD9,0X9FF8, 0x6e17,0x7e36,0x4e55,0x5e74,0x2e93,0x3eb2,0x0ed1,0x1ef0};uint16_t CRC16 (const char *buf, int len){int counter;    uint16_t CRC = 0; for (counter = 0; counter < Len; counter++) CRC = (crc<<8) ^ crc16tab[((crc>>8) ^ *buf++) &0x    00FF]; return CRC;}

The previous article refers to the CRC check code different institutions have different standards, where Redis follows the standard is the Crc-16-ccitt standard, which is also used by the XMODEM protocol CRC Standard, so also commonly used Xmodem CRC surrogate.

The Code of the algorithm principle is not the author's first, this is a more classic "byte-based check table method of the CRC Code Generation algorithm" (this article is "To extract the Sky Star" on the weekend, the actual exercise obtained, fully available! ）

The PHP client implements the Redis CRC16 validation standard method .

Function redisCRC16 (& $ptr) {$CRC _table=array (0x0000,0x1021,0x2042,0x3063,0x4084,0x50a5,0x60c6,0x70e7, 0x810    8,0x9129,0xa14a,0xb16b,0xc18c,0xd1ad,0xe1ce,0xf1ef, 0X1231,0X0210,0X3273,0X2252,0X52B5,0X4294,0X72F7,0X62D6, 0x9339,0x8318,0xb37b,0xa35a,0xd3bd,0xc39c,0xf3ff,0xe3de, 0X2462,0X3443,0X0420,0X1401,0X64E6,0X74C7,0X44A4, 0x5485, 0xa56a,0xb54b,0x8528,0x9509,0xe5ee,0xf5cf,0xc5ac,0xd58d, 0x3653,0x2672,0x1611,0x0630,0x76d7,0x66f6, 0X5695,0X46B4, 0XB75B,0XA77A,0X9719,0X8738,0XF7DF,0XE7FE,0XD79D,0XC7BC, 0x48c4,0x58e5,0x6886,0x78a7,0x0840, 0x1861,0x2802,0x3823, 0xc9cc,0xd9ed,0xe98e,0xf9af,0x8948,0x9969,0xa90a,0xb92b, 0x5af5,0x4ad4,0x7ab7,0x6a96, 0x1a71,0x0a50,0x3a33,0x2a12, 0xdbfd,0xcbdc,0xfbbf,0xeb9e,0x9b79,0x8b58,0xbb3b,0xab1a, 0x6ca6,0x7c87,0x4ce4, 0x5cc5,0x2c22,0x3c03,0x0c60,0x1c41, 0xedae,0xfd8f,0xcdec,0xddcd,0xad2a,0xbd0b,0x8d68,0x9d49, 0x7e97,0x6eb6, 0x5ed5,0x4ef4,0x3e13,0x2e32,0x1e51,0x0e70, 0xff9f,0xefbe,0xdfdd,0xcffc,0xbf1b,0xaf3a,0x9f59,0x8f78, 0x9188,0x81a9,0xb1ca,0xa1eb,0xd10c,0xc12d,0xf14e,0xe16f, 0x1080,0x00a1,0x30c2,0x20e3,0x5004,0x4025 , 0x7046,0x6067, 0x83b9,0x9398,0xa3fb,0xb3da,0xc33d,0xd31c,0xe37f,0xf35e, 0x02b1,0x1290,0x22f3,0x32d2,0x4235, 0x5214,0x6277,0x7256, 0xb5ea,0xa5cb,0x95a8,0x8589,0xf56e,0xe54f,0xd52c,0xc50d, 0x34e2,0x24c3,0x14a0,0x0481, 0x7466,0x6447,0x5424,0x4405, 0xa7db,0xb7fa,0x8799,0x97b8,0xe75f,0xf77e,0xc71d,0xd73c, 0x26d3,0x36f2,0x0691, 0x16b0,0x6657,0x7676,0x4615,0x5634, 0xd94c,0xc96d,0xf90e,0xe92f,0x99c8,0x89e9,0xb98a,0xa9ab, 0x5844,0x4865, 0X7806,0X6827,0X18C0,0X08E1,0X3882,0X28A3, 0xcb7d,0xdb5c,0xeb3f,0xfb1e,0x8bf9,0x9bd8,0xabbb,0xbb9a, 0x4a75,    0x5a54,0x6a37,0x7a16,0x0af1,0x1ad0,0x2ab3,0x3a92, 0XFD2E,0XED0F,0XDD6C,0XCD4D,0XBDAA,0XAD8B,0X9DE8,0X8DC9, 0X7C26,0X6C07,0X5C64,0X4C45,0X3CA2,0X2C83,0X1CE0,0X0CC1, 0XEF1F,0XFF3E,0XCF5D,0XDF7C,0XAF9B,0XBFBA,0X8FD9,    0X9FF8, 0X6E17,0X7E36,0X4E55,0X5E74,0X2E93,0X3EB2,0X0ED1,0X1EF0);    $CRC = 0x0000;for ($i = 0; $i < strlen ($ptr); $i + +) $CRC = $CRC _table[(($CRC >>8) ^ ord ($ptr [$i])] ^ (($CRC <<8) & Amp    0X00FFFF); return $CRC;} $test = Chr (0xC6). chr (0xCE). chr (0XA2). chr (0x03); Crc16-ccitt = 0xe2b4$key1= ' Key1 '; $key 2= ' Key2 '; $key 3= ' Key3 '; Echo $key 1_db=rediscrc16 ($key 1)%16384; The data storage slot with a key value of ' Key1 ' in Redis (cluster) is 9189echo "
"; Echo $key 2_DB=REDISCRC16 ($key 2)%16384; The data storage slot with a key value of ' Key2 ' in Redis (cluster) is 4998echo "
"; Echo $key 3_DB=REDISCRC16 ($key 3)%16384; The data storage slot with the key value ' Key3 ' in Redis (cluster) is 935

The Redis data partition slot obtained by the CRC16 algorithm obtains the server address of the partition:

First, let's look at the range of Redis partition slots in the local cluster environment (CD switches to the SRC directory under the Redis installation package directory [Note: The Redis unpacking package is not the directory address of the installed program]):

[[email protected] src]$./REDIS-TRIB.RB check 127.0.0.1:6384connecting to node 127.0.0.1:6384:okconnecting to nod e 127.0.0.1:6381:okconnecting to Node 127.0.0.1:6383:okconnecting to node 127.0.0.1:6379:okconnecting to node 127.0.0.1 : 6380:okconnecting to Node 127.0.0.1:6382:ok>>> performing Cluster Check (using node 127.0.0.1:6384) S:91329DAC B2ac77d9295ed46ecaaec6f2c415f7f6 127.0.0.1:6384 Slots: (0 slots) Slave replicates 0be8c0b96e23a5843ece9d86cb6d287c925 29a8cm:0be8c0b96e23a5843ece9d86cb6d287c92529a8c 127.0.0.1:6381 slots:10923-16383 (5461 Slots) Master 1 additional rep Lica (s) s:53926c1f8b757c6db2d53e12ee94b8c1a761e663 127.0.0.1:6383 slots: (0 slots) Slave replicates beff88cb6dcf6897a6 C6de36350032984a4bdf33m:2fb7a8edab2038d7fabe305dd0099de8bdf1f1e6 127.0.0.1:6379 slots:0-5460 (5461 slots) Master 1 AD Ditional Replica (s) m:beff88cb6dcf6897a6c6de36350032984a4bdf33 127.0.0.1:6380 slots:5461-10922 (5462 slots) Master 1 A Dditional Replica (s) s:4738074195072ae29c3f3160382e97c3b56a6392 127.0.0.1:6382 Slots: (0 slots) Slave replicates 2fb7a8edab2038d7fabe305dd0099d E8bdf1f1e6[ok] All nodes agree on slots configuration.>>> check for open slots...>>> check Slots Cove Rage ... [OK] All 16384 slots covered.

The above content can see the M-bit master, S is from (the official Redis cluster environment must be more than 6 number of even-numbered servers, otherwise unable to create the cluster environment) from the above output can see the partition slot server location: slots:10923-16383 (5461 slots) 127.0.0.1:6381 to 127.0.0.1:6384slots:0-5460 (5461 slots) 127.0.0.1:6379 master should be from the machine address to 127.0.0.1:6381slots : 5461-10922 (5462 slots) 127.0.0.1:6380 master should be from the machine address: 127.0.0.1:6383

There are three primary partitions, which can be positioned to different redis partitions to fetch the corresponding data by the slot position obtained, but if the number of primary partitions is changed, you need to reposition the slot partition range according to the actual partition number to access the Redis address according to Ceil (redisCRC16 (Redis $ Key value)%16384/intval (16384/Cluster master number) to obtain the corresponding Redis cluster server sequential address , and according to the corresponding interval address to the corresponding Redis zone master address, note: If the corresponding master zone hangs, don't worry, just change the address to the master main interval corresponding to the slave from the interval server address to obtain the Redis data, And this is the redis3.0 follow-up version of the subtleties, do not need a third-party plug-in can be in the service-side braking to implement the cluster master mode, will not affect the user normal reading, of course, if it is master and the corresponding slave hanging out there is no way, usually this chance is very small small to can be ignored ... About Redis Add Delete cluster node method please refer to the relevant information, pick up the star original, reproduced please indicate the source of the author, this article temporarily unknown!

