How to reduce the use of PHP Redis memory method Sharing (graphics)

This article mainly introduces the method of reducing the memory consumption of the PHP Redis. Has a good reference value. Let's take a look at the little series.

1. Reduce the benefits of Redis memory footprint

1. Helps reduce the time taken to create snapshots and load snapshots

2. Improve efficiency when loading aof files and rewriting aof files

3. Shorten the time required to synchronize from the server

4, no need to add additional hardware to enable Redis storage of more data

2. Short structure

Redis provides a set of configuration options for lists, collections, hashes, and ordered collections that enable Redis to store shorter structures in a more economical way.

2.1. Ziplist compression list (list, hash, sequel and)

Commonly used storage methods

When lists, hashes, and ordered collections are shorter or smaller in size, Redis uses a compact storage called Ziplist to store these structures.

Ziplist is a unstructured representation of three different types of objects, such as lists, hashes, and ordered collections, that store data in a serialized fashion, which needs to be decoded each time the serialized data is read and encoded each time it is written.

The difference between a two-way list and a compressed list:

In order to understand that compression lists are more memory-efficient than other data structures, we take a list structure as an example for further research.

Typical two-way list

In a typical bidirectional list, each value will have a node representation. Each node has a pointer to the previous and next node of the list, and a pointer to the string value that the node contains.

Each node contains a string value that is divided into three parts for storage. Includes the string length, the number of available bytes remaining in the string value, the string itself ending with a null character.

Example:

If a node stores the ' ABC ' string, a conservative estimate of 21 bytes is required on a 32-bit platform (three pointers + two int+ null characters: 3*4+2*4+1=21)

As the example shows, storing a 3-byte string requires an extra cost of at least 21 bytes.

Ziplist

A compressed list is a sequence of nodes, each of which contains two lengths and a string. The first length records the length of the previous node (used to traverse the compression list from back to forward); The second length is the length of the current point; the stored string.

Example:

Store string ' abc ', two lengths can be stored in 1 bytes, so the additional overhead is 2 bytes (two lengths is 1+1=2)

Conclusion:

The compression list is achieved by avoiding the storage of additional pointers and metadata, thereby reducing the additional overhead.

Configuration:

#listlist-max-ziplist-entries #表示允许包含的最大元素数量list-max-ziplist-value  #表示压缩节点允许存储的最大体积 #hash         #当超过任一限制后 , the Ziplist method will not be used for storage hash-max-ziplist-entries 512hash-max-ziplist-value 64#zsetzset-max-ziplist-entries 128zset-max-ziplist-value 64

Test List:

1. Establish test.php file

#test. Php<?php$redis=new Redis (); $redis->connect (' 192.168.95.11 ', ' 6379 '); for ($i =0; $i <512; $i + +) {   $ Redis->lpush (' Test-list ', $i. '-test-list '); #往test-list push into 512 data}?>

There are 512 data in the test-list and there are no restrictions in the configuration file.

2. Push a piece of data into the Test-list

At this time test-list contains 513 data, greater than the limit of 512 in the configuration file, the index will discard the Ziplist storage mode, using its original LinkedList storage method

Hashing is the same as an ordered set.

2.2, Intset Integer collection (set)

The precondition is that all member contained in the collection can be parsed into decimal integers.

Storing a collection in an ordered array can not only reduce memory consumption, but also increase the execution speed of the collection operation.

Configuration:

set-max-intset-entries 512 #限制集合中member个数，超出则不采取intset存储

Test:

Create a test.php file

#test. Php<?php$redis=new Redis (); $redis->connect (' 192.168.95.11 ', ' 6379 '); for ($i =0; $i <512; $i + +) {   $ Redis->sadd (' Test-set ', $i);  #给集合test-set insertion of 512 member}?>

2.3, Performance issues

Regardless of the list, hash, ordered collection, and collection, when the limit is exceeded, it is converted to a more typical underlying struct type. Because as the size of the compact structure grows, the speed at which these structures are manipulated becomes slower.

Test:

#将采用list进行代表性测试

Test ideas:

1, in the default configuration to test-list push 50,000 data, to see the time required, and then use Rpoplpush to push test-list data into the new list list-new, to see the time required

2, modify the configuration, List-max-ziplist-entries 100000, and then perform the same operation above

3, compare time, draw a conclusion

Test under Default configuration:

1. Insert data to view time

#test1. Php<?phpheader ("CONTENT-TYPE:TEXT/HTML;CHARSET=UTF8;"); $redis =new Redis (); $redis->connect (' 192.168.95.11 ', ' 6379 '); $start =time (); for ($i =0; $i <50000; $i + +) {   $ Redis->lpush (' Test-list ', $i. '-aaaassssssddddddkkk ');} $end =time (); echo "Insertion time is:". ($end-$start). ' S ';? >

Results take 4 seconds

2. Execute the appropriate commands to view time-consuming

#test2. Php<?phpheader ("CONTENT-TYPE:TEXT/HTML;CHARSET=UTF8;"); $redis =new Redis (), $redis->connect (' 192.168.95.11 ', ' 6379 '); $start =time (); $num =0;while ($redis->rpoplpush (' Test-list ', ' test-new ') {  $num +=1;} Echo ' executions are: '. $num. " <br/> "; $end =time (); echo" Time is: ". ($end-$start). ' S ';? >

Change the configuration file under test

1. Modify the configuration file first

List-max-ziplist-entries 100000 #将这个值修改大一点, can better highlight the impact on performance

List-max-ziplist-value #此值可不做修改

2. Inserting data

Executive test1.php

The result: time consuming 12s

3. Execute the appropriate commands to view time-consuming

Executive test2.php

Result: Execution count: 50000, time consuming 12s

Conclusion:

In this machine to perform test 50,000 data is 8s, if in high concurrency, long compression list and large integer set will not be any optimization, but the performance is reduced.

3, piece structure

The essence of sharding is to divide the data into smaller parts based on simple rules, and then decide where to send the data based on the part that the data belongs to. Many databases use this technique to expand storage space and increase the amount of load they can handle.

In conjunction with the foregoing, it is not difficult to find the importance of the Shard structure for Redis. So we need to make the appropriate adjustments in the configuration file about the Ziplist and the Intset configuration.

3.1. Shard Hash

#ShardHash. class.php

<?phpclass shardhash{Private $redis = "; #存储redis对象/** * @desc Constructor * * @param $host string | Redis host * @param $port int |    Port */Public function construct ($host, $port =6379) {$this->redis=new redis ();  $this->redis->connect ($host, $port); }/** * @desc calculate the Shard ID of a key * * @param $base String | Base Hash * @param $key string | The key name to store in the Shard hash * @param $total int | Expected total number of non-digital shards * * @return String | Returns the Shard key */Public function Shardkey ($base, $key, $total) {if (Is_numeric ($key)) {$shard _id=decbin (substr (bi Ndec ($key), 0,5)); #取 $key binary High five-bit decimal value} else {$shard _id=crc32 ($key)% $shards; #求余取模} return $base. '  _ '. $shard _id; }/** * @desc shard Hash hset operation * * @param $base String | Base Hash * @param $key string | The key name to store in the Shard hash * @param $total int | Estimated total elements * @param $value String/int | Value * * @return BOOL |    Whether Hset succeeded */Public function Shardhset ($base, $key, $total, $value) {$shardKey = $this->shardkey ($base, $key, $total); Return $this->redis->hset ($shardKey, $key, $value); }/** * @desc shard Hash hget operation * * @param $base String | Base Hash * @param $key string | The key name to store in the Shard hash * @param $total int | Estimated total elements * * @return String/false |    Successfully returned value */Public Function shardhget ($base, $key, $total) {$shardKey = $this->shardkey ($base, $key, $total);  return $this->redis->hget ($shardKey, $key); }} $obj =new Shardhash (' 192.168.95.11 '); Echo $obj->shardhget (' hash-', ' key ', 500);? >

Hash shards are based on the basic key and the hash contains the key to calculate the Shard key ID, and then stitching the underlying key into a complete partition key. When executing hset and hget and most hash commands, the key (field) must be processed by the Shardkey method before the Shard key can be used for the next operation.

3.2, the Shard type collection

How do you construct a partitioned collection to make it more memory-efficient and more powerful? The main idea is to convert the stored data in the collection to data that can be parsed into decimal without changing its original functionality. As mentioned earlier, when all members of a collection can be parsed into decimal data, Intset storage is used, which not only saves memory, but also improves the performance of the response.

Example:

If you want a large website, you need to store unique user visits per day. You can then use the unique identifier of the user to convert to a decimal number, and then into the Shard set.

#ShardSet. class.php

<?phpclass shardset{Private $redis = "; #存储redis对象/** * @desc Constructor * * @param $host string | Redis host * @param $ Port int |    Port */Public function construct ($host, $port =6379) {$this->redis=new redis ();  $this->redis->connect ($host, $port); }/** * @desc calculates the Shard key based on the underlying key and the hash contains the key * * @param $base String | Base Hash * @param $key string | The key name to store in the Shard hash * @param $total int | Expected Total Shards * * @return String | Returns the Shard key */Public function Shardkey ($base, $member, $total =512) {$shard _id=crc32 ($member)% $shards; #求余取模 return $base. '  _ '. $shard _id; }/** * @desc count unique user daily visits * * @param $member int | User Unique identifier * * @return string | OK means that count plus 1 false indicates that the user has visited the No 1 */Public Function count ($member) {$shardKey = $this->shardkey (' count ', $member, $to TAL=10);     # $totla a little bit smaller for testing $exists = $this->redis->sismember ($shardKey, $member);      if (! $exists) #判断member今天是否访问过 {$this->redis->sadd ($shardKey, $member);   $this->redis->incr (' count ');   $ttl 1= $this->redis->ttl (' count '); if ($ttl 1===-1) $this->redis->expireat (' Count ', strtotime (date (' y-m-d 23:59:59 '));      #设置过期时间 $ttl 2= $this->redis->ttl ($shardKey); if ($ttl 2===-1) {$this->redis->expireat ("$shardKey", Strtotime (Date (' y-m-d 23:59:59 ')); #设置过期时间 # Echo $shardKey;  #测试使用} #echo $shardKey;    #测试使用 return ' OK ';  } return ' false ';  }} $str =substr (MD5 (UNIQID ()), 0, 8);   #取出前八位 # $str As the customer's unique identifier $str=hexdec ($STR); #将16进制转换为十进制 $obj =new shardset (' 192.168.95.11 '); $obj->count ($STR);? >

4. Converting information packaging into storage bytes

Combined with the previously mentioned Shard Technology, a string shard structure is used to store information for a large number of consecutive ID users.

Use a fixed-length string to allocate n bytes for each ID to store the appropriate information.

Next we will use the storage user country, the province example to explain:

If a user needs to store both Chinese and Guangdong information, using the UTF8 character set, then at least 5*3=15 bytes will be consumed. This will take a lot of resources if the site is large enough to use. The next way we do this is for each user to have just two bytes to complete the stored information.

Specific thinking steps:

1, first of all, we set up a corresponding ' information form ' for the national and National provinces.

2, after the ' Information Form ' is built, it also means that each country, province has the corresponding index number

3, see here everyone should think of it, is to use two index as the user storage information, but it should be noted that we also need to deal with these two indexes corresponding

4, the index as an ASCII code, converted to the corresponding ASCII (0~255) specified by the character

5, using the above-mentioned Shard technology, fixed-length fragmented string structure, the user's storage location to find (a string in Redis can not exceed 512M)

6. Write and remove information (GetRange, SetRange)

Implementation code:

#PackBytes. class.php

<?php# Pack Storage Bytes # Store user country, province information class packbytes{private $redis = "; #存储redis对象/** * @desc Constructor * * @param $host string | Redis host * @param $port int |    Port */Public function construct ($host, $port =6379) {$this->redis=new redis ();  $this->redis->connect ($host, $port); }/** * @desc processing and caching State province data * @param $countries string | First class data, country String * @param $provinces two-dimensional array | Type II data, National provinces array * @param $cache 1/0 | Whether to use cache, default 0 does not use * * @return Array | Total data returned */Public function DealData ($countries, $provinces, $cache =0) {if ($cache) {$result = $this->redis-&gt      ; get (' cache_data ');    if ($result) return unserialize ($result);    } $arr =explode ("', $countries);    $AREAARR []= $arr;    $AREAARR []= $provinces;    $cache _data=serialize ($AREAARR);    $this->redis->set (' Cache_data ', $cache _data);  return $AREAARR; }/** * @desc to convert specific information into encoded information by Table index * * @param $countries, $provinces, $cache | Reference Dealdata Method * @param $country String | Specific Information-Country * @param $provInce String | Specific Information-province * * @return String | Returns the encoded information for the transformation */Public function GetCode ($countries, $provinces, $country, $province, $cache =0) {$dataArr = $this->dealdat    A ($countries, $provinces, $cache =0); $result =array_search ($country, $DATAARR [0]);  #查找数组中是否含有data1 if ($result ===false) #判断是否存在 return chr (0). chr (0);    #不存在则返回初始值 $code =CHR ($result); $result =array_search ($province, $DATAARR [1][$country]);    #查找数组中是否含有data2 if ($result ===false) return $code. chr (0);   Return $code. chr ($result); The character specified by the #返回对应ASCII (0~255)}/** * @desc calculates the relevant location information for the user to store the encoded data * * @param $userID int | User's ID * * @return array |   Returns an array containing the Shard ID of the data store, and the storage location (offset) belonging to the user */Public Function saveposition ($userID) {$shardSize =pow (2, 3);    #每个分片的大小 $position = $userID * *;  #user的排位 $arr [' Shardid ']=floor ($position/$shardSize);   #分片ID $arr [' offset ']= $position% $shardSize;  #偏移量 return $arr; }/** * @desc | Consolidation method to place encoded information in the corresponding position of string in Redis * * @param $userID int | User ID  * @param $countries String | First class data, country String * @param $provinces two-dimensional array | Type II data, National provinces array * @param $country string | Specific Information-Country * @param $province string | Specific information--province * @param $cache 1/0 | Whether to use cache, default 0 does not use * * @return successfully return write location/fail false */Public function Savecode ($userID, $countries, $provinces, $country, $provinc    E, $cache =0) {$code = $this->getcode ($countries, $provinces, $country, $province, $cache =0); $arr = $this->saveposition ($userID);  #存储相关位置信息 return $this->redis->setrange (' Save_code_ ' $arr [' Shardid '], $arr [' offset '], $code); }/** * @desc get information about the country and province of the user * * @param $userID int | User ID * * @return array |    Returns an array containing country and province information */Public Function getMessage ($userID) {$position = $this->saveposition ($userID); $code = $this->redis->getrange (' Save_code_ '. $position [' Shardid '], $position [' offset '], $position [' Offset ']+1    );    $arr =str_split ($code); $AREAARR = $this->dealdata (",", $cache =1); #使用缓存数据 $message [' Country ']= $areaArr [0][ord ($arr [0])];    $message [' Province ']= $areaArr [1][$message [' Country ']][ord ($arr [1])];  return $message; }}header ("CONTENT-TYPE:TEXT/HTML;CHARSET=UTF8;"); $countries = "No china Japan Vietnam Korea Russia Pakistan USA"; $provinces =array (' None ' =>array (' none '), ' China ' =>array (' None ', ' Guangdong ', ' Hunan ', ' Hubei ', ' Guangxi ', ' Yunnan ', ' Hunan ', ' Hebei '), ' Japan ' =>array (' None ', ' Turtle grandson area ', ' King Eight district ', ' Umayyad Ghost zone ', ' devil zone ', ' Turnip Head area '), $obj =new packbytes (' 192.168.95.11 ');/*# Data processing and cache it into Redis $b= $obj->dealdata ($countries, $provinces); echo "<pre>";p Rint_r ($b); echo "</pre>"; Die *//* #存储用户国家省份信息 $country = ' China '; $province = ' Guangdong '; $result = $obj->savecode (0, $countries, $provinces, $country, $ Province), echo "<pre>";p Rint_r ($result), echo "</pre>", *//* #取出用户国家省份信息 $a = $obj->getmessage, echo "<pre>";p rint_r ($a); echo "</pre>";d ie;*/?>

Test:

1, Dealdata processing information, that is, ' Information Form form '

2, Savecode ()

Userid	Countries	Provinces
0	China	Guangdong
13	Japan	Turtle grandson Area
15	Japan	Wang Eight District

3, GetMessage ()