Reduces PHP Redis memory usage and phpredis memory usage

Reduces PHP Redis memory usage and phpredis memory usage

1. Advantages of reducing redis memory usage

1. It helps reduce the time used to create and load snapshots.

2. Improve the efficiency of loading AOF files and rewriting AOF files

3. shorten the time required for synchronization from the server

4. redis can store more data without adding additional hardware

2. Short Structure

Redis provides a set of configuration options for lists, sets, hashes, and ordered sets. These options allow redis to store shorter structures in a more economical way.

2.1 ziplist compression list (list, hash, sequent and)

Storage Method

When the length of a list, hash, or ordered set is short or small, redis uses a compact storage method called ziplist to store these structures.

Ziplist is an unstructured representation of the list, hash, and ordered set objects. It stores data in a serialized manner, the serialized data needs to be decoded every time it is read and encoded every time it is written.

Differences between a two-way list and a compressed list:

To understand that the compressed list is more memory-saving than other data structures, we will take the list structure as an example for further research.

Typical bidirectional list

In a typical bidirectional list, each value is represented by a node. Each node has a pointer pointing to the previous node and the next node of the linked list, and a pointer pointing to the string value contained in the node.

Each node contains string values stored in three parts. Including the string length, the number of available bytes in the string value, and the string itself ending with an empty character.

Example:

If a node stores the 'abc' string, the additional overhead of 21 bytes is estimated conservatively on a 32-bit platform (three pointers + two int + null characters: 3*4 + 2*4 + 1 = 21)

The example shows that the additional overhead of at least 21 bytes is required to store a 3-byte string.

Ziplist

A compressed list is a sequence composed of nodes. Each node contains two lengths and a string. The first length records the length of the previous node (used to traverse the compressed list from the back to the front), the second length is the length of the current vertex of the record, and the stored string.

Example:

The storage string 'abc' can be stored in 1 byte. Therefore, the additional overhead is 2 bytes (two lengths: 1 + 1 = 2)

Conclusion:

Compressing a list reduces the overhead by avoiding storing additional pointers and metadata.

Configuration:

# Listlist-max-ziplist-entries 512 # indicates the maximum number of elements allowed to be included. list-max-ziplist-value 64 # indicates the maximum size allowed by the compression node. # hash # When the size exceeds any after the limit, the ziplist method will not be used to store hash-max-ziplist-entries 512hash-max-ziplist-value 64 # zsetzset-max-ziplist-entries 128zset-max-ziplist-value 64

Test list:

1. Create the test. php file.

# Test. php <? Php $ redis = new Redis (); $ redis-> connect ('2017. 168.95.11 ', '123'); for ($ I = 0; $ I <6379; $ I ++) {$ redis-> lpush ('test-list ', $ I. '-test-list'); # Push 512 data records to test-list.}?>

In this case, the test-list contains 512 pieces of data, which is not restricted in the configuration file.

2. Push another data entry to test-list.

In this case, the test-list contains 513 pieces of data, which is greater than the limit of 512 pieces in the configuration file. The ziplist storage method is abandoned for the index, and the original sort list storage method is used.

The same is true for hash and ordered set.

2.2. intset Integer Set (SET)

Prerequisite: All member contained in the set can be parsed as a decimal integer.

Storing a set in an ordered array not only reduces memory consumption, but also increases the execution speed of the set operation.

Configuration:

Set-max-intset-entries 512 # limit the number of members in the set. If the number exceeds the limit, intset storage is not used.

Test:

Create the test. php file

# Test. php <? Php $ redis = new Redis (); $ redis-> connect ('2017. 168.95.11 ', '123'); for ($ I = 0; $ I <6379; $ I ++) {$ redis-> sadd ('test-set ', $ I); # insert 512 members to the test-set?>

2.3 performance problems

Regardless of the list, hash, ordered set, and set, when the limit is exceeded, it is converted to a more typical underlying structure type. As the size of the compact structure increases, the operation speed of these structures will become slower and slower.

Test:

# List will be used for representative testing

Test ideas:

1. Push 50000 pieces of data to test-list under the default configuration to view the required time. Then, use rpoplpush to push all test-list data to the new list-new, view the time required

2. modify the configuration, list-max-ziplist-entries 100000, and then perform the same operation as above.

3. Compare the time and draw a conclusion

Test by default:

1. Insert data and view the time

# Test1.php <? Phpheader ("content-type: text/html; charset = utf8;"); $ redis = new Redis (); $ redis-> connect ('2017. 192. 168.95.11 ', '123'); $ start = time (); for ($ I = 0; $ I <6379; $ I ++) {$ redis-> lpush ('test-list', $ I. '-aaaassssssddddkkk');} $ end = time (); echo "insertion time :". ($ end-$ start ). 'S ';?>

The result takes 4 seconds.

2. Execute the corresponding command to view the time consumed

# Test2.php <? Phpheader ("content-type: text/html; charset = utf8;"); $ redis = new Redis (); $ redis-> connect ('2017. 192. 168.95.11 ', '123'); $ start = time (); $ num = 0; while ($ redis-> rpoplpush ('test-list ', 'test-new') {$ num + = 1;} echo :'. $ num. "<br/>"; $ end = time (); echo :". ($ end-$ start ). 'S ';?>

Change the configuration file to test

1. modify the configuration file first.

List-max-ziplist-entries 100000 # modify this value a little bit to better highlight the impact on performance

List-max-ziplist-value 64 # do not modify this value

2. Insert data

Run test1.php

The result is 12 s.

3. Execute the corresponding command to view the time consumed

Run test2.php

Result: execution times: 50000, which takes 12 s

Conclusion:

If you perform a test on the local machine for 50000 pieces of data, the difference is 8 s. In high concurrency, the long compression list and the large integer set cannot be optimized, but the performance is reduced.

3. Chip structure

The essence of sharding is to divide the data into smaller parts based on simple rules, and then decide the location where the data is sent based on the part of the data. Many databases use this technology to expand their storage space and increase the load they can handle.

In combination with the previous sections, it is not difficult to find the significance of the partition structure for redis. Therefore, we need to make appropriate adjustments to ziplist and intset configurations in the configuration file.

3.1 split-chip hash

# ShardHash. class. php

<? Phpclass ShardHash {private $ redis = ''; # store redis object/*** @ desc constructor ** @ param $ host string | redis host * @ param $ port int | port */public function _ construct ($ host, $ port = 6379) {$ this-> redis = new Redis (); $ this-> redis-> connect ($ host, $ port );} /*** @ desc calculates the shard ID of a key. ** @ param $ base string | basic hash * @ param $ key string | Name of the key to be stored in the shard hash * @ param $ total int | estimated total number of non-numeric parts ** @ return string | return partition key */publi C function shardKey ($ base, $ key, $ total) {if (is_numeric ($ key) {$ shard_id = decbin (substr (bindec ($ key), 0, 5 )); # Take the decimal value of the $ key binary high five digits} else {$ shard_id = crc32 ($ key) % $ shards; # evaluate the remainder modulo} return $ base. '_'. $ shard_id ;} /*** @ desc sharded hset operation ** @ param $ base string | basic hash * @ param $ key string | Name of the key to be stored in the sharded hash * @ param $ total int | estimated total number of elements * @ param $ value string/int | value ** @ return bool | whether hset is successful */public functi On shardHset ($ base, $ key, $ total, $ value) {$ shardKey = $ this-> shardKey ($ base, $ key, $ total ); return $ this-> redis-> hset ($ shardKey, $ key, $ value );} /*** @ desc sharded hget operation ** @ param $ base string | basic hash * @ param $ key string | Name of the key to be stored in the sharded hash * @ param $ total int | estimated total number of elements ** @ return string/false | value */public function shardHget ($ base, $ key, $ total) {$ shardKey = $ this-> shardKey ($ base, $ key, $ total); return $ This-> redis-> hget ($ shardKey, $ key) ;}}$ obj = new ShardHash ('2017. 168.95.11 '); echo $ obj-> shardHget ('hash-', 'key', 500);?>

Hash partitions calculate the partition key ID based on the base key and the key contained in the hash, and then splice it with the base key to form a complete partition key. When executing hset, hget, and most hash commands, you must first process the key (field) through the shardKey method to obtain the partition key for the next operation.

3.2. sharded collection

How can we create a chip collection to save memory and improve performance? The main idea is to convert the data stored in the set into data that can be parsed into decimal digits without changing its original functions. As mentioned above, when all the members in the set can be parsed into decimal data, intset storage will be used, which not only saves memory, it can also improve the response performance.

Example:

If you want a large website to store the only user traffic each day. Then, you can convert the unique identifier of a user into a decimal number and store it in a chip set.

# ShardSet. class. php

<? Phpclass ShardSet {private $ redis = ''; # store redis object/*** @ desc constructor ** @ param $ host string | redis host * @ param $ port int | port */public function _ construct ($ host, $ port = 6379) {$ this-> redis = new Redis (); $ this-> redis-> connect ($ host, $ port );} /*** @ desc calculate the partition key based on the basic key and the key contained in the hash. ** @ param $ base string | basic hash * @ param $ key string | key name * @ param $ total int | expected total number of parts ** @ return string | return partition key */p Ublic function shardKey ($ base, $ member, $ total = 512) {$ shard_id = crc32 ($ member) % $ shards; # evaluate the remainder modulo return $ base. '_'. $ shard_id ;} /*** @ desc calculates the daily access volume of a unique user ** @ param $ member int | Unique User Identifier ** @ return string | OK indicates that count and 1 false indicate that the user has accessed 1 */public function count ($ member) {$ shardKey = $ this-> shardKey ('Count', $ member, $ total = 10 ); # $ totla tune down to $ exists = $ this-> redis-> sismember ($ shardKey, $ member); if (! $ Exists) # judge whether member has accessed {$ this-> redis-> sadd ($ shardKey, $ member) today ); $ this-> redis-> incr ('Count'); $ ttl1 = $ this-> redis-> ttl ('Count '); if ($ ttl1 ===- 1) $ this-> redis-> expireat ('Count', strtotime (date ('Y-m-d 23:59:59 '))); # Set the expiration time $ ttl2 = $ this-> redis-> ttl ($ shardKey); if ($ ttl2 ===-1) {$ this-> redis-> expireat ("$ shardKey", strtotime (date ('Y-m-d 23:59:59 '); # Set the expiration time # echo $ shardKey; # Test Run} # echo $ shardKey; # test run r Eturn 'OK';} return 'false'; }}$ str = substr (md5 (uniqid (), 0, 8 ); # retrieve the first eight digits # use $ str as the unique identifier of the customer $ str = hexdec ($ str); # convert the hexadecimal format to the decimal format $ obj = new ShardSet ('123. 168.95.11 '); $ obj-> count ($ str);?>

4. Package and convert information into storage bytes

Based on the sharding technology mentioned above, the string sharding structure is used to store information for a large number of consecutive ID users.

A fixed-length string is used to allocate n Bytes for each ID to store the corresponding information.

Next, we will explain the example of storing the user's country and province:

If a user needs to store the information of China and Guangdong Province and uses the utf8 character set, at least 5*3 = 15 bytes are required. If the website has a large number of users, this will occupy a lot of resources. The following method is used. Each user only needs to occupy two bytes to store the information.

Steps:

1. First, we will create an information table for country and province information'

2. After the 'information table' is created, each country and province has an index number.

3. You should have thought of it here, right? Two indexes are used as the user's storage information, but note that we still need to process these two indexes accordingly.

4. Use the index as an ASCII code and convert it to the corresponding ASCII code (0 ~ 255) specified characters

5. Use the sharding technology mentioned above to determine the long shard string Structure and locate the user's storage location (a string in redis cannot exceed 512 MB)

6. Write and retrieve information (getrange and setrange)

Implementation Code:

# PackBytes. class. php

<? Php # package storage bytes # store user country and province Information class PackBytes {private $ redis = ''; # store redis object/*** @ desc constructor ** @ param $ host string | redis host * @ param $ port int | port */public function _ construct ($ host, $ port = 6379) {$ this-> redis = new Redis (); $ this-> redis-> connect ($ host, $ port );} /*** @ desc process and cache the data of national provinces * @ param $ countries string | the first type of data, country string * @ param $ provinces two-dimensional array | second type of data, province array * @ param $ cache 1/0 | indicates whether to use the cache. The default value is 0. Use ** @ return array | to return the total data */public function dealData ($ countries, $ provinces, $ cache = 0) {if ($ cache) {$ result = $ this-> redis-> get ('cache _ data'); if ($ result) return unserialize ($ result );} $ arr = explode ('', $ countries); $ areaArr [] = $ arr; $ areaArr [] = $ provinces; $ cache_data = serialize ($ areaArr ); $ this-> redis-> set ('cache _ data', $ cache_data); return $ are1_r ;} /*** @ desc converts the specific information to the encoding information based on the table index. ** @ param $ count Ries, $ provinces, $ cache | refer to the dealData Method * @ param $ country string | specific information -- country * @ param $ province string | specific information -- province ** @ return string | return the converted encoding information * /public function getCode ($ countries, $ provinces, $ country, $ province, $ cache = 0) {$ dataArr = $ this-> dealData ($ countries, $ provinces, $ cache = 0 ); $ result = array_search ($ country, $ dataArr [0]); # Check whether the array contains data1 if ($ result = false) # determine whether return chr (0) exists ). chr (0); # returns the initial value if it does not exist. $ Code = chr ($ result); $ result = array_search ($ province, $ dataArr [1] [$ country]); # Check whether the array contains data2 if ($ result = false) return $ code. chr (0); return $ code. chr ($ result); # returns the corresponding ASCII (0 ~ 255) the specified character}/*** @ desc calculates the location information of the user's stored encoding data ** @ param $ userID int | user ID ** @ return array | returns an array contains the shard ID during data storage and the storage location (offset) of the user) */public function savePosition ($ userID) {$ shardSize = pow (2, 3); # the size of each shard $ position = $ userID * 2; # user ranking $ arr ['shard'] = floor ($ position/$ shardSize); # shard ID $ arr ['offset'] = $ position % $ shardSize; # offset return $ arr;}/*** @ desc | the integration method stores the encoding information to the corresponding string location in redis. ** @ param $ us ErID int | user ID * @ param $ countries string | Type 1 data, country string * @ param $ provinces two-dimensional array | Type 2 data, province array * @ param $ country string | country * @ param $ province string | province * @ param $ cache 1/0 | indicates whether to use the cache, by default, 0 does not use ** @ return to return the write success location/failure false */public function saveCode ($ userID, $ countries, $ provinces, $ country, $ province, $ cache = 0) {$ code = $ this-> getCode ($ countries, $ provinces, $ country, $ province, $ cache = 0 ); $ Arr = $ this-> savePosition ($ userID); # store the relevant location information return $ this-> redis-> setrange ('Save _ code _'. $ arr ['shard'], $ arr ['offset'], $ code );} /*** @ desc obtain the user's country and province information ** @ param $ userID int | user ID ** @ return array | returns an array containing country and province information */public function getMessage ($ userID) {$ position = $ this-> savePosition ($ userID); $ code = $ this-> redis-> getrange ('Save _ code _'. $ position ['shard'], $ position ['offs'], $ position ['offs Et '] + 1); $ arr = str_split ($ code); $ arew.r = $ this-> dealData ('','', $ cache = 1 ); # Use cached data $ message ['country'] = $ arew.r [0] [ord ($ arr [0])]; $ message ['province '] = $ arew.r [1] [$ message ['country'] [ord ($ arr [1])]; return $ message ;}} header ("content-type: text/html; charset = utf8;"); $ countries = "None China Japan Vietnam North Korea Russia Pakistan United States "; $ provinces = array ('none' => array ('None'), 'China' => array ('none', 'guangdong ', 'hunan', 'hubei ', 'guangxi ', 'yunnan', 'hunan ', 'hebei'), 'Japan '=> Rray '),); $ obj = new PackBytes ('20180101. 168.95.11 ');/* # process data and cache it to redis $ B = $ obj-> dealData ($ countries, $ provinces); echo "<pre> "; print_r ($ B); echo "</pre>"; die; * // * # store the user's National province Information $ country = 'China'; $ province = 'guangdong '; $ result = $ obj-> saveCode (0, $ countries, $ provinces, $ country, $ province); echo "<pre>"; print_r ($ result ); echo "</pre>"; * // * # retrieve the province information of the user's country $ a = $ obj-> getMessage (15); echo "<pre>"; print _ R ($ a); echo "</pre>"; die; */?>

Test:

1. dealData processed information, that is, 'table of information'

2. saveCode ()

UserID	Country	Province
0	China	Guangdong
13	Japan	Chusun District
15	Japan	Wangba District

3. getMessage ()

The above is all the content of this article. I hope this article will help you in your study or work. I also hope to provide more support to the customer's home!