Redis Foundation, advanced features and performance tuning __redis

Redis Foundation, advanced features and performance tuning
Kelgon attention 2017.02.28 16:22 words 12597 read 16333 comments 5 like 154 appreciate 2

This paper will start from the basic characteristics of redis, through the description of the REDIS data structure and the main command of the Redis basic capabilities of the intuitive introduction. After that, the advanced capabilities provided by Redis are described in more detail and guidance in the areas of deployment, maintenance, and performance tuning.
This article is suitable for the common developers who use Redis, as well as the architecture designers who make selection, architecture design and performance tuning for Redis. 

Catalog

Overview Redis data structure and related common command data persistent memory management and data elimination mechanism pipelining transaction and scripting Redis performance tuning master-slave replication and cluster fragmentation Redis Java Client selection

  Overview

Redis is an open-source, memory-based structured data storage medium that can be used as a database, caching service, or messaging service.
Redis supports a variety of data structures, including strings, hash tables, lists, collections, ordered sets, bitmaps, hyperloglogs, and so on.
Redis has the capability of LRU phase-out, transactional implementation, and different levels of hard disk persistence, and supports a replica set and a highly available solution through Redis Sentinel, as well as supporting data automatic fragmentation capabilities through Redis cluster.

The main functions of Redis are based on single-threaded model implementations, which means that Redis uses a thread to service all client requests, while Redis uses non-blocking io and finely optimizes the algorithmic time complexity of various commands, which means: Redis is thread-safe (because there is only one thread), all of its operations are atomic and are not very fast due to concurrent generation of data exception Redis (because of the use of non-blocking io, and most of the algorithm time complexity of the command is O (1)) It is dangerous to use the high time-consuming Redis command. Consumes a large amount of processing time for a unique thread, causing all requests to be slowed down. (for example, the keys command of O (N) with time complexity, strictly prohibited for use in production environments) 

REDIS data structures and related common commands

This section describes the main data structures supported by Redis, and the associated common Redis commands. This section only provides a brief introduction to the Redis command and lists only the more commonly used commands. If you want to know the complete set of REDIS commands, or learn more about how to use a command, refer to the official documentation: Https://redis.io/commands 

Key

Redis uses the Key-value basic data structure, any binary sequence can be used as a Redis key (such as a normal string or a JPEG image)
Some things to note about key: Do not use too long key. For example, using a 1024-byte key is not a good idea, not only consumes more memory, but also causes the search efficiency to reduce the key short to lack of readability is also bad, such as "U1000FLW" than "User:1000:followers", Save a few storage space, but caused the readability and maintainability of the trouble is best to use a unified specification to design key, such as "object-type:id:attr", with this specification designed by the key may be "user:1000" or "comment:1234: Reply-to "Redis allows the maximum key length to be 512MB (the length limit for value is also 512MB) 

String

String is the underlying data type for Redis, Redis has no concept of data types such as int, Float, Boolean, and all basic types are represented by string in Redis.

Common commands related to string: set: Sets value for a key, can be used with the EX/PX parameter to specify the validity period of the key, through the NX/XX parameter for the existence of key difference operation, time complexity O (1) Get: Obtain a key corresponding to the value , Time complexity O (1) Getset: Set value for a key and return the original value of the key, Time complexity O (1) Mset: Set value for multiple keys, time complexity O (N) Msetnx: With Mset, If any of the specified keys already exist, no action is taken, time complexity O (n) mget: Gets the value of multiple key, time complexity O (n)

As mentioned above, the basic data type of Redis is only string, but Redis can use string as an integral or floating-point number, mainly in the INCR, DECR class commands: INCR: The value of the key corresponding to 1, and return the value of the increment. Works only on string data that can be converted to an integral type. Time complexity O (1) Incrby: The value of the key corresponds to the specified integer value, and returns the value from the increment. Works only on string data that can be converted to an integral type. Time complexity O (1) Decr/decrby: With Incr/incrby, from the increase to self reduction.

The INCR/DECR Series command requires that the value type of the operation be string and can be converted to a 64-bit signed integer number, otherwise an error is returned.
In other words, the value of the INCR/DECR Series command must be in the range of [ -2^63 ~ 2^63-1].

As mentioned above, Redis uses a single-threaded model, which is naturally thread-safe, making the INCR/DECR command very convenient for precise control in high concurrency scenarios. 

Example 1: Inventory Control

In the high concurrency scene to achieve inventory margin of the accurate calibration, to ensure that the situation does not appear oversold.

Set up inventory Totals:

SET inv:remain "100"

Inventory deduction + Allowance Check:

DECR Inv:remain

When the DECR command returns a value greater than or equal to 0 o'clock, the inventory margin checksum is passed, and if a value less than 0 is returned, the inventory is depleted.

Assuming that there are 300 concurrent requests for inventory deduction, Redis can ensure that the 300 requests receive 99 to 200 of the return value, each request is given a unique return value, and will never find the same return value for two requests. 

Example 2: self-adding sequence generation

Implement a sequence function similar to RDBMS, generating a series of unique serial numbers

Set start value for series:

SET sequence "10000"

Gets a sequence value:

INCR sequence

Use the return value directly as a sequence.

Get a batch (for example, 100) sequence values:

Incrby Sequence 100

Assuming the return value is N, then the value of [N-99 ~ N] is the available sequence value.

When more than one client requests the Redis sequence at the same time, Redis can ensure that the sequence values or sequence ranges obtained by each client are globally unique, and that there is no case where different clients get duplicate sequence values. 

List

The Redis list is a linked-form data structure, and you can use commands such as Lpush/rpush/lpop/rpop to perform inserts and popup elements at both ends of the list. Although the list also supports the ability to insert and read elements on a specific index, the time complexity is high (O (N)) and should be used with care.

Common commands related to lists: Lpush: Inserts 1 or more elements into the left side (that is, the head) of the specified list, returning the inserted list length. Time complexity O (N), N is the number of inserted elements Rpush: With Lpush, insert 1 or more elements into the right side of the specified list (that is, the tail) Lpop: Removes an element from the left side of the specified list (that is, the head) and returns the time Complexity O (1) Rpop: With Lpop, Removes 1 elements from the right side (that is, the tail) of the specified list and returns LPUSHX/RPUSHX: similar to Lpush/rpush, the difference is that if the key of the lpushx/rpushx operation does not exist, no action is made Llen: Returns the length of the specified list, Time complexity O (1) Lrange: Returns the specified range of elements in the specified list (two-terminal contains, that is, lrange key 0 10 returns 11 elements), Time complexity O (N). You should control the number of elements you get at a time, and a large range of list elements at a time can cause delays, and avoid complete traversal operations such as Lrange key 0-1 for an unpredictable list of length.

List-related commands that should be used with caution: lindex: Returns the element on the specified index specified in the list, and returns nil if the index is out of bounds. The index value is a loop, that is, 1 represents the last position of the list, and 2 represents the second position of the list. Time complexity O (n) LSET: The specified list specifies the element on index to value, and if the index crosses the line, returns an error, time complexity O (n), and if the action is a head/tail element, the time complexity is O (1) Linsert: Inserts a new element before/after the specified element in the specified list and returns the length of the list after the operation. Returns-1 if the specified element does not exist. If the specified key does not exist, no action is made, Time complexity O (N)

Because the list of Redis is a linked list structure, the above three command algorithm efficiency is low, need to traverse the list, the command time is not predictable, in the case of large list length will increase significantly, should be used with caution.

In other words, Redis's list is actually designed to implement queues, rather than to implement a list like ArrayList. If you are not trying to implement a two-terminal queue, try not to use the Redis list data structure.

In order to better support the characteristics of queues, Redis also provides a series of blocking operation commands, such as Blpop/brpop, to implement a blockingqueue-like capability that blocks the connection when the list is empty, until an object in the list can be returned. For blocking class commands, please refer to the "Blocking Operations on Lists" section in the official documentation (HTTPS://REDIS.IO/TOPICS/DATA-TYPES-INTRO) for details. 

Hash

Hash is a hash table, Redis hash and the traditional hash table, is a field-value type of data structure, can be understood as the HashMap moved into the Redis.
Hash is great for data that represents object types, with field in the hash corresponding to the field of the object.
The advantages of hashing include: You can find two-dollar lookup, such as "find the age of the user ID 1000" than the whole object after serialization as a string storage method, hash can effectively reduce the consumption of network transport when using hash to maintain a collection, Provides a much more efficient random access command than list

Hash-related common commands: Hset: Sets the field in the hash for key to value. If the hash does not exist, one is created automatically. Time complexity O (1) Hget: Returns the value of the field field in the specified hash, time complexity O (1) Hmset/hmget: With Hset and hget, you can bulk operate multiple field under the same key, time complexity: O (N), n is the field number of an operation Hsetnx: Same as Hset, but if field already exists, Hsetnx does nothing, Time complexity O (1) hexists: Determines whether field in the specified hash exists, returns 1, does not exist return 0, time complexity O ( 1) Hdel: Deletes the field (one or more) in the specified hash, time complexity: O (n), n is the field number of the Operation Hincrby: with the Incrby command, Incrby a field in the specified hash, time complexity O (1)

The hash related command should be used with caution: Hgetall: Returns all Field-value pairs in the specified hash. Returns the result as an array in which field and value appear alternately. Time complexity O (n) hkeys/hvals: Returns all field/value in specified hash, time complexity O (n)

The above three commands will be full traversal of the hash, the number of field in the hash and the length of the order of linear correlation, for the size of the unpredictable hash, you should strictly avoid using the above three commands, but instead use the Hscan command for cursor-type traversal, see https:// Redis.io/commands/scan

  Set

Redis set is a unordered, not repeatable string collection.

Common commands related to set: Sadd: Adds 1 or more member to the specified set, and automatically creates one if the specified set does not exist. Time complexity O (n), n for added member number Srem: Removes 1 or more member from specified set, time complexity O (n), and N is the number of member removed Srandmember: Randomly returns 1 or more member from the specified set. Time complexity O (n), n is the number of member returned SPOP: Randomly removes from the specified set and returns count member, time complexity O (n), n is the number of member removed SCard: Returns the number of member in the specified set, time complexity O (1 Sismember: Determines whether the specified value exists in the specified set, and the Time Complexity O (1) Smove: Moves the specified member from one set to another set

Caution Set related commands: Smembers: Returns all member in the specified hash, time complexity O (n) Sunion/sunionstore: Computes the set of multiple sets and returns/stores to another set, time complexity O (n), N is the total number of Sinter/sinterstore for all collections participating in the calculation: computes the intersection of multiple sets and returns/stores them to another set, the time complexity O (n), and N is the total member number of all the collections that participate in the calculation sdiff/ Sdiffstore: Calculates the difference between 1 set and 1 or more sets and returns/stores it to another set, time complexity O (n), n is the total member of all the collections that participate in the calculation

These commands involve a large amount of computation and should be used with caution, especially when the set dimensions that are involved in the calculation are not known, and should be strictly avoided. Consider using the Sscan command traversal to get the entire member of the related set (see Https://redis.io/commands/scan) If you need to do a set/intersect/set calculation, which can be done on the client, Or on a slave that does not serve a real-time query request. 

Sorted Set

Redis Sorted Set is an ordered, not repeatable string collection. Each element in the Sorted set needs to be assigned a fraction (score), and Sorted set will sort the elements in ascending order according to score. If more than one member has the same score, the dictionary order is sorted in ascending order.

Sorted set is ideal for ranking.

Sorted Set's primary command: Zadd: Add 1 or more member to the specified Sorted set, time complexity O (Mlog (N)), M is the added member number, N is the number of member in Sorted set Zrem: Deletes 1 or more member from the specified sorted set, time complexity O (Mlog (N)), M is the number of member deleted, N is the number of member in sorted set Zcount: Returns the specified sorted Specifies the number of member in the score range in set, time complexity: O (log (N)) Zcard: Returns the member number in the specified sorted set, time complexity O (1) Zscore: Returns the specified sorted Score of the specified member in set, Time complexity O (1) Zrank/zrevrank: Returns the rank of the specified member in sorted set, Zrank returns the rank sorted in ascending order, and Zrevrank returns the rank sorted in descending order. Time Complexity O (log (n)) Zincrby: With Incrby, the score of specified member in the specified sorted set is increased, time complexity O (log (n))

Caution Sorted set related commands: Zrange/zrevrange: Returns all Member,zrange in the specified ranking range in the specified sorted set to sort by score ascending, Zrevrange sorted by score, time complexity O (log (N) +m), M is the number of the member returned for this time Zrangebyscore/zrevrangebyscore: Returns all member in the specified score range in the designated sorted set, and returns the result in ascending/descending order, Min and Max can be specified as-inf and +inf, and the delegate returns all member. Time Complexity O (log (N) +m) Zremrangebyrank/zremrangebyscore: Removes all member in the sorted set in the specified rank range/specified score range. Time Complexity O (log (N) +m)

The above commands should try to avoid passing parameters such as [0-1] or [-inf +inf] to a one-time complete traversal of the sorted set, especially if the size of the set is unpredictable sorted. You can iterate through the cursor via the Zscan command (see Https://redis.io/commands/scan for details). Or the number of returned member (for Zrangebyscore and Zrevrangebyscore commands) is restricted by the limit parameter to enable a cursor-style traversal. Bitmap and Hyperloglog

Redis of these two data structures are not commonly used, in this article only briefly introduced, if you want to learn more about the two data structures and their related commands, please refer to the Official document Https://redis.io/topics/data-types-intro in the relevant chapters

Bitmap is not an actual data type in Redis, but rather a way to use string as a bitmap. It can be understood to convert a string to a bit array. Using bitmap to store simple data of true/false types is extremely space-saving.

Hyperloglogs is a data structure that is primarily used for quantitative statistics, which, like set, maintains a collection of duplicates, but Hyperloglogs does not maintain specific member content and maintains only the number of member. That is, hyperloglogs can only be used to compute the number of elements that are not duplicated in a collection, so it saves a lot of memory space than set. 

Other common commands EXISTS: Determines whether the specified key exists, returns 1 representing the presence, 0 does not exist, Time complexity O (1) DEL: Deletes the specified key and its corresponding value, time complexity O (n), n for the deleted key number expire/ Pexpire: Set the validity period for a key, in seconds or milliseconds, Time complexity O (1) Ttl/pttl: Returns the valid time for a key, in seconds or milliseconds, Time complexity O (1) Rename/renamenx: Rename the key to Newkey. When using rename, if Newkey already exists, its value is overwritten, and if Newkey already exists when using Renamenx, no action is taken, Time complexity O (1) Type: Returns the type of the specified key, string, list, set, Zset , Hash. Time complexity O (1) config get: Gets the current value of a Redis configuration item, you can use the * wildcard character, time complexity O (1) config set: Set a new value for a Redis configuration item, Time complexity O (1) config REWRITE: Allow Redis to reload the configuration 

data persisted in redis.conf

Redis provides the ability to automatically persist data to a hard disk on a regular basis, including the RDB and aof two scenarios, each with its own strengths and short boards, which can be combined to run simultaneously to ensure data stability. You must use data persistence.

The Redis data persistence mechanism can be closed. If you only use Redis as a caching service, all the data stored in Redis is not the main body of the data, but just a synchronous backup, you can turn off the Redis data persistence mechanism.
In general, however, it is still recommended that at least the RDB mode of data be persisted, since: The persistence of the RDB approach is almost without loss of the performance of the Redis itself, and the only thing the Redis main process needs to do when RDB is persisted is to fork a subprocess, all of which are completed by the child Redis can automatically revert to the data recorded in the previous RDB snapshot, regardless of the reason crash it. This eliminates the process of manually synchronizing data from other data sources, such as DB, and is faster than any other data recovery method now that the hard disk is so big, there's really no shortage of places 

RDB

With RDB persistence, Redis periodically saves the snapshot of the data to a RBD file and automatically loads the Rdb file at startup and restores the previously saved data. You can configure the Redis for snapshot retention in the configuration file:

save [seconds] [changes]

If [changes] data modification occurs within [seconds] seconds, a RDB snapshot is saved, for example

Save 60 100

Allows Redis to check data changes every 60 seconds, and if 100 or more data changes occur, RDB snapshots are saved.
Multiple save directives can be configured to allow Redis to perform multi-level snapshot-saving policies.
Redis The Rdb snapshot is turned on by default, and the default Rdb policy is as follows:

Save 900 1
Save
60 10000

You can also manually trigger Rdb snapshot saves by using the Bgsave command.

Advantages of RDB: minimal impact on performance. As mentioned earlier, Redis fork a subprocess when saving a Rdb snapshot, with little effect on the efficiency of Redis processing client requests. Each snapshot generates a complete data snapshot file, so it can be supplemented by other means of keeping snapshots of multiple point-in-time points (for example, backing up snapshots of 0 per day to other storage media) as a very reliable means of disaster recovery. Using RDB files for data recovery is much faster than using AOF.

RDB's disadvantage: Snapshots are generated on a regular basis, so a portion of the data is lost more or less when Redis crash. If the dataset is very large and the CPU is not strong enough (such as a single core CPU), Redis may consume a relatively long time (up to 1 seconds) in the fork process, affecting client requests during this period. 

aof

When using aof persistence, Redis records every write request in a log file. When the Redis restarts, all the write orders recorded in the AoF file are executed once, ensuring that the data is restored to the latest.

AoF is turned off by default and, if you want to turn on, configure the following:

AppendOnly Yes

AOF provides three fsync configurations, always/everysec/no, specified by configuration item [Appendfsync]: Appendfsync No: Do not fsync, the timing of the flush file to the OS decision, the fastest Appendfsync always: A fsync operation is performed every time a log is written, with the highest data security, but the slowest appendfsync everysec: A compromise, which is referred to the background thread Fsync once per second

As aof constantly record write operations logs, there will be some useless logs, such as the execution of the command set Key1 "ABC" at some point in time, followed by a set Key1 "BCD" at some point, so the first command is obviously useless. A lot of useless logs can make aof files too large, and it will take too long to recover data.
So Redis provides a aof rewrite feature that can rewrite aof files, leaving only the minimum set of writes that can restore data to the latest state.
AoF rewrite can be triggered by the bgrewriteaof command, or you can configure Redis to automatically perform periodically:

Auto-aof-rewrite-percentage
auto-aof-rewrite-min-size 64MB

The implication of the above two-line configuration is that Redis records the AOF log size after rewrite when the rewrite is aof, and automatically aof aof when the rewrite log size is increased by 100% on that basis. At the same time, if the size of the increase does not reach 64MB, it will not be rewrite.

Advantages of AOF: Most secure, when Appendfsync always is enabled, any written data is not lost and only 1 seconds of data can be lost with the Appendfsync everysec enabled. AOF files are not corrupted in the event of a power outage, and can be easily repaired using the redis-check-aof tool, even if a log is written in half. AOF file easy to read, can be modified, after some errors in the data cleanup operations, as long as the aof file does not rewrite, you can back up aof files, the wrong command to delete, and then restore data.

AoF disadvantage: aof files are usually more performance-consuming than RDB files RDB high data recovery rate than RDB slow memory management and data elimination mechanism maximum memory settings

By default, in 32-bit OS, Redis uses up to 3GB of memory, with no restrictions on the 64-bit OS.

When using Redis, you should have a basic accurate estimate of the maximum space occupied by the data and set the maximum memory for Redis. Otherwise, Redis will consume unlimited memory in 64-bit OS (use swap space when the physical memory is full), which can easily cause a variety of problems.

Control the maximum memory used by Redis through the following configuration: