First, Redis Introduction
Redis is an open source, BSD-licensed, Advanced key-value (Key-value) cache and Storage (store) system. Because Redis keys include string,hash,list,set,sorted set,bitmap and Hyperloglog, they are often referred to as data structure servers .
You can run atomic operations on these types, such as appending strings, increasing the values in the hash, adding an element to the list, calculating the intersection of the set, the set and the difference set, or the element that gets the highest rank from the ordered collection.
To meet high performance, Redis uses a memory (in-memory) DataSet (DataSet). Depending on your usage scenario, you can either dump the dataset to disk at intervals or append each command to the log to persist.
Redis also supports master-slave asynchronous replication , very fast non-blocking initial synchronization , and automatic re-interconnection of local resynchronization when the network is disconnected . Features include:
- Transaction
- Persistence of
- Master-Slave asynchronous replication
- Very fast, non-blocking initial synchronization
- Auto-re-connect local resynchronization when network is disconnected
- Subscribe/Publish
- Lua Script
- Keys with TTL
- LRU Recycle key
- Automatic failover (failover)
- Wait a minute
You can use Redis in multiple languages.
Redis is written in the ANSI C language and runs on most POSIX systems, such as Linux, *bsd, OS X, without additional reliance. Redis is developed and fully tested under the Linux and OS X two operating systems, and we recommend Linux as a deployment environment. Redis can also run on Solaris-derived systems, such as Smartos, but support needs to be strengthened. There are no officially supported Windows builds, but Microsoft has developed and maintained a 64-bit version of Windows.
Second, the data type 2.1 key-value1, about the key:
API Document: Http://www.redis.cn/commands.html#generic
- Binary security
- Include spaces
- 512M Max
- Not too long, too long to consider MD5 or other ways to extract signatures
- Don't be too short to lose readability: for example, User:1000:follower is more appropriate.
- Naming style should be unified
- Support key expiration policy, accurate to milliseconds
- Support Key Space
2. About value
The following types of support are supported:
- A binary security (Binary-safe) string.
- List : A collection of string elements (element) sorted by Insertion Order (collection). is usually a linked list.
- Collection : A unique, unordered collection of string elements.
- ordered collections : similar to collections, but each string element is associated with a floating-point number called a fraction (score). Elements are always sorted by fractions, so you can retrieve a range of elements (for example, give me the first 10, or the last 10 elements).
- Hash : A map consisting of a field and its associated values. Both the field and the value are string types. This is very similar to hashing/hashing in Ruby or Python.
- bit Array (bitmap): Use a special command to treat a string as a bit array: You can set or clear a single bit value, count the number of bits that are 1, look for the first reset or bit, and so on.
- Overweight logarithm (Hyperloglog): This is a probabilistic data structure used to estimate the cardinality of a set (cardinality, also known as a potential, translator's note). Don't be afraid, it's easier than it looks, and it'll be revealed later.
2.2 Strings String
Http://www.redis.cn/commands.html#string
- Binary Security Value
- The numeric type supports atomic increase and decrease
- Support for specifying expiration at creation time
- ...
2.3 Listing List
Http://www.redis.cn/commands.html#list
- is a double-ended list, so it can be a stack , or it can be a queue
- Support for blocking operations
- Support Caps list: Capped
When to use the list:
Take an example to illustrate:
- For example, you can record the latest user-submitted updates, use the list as a stack
- 2 Popular Ruby libraries, Resqu and Sidekiq, use the Redis list as hooks for background work, thinking of the list as a middleware, where the generator adds items to it, and consumers consume and perform tasks.
Let's take a few examples to illustrate how the list structure automatically creates keys and deletes them.
Insert 3 elements: Automatically created MyList
1 2 3
Delete all, automatically delete the MyList
192.168.0.211:6379>Lpop mylist"3"192.168.0.211:6379>Lpop mylist"2"192.168.0.211:6379>Lpop mylist"1"192.168.0.211:6379>Lpop mylist (nil)192.168.0.211:6379>exists mylist (integer)0
2.4 Hashing Hash
Http://www.redis.cn/commands.html#hash
- A hash is a collection of field value pairs (fields-values pairs), which is considered a map.
- Hashes that have a few fields (a small amount of about 100) are stored in a way that consumes little storage space, so you can store millions of of objects in a small Redis instance.
- Each hash can store up to more than 4 billion field value pairs (Field-value pair).
As the most commonly used redis structure, there are generally 2 design ideas, multiple keys and a single key.
The approximate design of multiple keys:
Hmset User:19771 hmset User:100119711
The approximate design of a single key:
+ " {username:antirez,birthyear:1977,verified:1} " 1001"{username:jerry,birthday:1971,verified:2}"
2.5 unordered Set Set
Http://www.redis.cn/commands.html#set
- Unordered string
- Duplicate not allowed
- Supports set operations, intersection, set, and difference sets
2.6 Ordered set
Http://www.redis.cn/commands.html#sorted_set
About order:
- If A and B are elements that have different fractions, A.score > B.score, then a > B.
- If A and B are elements that have the same score, if the dictionary order A is greater than B, then a > B. A and B cannot be the same, because the sorted collection can have only unique elements. (2.8 new features)
Characteristics:
- Implemented by a double-ended data structure, including a skip table (Skiplist) and a Hashtable (Hashtable), The O (log (N)) operation is performed each time it is added.
- Ordered, not ordered when requested, the order is dependent on the data structure that represents the ordered collection.
Third, use Twitter to demonstrate the above structure
This is a simple example that contains only 4 tables: User, followers (following), fans (followers), Posts (updates)
1. User:
#1 . Use a string to produce a unique user_idINCR + #2 . Use the first design of the hash structure to store the username password hmset User: username Antirez password P1pp0
If you want to support Getuseridbyusername (String username)-like operations:
# Use hash structure to store username-> in a key way IDhset +
2. Fans and concerns:
Each user may have more than one fan, and a user----fan mapping can be stored using an ordered set , using time as a score
# user1000 Fans added one is: user1234, Time is ...
zadd followers:14012676181234
The same is true of attention:
# user1000 can also focus on each other 1234 zadd followering:14012676181234
3. Posts
Here is not too troublesome, the post is the only update, that is posts
The most suitable structure here is the list of lists, using the stack, put the latest on the top of the stack
The list is then sorted by Llen and Lrange, which does not show:
Posts:New is lpushed here.
Iv. persistence mechanism
The biggest difference between Redis and memcached is that Redis can do both caching and storage . Storage relies on the persistence mechanism of redis.
2 Kinds of support: Rdb and AOF.
4.1 rdb1, what is an RDB:
- An RDB is the use of snapshot (snapshotting) technology to save a dataset.
- This means that the RDB is a point-in-time data backup !
- by default , Redis uses the mechanism of the RDB to persist, a binary file named Dump.rdb .
- You can set the data set to save at least m changes within n seconds. Settings such as "
save 60 1000 ", can also be manually save or Bgsave command.
2, Working principle:
- Redis calls fork () to create child process C1.
- Using the copy-on-write mechanism, C1 begins writing a data set to a temporary RDB file.
- Replace the old file after the C1 is complete.
3. RDB Advantages:
- Point-in-time, suitable for backup
- Cannot be modified after writing, so it can be replicated for transmission to remote data centers as disaster-tolerant technology, such as S3
- RDB performs well because only fork () one sub-process is required
- An RDB is faster than aof based on a Redis instance restart
4. Rdb Disadvantages:
- Possibility of loss of gap data based on point-in-time, data gaps
- If the fork () sub-process is too frequent, if the data set is large and CPU performance is not strong, an accident will occur, or even stop the service.
4.2 Aof1, what is aof:
- Append only file, that is, append files only, sequential IO
- Append the data set command that modifies Redis
- AOF support for each command append, append per s, and do not actively append dependent on the operating system three ways
- 1 Fsync operation per second is recommended
2, AoF principle:
- The Redis process, assuming the P process, called Fork, created a child process C
- C Start writing aof files to a temporary file
- P in memory buffers accumulate new changes during this time (while new changes are written to the old aof file to ensure security)
- When C is finished, send a signal to P and p to add the contents of the buffer to the end of the aof file.
- Redis automatically renames the AoF file, replacing the old for the new
3, AOF Advantages:
- More secure, if you use a policy per second, you lose up to 1s of data
- Additional files, easy to fix, secure, Redis and special repair tools
- AoF content easy to understand
- The aof file supports automatic rewriting to control file size
4, AoF Disadvantages:
- As a fallback strategy, aof files are usually much larger than the RDB
- AoF safe, but slow
- AOF a rare bug for special commands
4.3 How to choose
- If you use 2 at the same time, you can reach the level of data security provided by PostgreSQL!
- If you are not concerned at all, you can turn off both aof and RDB, simply as a cache instead of storage!
- If you care about the data, you need to store it, but you can receive a few minutes of data loss, you can use only rdb!
- If you want to use aof alone, discouraged, because the data must be backed up, the backup requires an RDB, and avoids the aof bug, preferring 2 to use!
4.4 How to back up
Redis is very friendly with data backup because you can copy an RDB file while the database is running: The Rdb file is not modified once it is generated, the file is generated into a temporary file, and when the new snapshot is completed, the rename (2) Atomic modified file name is automatically used as the target file.
This means that it is completely safe to copy an RDB file while the server is running. Here are our suggestions:
- Create a scheduled task (cron job), create an RDB snapshot every one hours to a directory, and a daily snapshot in another directory.
- Be sure to use the Find command to delete the old snapshot every time the script runs: For example, you can save a snapshot of every hour in the last 48 hours, one two months per day. Note Add datetime information when you name the snapshot.
- Transfer your RDB snapshot to your data center at least once a day, or at least to the physical machine running your Redis instance.
4.5 How to repair aof files
It is possible that the server crashes (crash) when writing the AOF file, and Redis cannot load after the file is corrupted. If this happens, you can use the following steps to solve the problem:
- Create a copy of the AOF for backup.
- Use the Redis redis-check-aof tool to repair the original file:
- $ redis-check-aof--fix
- Use Diff-u to check what's different with two of files. Restart the server with the repaired file.
V. Replication 1, features of Redis replication:
An active (master-slave) replication that uses and configures very simple, allowing Redis to cost an exact copy of the master server from the server.
- Asynchronous replication : Starting at 2.8, the amount of data in the data processed from the replication stream is periodically reported from the server. A primary server can have multiple slave servers.
- support complex structure , master->s1->s2 ....
- Master is non-blocking when the primary server is replicating.
- Redis replication is also non-blocking on slave.
- Slave is also able to continue to process requests using the old version of the dataset by configuring the redis.conf in the slave. You can configure an error to be returned from the server to the client when the replication stream is down. Then, after the initialization synchronization is complete, the old data set needs to be deleted and the new dataset needs to be loaded. during this short window period, the incoming connection is blocked from the server .
- replication can be used to support scalability , to handle read- only queries with multiple slave (for example, a heavy sort operation can be assigned to a slave server), or to be used only as data redundancy .
- You can use replication to prevent the primary server from spending all of the data set disks : You only need to configure the redis.conf of your home server to prevent saving (all "save" instructions) and then connect to a constantly replicating slave server. However, this setting ensures that the primary server does not restart automatically.
2, use Redis master-slave to pay attention to security: that data is not lost
(1). We set up node A as the primary server, turn off persistence, and node B and node C replicate from Node A.
(2). A crashes, but it has an automatic restart system that restarts the process. However, because persistence is turned off, the node restarts with an empty data set.
(3). Node B and node C are copied from empty a, so they completely destroy their copy of the data.
When the sentry is turned on, it is advisable to turn off the automatic restart of the primary server to prevent the sentry from detecting the automatic restart of master.
3, the principle of Redis replication
- When slave connects to master, the connection sends a sync command, whether it is the first time or reconnecting.
- The primary server starts saving in the background and starts buffering all newly received commands that modify the dataset.
- After the primary server is saved, a copy of the database file is sent to the slave server, saved from the server to disk, and loaded into memory.
- The primary server sends a buffered command to the slave server.
-is done through a command flow, in the same format as the Redis protocol.
-You can send the sync command to a working Redis port with telnet , and you'll see a lot of transmissions.
-When the master-slave link is disconnected, it can be automatically re-connected from the server. If the primary server receives multiple concurrent synchronization requests, only one background is executed to hold all servers in the service.
-When re-connected, always perform a full-volume synchronization, after 2.8, you can choose to perform a partial synchronization. (partial resynchronization)
4, support partial resynchronization (partial resynchronization)
That is, after the replication is broken, and then re-connected, you can continue the copy process without requiring a full resynchronization.
By creating a memory buffer (in-memory backlog) Implementation of a replication stream on the primary server. The primary server and all the slave servers record a copy offset (offset) and a primary server ID (run ID), which is re-linked from the server when the link is broken, and the requesting server resumes replication.
If the following conditions are true:
- The primary server has the same run ID
- Offsets specified offset are available in the replication buffer
You can proceed with incremental replication or perform a full resynchronization.
5, diskless Copy, test: (diskless replication)
Full full-volume synchronization requires that an RDB file be created on disk and then loaded from disk with the same RDB to the slave server, in general.
Therefore, after the 2.8.18 release, the experimental support for diskless replication.
The principle is that a child process sends an RDB to the slave server via a line (write) and does not need to use disk as intermediate storage.
6, the default slave is read-only:
After the 2.6 release, by default, read-only mode is turned on from the server.
This behavior is controlled by the slave-read-only in Conf.
It is also supported to modify the runtime through config set, and it is meaningful to close read-only in some failover policies.
For example, consider using a traditional high-availability model to build a schema, such as using keep-alived to maintain high availability, which requires slave to support write mode.
7, Support password Authentication 8, support n copies to write the mechanism:
A health monitoring mechanism is configured on master that allows write operations when there is a successful detection of n healthy slave.
Set on the Redis master server:
Min-slaves-to-write <number of Slaves>
Min-slaves-max-lag <number of Seconds>
(1) The Redis Primary server pings the master server per second, escalating the amount of data that has been processed for the replication stream.
(2) Redis master server records the last time a ping was received from each server, health monitoring
(3) The user configures the minimum number of slave servers, each with a lag lag of less than the maximum number of seconds.
Redis (1)
