Looking at the queue technology in Redis cluster mode with publish/subscribe function (i)

About Redis

Redis is fully open source free and is a high-performance Key-value database.

Redis and other Key-value cache products have the following three features:

Redis supports data persistence, which saves data in memory on disk and can be loaded again for use when restarting.

Redis not only supports simple key-value types of data, but also provides storage of data structures such as List,set,zset,hash.

Redis supports backup of data, that is, Master-slave mode of data backup.

The performance is very high –redis can read the speed is 110,000 times/s, the write speed is 81,000 times/s.

Installation of Redis

wget http://download.redis.io/releases/redis-3.2.4.tar.gz

TAR–ZXVF redis-stable.tar.gz Decompression

Make compilation

Problems may occur make[3]: Gcc:command not found, GCC compilation tool not installed

Once the compilation is successful, go to the SRC folder and perform the make install for Redis installation

Make install-to/usr/local/bin directory: Redis-server,redis-cli,redis-check-aof,redis-check-dump

REDIS-SERVER–V Check if the installation is successful

Start Redis, go to the redis.conf directory, execute

Redis-server redis.conf. If not followed by redis.conf, start with the default configuration

Redis configuration file

REDIS.CONF configuration items are described below:

1. Redis is not running as a daemon by default and can be modified by this configuration item, enabling the daemon with Yes

daemonize Yes

2. When Redis is running as a daemon, Redis writes the PID to the/var/run/redis.pid file by default and can be specified by pidfile

Pidfile/var/run/redis.pid

3. Specify the Redis listening port, the default port is 6379, the author in his own article blog post explains why 6379 as the default port, because 6379 Merz the corresponding number on the phone keypad, and Merz from Italy showgirl Alessia Merz name

Port 6379

4. The host address of the binding

bind 127.0.0.1 this Ip to be set up as your server's Ip

5. If the client closes the connection after a long period of inactivity, if it is specified as 0, the function is turned off

Timeout

6. Specify logging level, Redis supports four levels in total: Debug, verbose, notice, warning, default is verbose

loglevel verbose

7. Logging mode, default to standard output, if Redis is configured to run as daemon, and this is configured as logging mode as standard output, the log will be sent to/dev/null

logfile stdout

8. Set the number of databases, the default database is 0, you can use the Select <dbid> command to specify the database ID on the connection

databases

9. Specify how many times the update operation will synchronize the data to the data file and can be combined with multiple conditions

Save <seconds> <changes>

There are three conditions available in the Redis default configuration file:

Save 1

Save

Save 10000

Represents 1 changes in 900 seconds (15 minutes), 5 changes in 300 seconds (10 minutes), and 60 changes in 10,000 seconds.

10. Specify whether to compress the data when storing to the local database, the default is Yes,redis with LZF compression, if you want to save CPU time, you can turn off this option, but it will cause the database file to become huge

rdbcompression Yes

11. Specify the local database file name, the default value is Dump.rdb

dbfilename Dump.rdb

12. Specify the local database to hold the directory

dir./

13. Set the IP address and port of the master service when this machine is a Slav service, and it will automatically synchronize data from master when Redis boots

slaveof <masterip> <masterport>

14. When the master service is password protected, the Slav service connects the password of master

Masterauth <master-password>

15. Set up the Redis connection password, if the connection password is configured, the client will need to provide the password via auth <password> command when connecting to Redis, turn off by default

Requirepass foobared

16. Set the maximum number of client connections at the same time, the default is unlimited, Redis can open the number of client connections for the Redis process can open the maximum number of file descriptors, if set maxclients 0, indicating no restrictions. When the number of client connections reaches the limit, Redis closes the new connection and returns the max number of clients reached error message to the client

maxclients

17. Specify the maximum Redis memory limit, Redis will load data into memory when it is started, Redis will try to clear expired or expiring key first, when this method processing, still reach the maximum memory setting, will no longer write operation, but still can read operation. Redis new VM mechanism will store key memory, value will be stored in swap area

maxmemory <bytes>

18. Specifies whether logging occurs after each update operation, which, by default, writes data to disk asynchronously and, if not turned on, may result in data loss over a period of time when power is lost. Because the Redis itself synchronizes data files in sync with the save conditions above, some data will only exist in memory for a period of time. Default is No

appendonly No

19. Specify the update log file name, default to Appendonly.aof

appendfilename appendonly.aof

20. Specify the update log condition, a total of 3 optional values:
No: Indicates that the data cache of the operating system is synchronized to disk (fast)
always: Represents a manual call to Fsync () to write data to disk (slow, secure) after each update operation
everysec: Indicates synchronization once per second (trade-offs, default values)

appendfsynceverysec

21. Specify whether to enable the virtual memory mechanism, the default value is no, a simple introduction, the VM mechanism to store data paging, by Redis will be less accessible pages, such as cold data swap to disk, Access to multiple pages is automatically swapped out into memory by the disk (in a later article I will carefully analyze the Redis VM mechanism)

Vm-enabledno

22. Virtual memory file path, default value is/tmp/redis.swap, cannot be shared by multiple Redis instances

Vm-swap-file/tmp/redis.swap

23. Store all data greater than vm-max-memory in virtual memory, regardless of the vm-max-memory settings, all index data is memory stored (REDIS index data is keys), that is, when the vm-max-memory is set to 0, In fact, all value is present on disk. The default value is 0

vm-max-memory0

24.Redis swap file is divided into a lot of page, an object can be saved on more than one page, but a page can not be shared by multiple objects, Vm-page-size is based on the size of the stored data to set, the author suggests that if you store many small objects, The page size is best set to 32 or 64bytes, and if you store large objects, you can use a larger page, and if you are unsure, use the default values

vm-page-size32

25. Set the number of pages in the swap file, since the page table (a bitmap that indicates that the page is idle or used) is in memory and consumes 1byte of memory per 8 pages on disk.

vm-pages134217728

26. Set the number of threads to access swap files, preferably do not exceed the machine's core number, if set to 0, then all the swap file operation is serial, may cause a relatively long delay. The default value is 4

VM-MAX-THREADS4

27. Set when replying to the client, whether to merge the smaller package into one package send, the default is to turn on

Glueoutputbufyes

28. Specifies that a special hashing algorithm is used when more than a certain number or maximum element exceeds a critical value

Hash-max-zipmap-entries64

Hash-max-zipmap-value

29. Specify whether to activate the reset hash, which is on by default (described later in the introduction of the Redis hashing algorithm)

Activerehashingyes

30. Specify other profiles that can use the same configuration file across multiple Redis instances on the same host, while each instance has its own specific configuration file

include/path/to/local.conf

Redis data types

Redis supports five types of data: string (String), hash (hash), list, set (set), and Zset (sortedset: Ordered set).

String:

Get, set. Note: a key can store up to 512MB.

Hash:

Hmset, Hmget, Hkeys, hvals

List:

Lpush, Lrange

Set

Sadd, Smembers

Zset

Zadd Collection Name Serial Number Collection Elements

Zadd Myzset 1 ZLH

Zadd MySet 2 Tom 3 Jim

Zcard Myzset View the number of members of the Zset collection

Zrange Mzyset 0-1

Zrange Mzyset 0-1 Withscores

Zrankmzyset Jim Gets the subscript position of the Zset member

Zcountmyset 1 3

Zrem Myzset ZLH To delete a specified member or multiple members

Zscore Myzset ZLH

Zincrby Myzset 4 Tom Tom's score value plus 4

Zrangebyscore Myzset 1 5

Redis Publish Subscriptions

A Redis Publish Subscription (PUB/SUB) is a message communication pattern: the Sender (pub) sends a message and the Subscriber (sub) receives the message.

Subscribe DONGNAO Client Subscription message, Dongnao for the appropriate channel

Publish Dongnao message is published, and clients subscribing to the channel receive the message

Redis transactions

Redis transactions can execute multiple commands at once, with the following two important guarantees:

A transaction is a separate quarantine operation: All commands in a transaction are serialized and executed sequentially. The transaction is not interrupted by a command request sent by another client during execution.

A transaction is an atomic operation: the commands in a transaction are either all executed or none of them are executed.

A transaction goes through the following three stages from start to execution:

Begins a transaction.

command to queue.

Executes a transaction.

MULTI Start a thing

Set Dongnao Jack

Get Dongnao

Hmset map Key1 value1 Key2 value2

Hmget Map Key1

Exec executes things

Redis Data Backup and recovery

By default, the Redis server program automatically iterates through the database every once in a while, writing memory snapshots in a file called "Dump.rdb", a persistent mechanism called snapshot. With snapshot, if the server goes down and the Redis server program restarts, Redis automatically loads the DUMP.RDB and restores the database state to the last time it was snapshot.

To manually make a memory backup:

Save

Redis Security

Configget Requirepass Get the password

ConfigSet Requirepass 123

Configget Requirepass

Auth Password Password verification

Redis Persistence snapshotting Snapshot Persistence

Rdb

Snapshots are the default persistence mode. This is the way in which the in-memory data is written to the binary in a snapshot, the default file name is Dump.rdb. You can automatically make snapshot persistence by configuring settings. We can configure Redis to take snapshots automatically if more than M key is modified in n seconds, the following is the default snapshot save configuration:

Save 1 #900秒内如果超过1个key被修改 to initiate a snapshot save
Save #300秒内容如超过10个key被修改, the snapshot save is initiated
Save 60 10000

The client can also use the Save or Bgsave command to notify Redis to do a snapshot persistence, and each snapshot persistence is to write the memory data to disk one at a time, not the incremental synchronization of only dirty data. If the amount of data is large, and more write operations, it will inevitably cause a large number of disk IO operations, may seriously affect performance. In addition, because the snapshot is done at a certain interval, if Redis is accidentally down, all changes after the last snapshot will be lost.

append-only

AOF

Redis appends each received write command to the file via the Write function (default is appendonly.aof). When Redis restarts, the contents of the entire database are rebuilt in memory by re-executing the write commands saved in the file.

Appendonlyyes #启用aof持久化方式
# Appendfsync always #每次收到写命令就立即强制写入磁盘, slowest, but guaranteed full persistence, not recommended
Appendfsync everysec #每秒钟强制写入磁盘一次, a good compromise in performance and durability, recommended
# Appendfsync No #完全依赖os, best performance, no guarantee of durability

Redis Master-Slave

Redis master-slave replication process: When slave is configured, slave establishes a connection with master and sends the sync command. Whether it is the first connection or reconnection, master initiates a background process, saves the DB snapshot to a file, and the master master process starts collecting new write commands and caches. After the background process finishes writing the file, master sends the file to Slave,slave to save the file to the hard disk, then loads it into memory, then master forwards the cached command to the slave, and the subsequent master sends the received write command to slave. If master receives multiple slave simultaneous connection commands at the same time, master initiates only one process to write database mirroring and then sends it to all slave. Master synchronizes data with non-blocking and can receive read and write requests from users. However, on the slave side is blocking mode, slave is not able to respond to client queries when synchronizing master data.

You can disable data persistence in master, just comment out all the save configurations in the master configuration file and configure data persistence on slave only

Redis Cluster

There are 16,384 hash slots built into the Redis cluster, and when a key-value is required to be placed in a Redis cluster, Redis first calculates a result using the CRC16 algorithm for key and then the result to 16384 for the remainder so that each key corresponds to a number 0 -16383 hash slots, Redis maps the hash slots to different nodes roughly equal to the number of nodes. Each node of the cluster is responsible for a portion of the hash slot. This structure makes it easy to add or remove nodes, and either adding or modifying a node does not cause the cluster to be unusable. The advantage of using hash slots is that you can easily add or remove nodes. When you need to add nodes, you just need to move some of the other node's hash slots to the new node; When you need to remove a node, just move the hash slot on the removed node to the other node, and at this point, we don't have to stop all redis services when we add or remove nodes later. Hash Slots

CentOS-based clustering

1, the cluster must have at least 3 nodes, each node has a master-slave structure

Create a new three folder, put the configuration file CP in, then modify the configuration file port, release the support cluster configuration, start the node

cluster-enabled Yes

Cluster-config-file nodes.conf

Cluster-node-timeout 5000

2, install Ruby, must be more than 2.0, more than 2.0 comes with RubyGems Package Manager

Yuminstall Ruby

3, Gem command install Redis package, add REDIS-TRIB.RB call Redis interface Package

Geminstall Redis

4. Manage Redis clusters with REDIS-TRIB.RB cluster tools

./redis-trib.rb Create--replicas 1 120.77.22.187:7000120.77.22.187:8000 120.77.22.187:9000 120.77.22.187:7010 120.77.22.187:8010120.77.22.187:9010

--REPLICAS1 setting the number of nodes from

Enter cluster mode to operate Redis command,-C cluster mode

Redis-cli-c-h–p

Redis Cluster Add node

The main thing is to see the phenomenon, slots transfer, the corresponding data will change?

Adding nodes

./redis-trib.rb Add-node 127.0.0.1:7006 127.0.0.1:7000

View node Status

Cluster nodes

Move hash slots, hash slot reallocation

./redis-trib.rb Reshard 127.0.0.1:7000

Enter all to randomly transfer from all the master nodes to get enough XX hash slots

Redis Cluster Delete node

Turn the hash slots out and go to the other master node

./redis-trib.rb Reshard 127.0.0.1:7000

Then delete the node

./redis-trib.rb Del-node 120.77.22.187:902094ee1415a16076f4070c5dd4b94defc14618dbb8

Redis-sentinel High Availability

Redis-sentinel is a highly available (HA) solution that is officially recommended by Redis, and if Master is down, the Redis itself (including many of its clients) does not implement automatic primary and standby switching when using Redis for master-slave high-availability scenarios. The Redis-sentinel itself is also a standalone process that monitors multiple Master-slave clusters and discovers that the master can be switched on automatically after the outage.

Its main function has the following points

Periodically monitor whether Redis works as expected;

If a Redis node is found to be running, it can notify another process (such as its client);

Ability to switch automatically. When a master node is unavailable, it is possible to elect one of the master's multiple slave (if there are more than one slave) as the new master, The other slave node changes the address of the master that it follows to the new address of the slave that is promoted to master.

It is important to note that the configuration file is dynamically modified during Sentinel operation, for example, when a primary standby switch occurs, the master in the configuration file is modified to another slave. This allows Sentinel to restore the status of the Redis cluster that was previously monitored by this configuration when it restarts.

Sentinel Monitor MyMaster 127.0.0.1 6379 1

Sentinel Down-after-milliseconds MyMaster 60000

#sentinel Can-failover MyMaster Yes

Sentinel Failover-timeout MyMaster 180000

Sentinel Parallel-syncs MyMaster 1

When Sentinel is clustered, the solution to this problem becomes simple, requiring more than one sentinel to communicate with each other to confirm that a master is really dead, and this 2 represents that when 2 Sentinel in the cluster thinks Master is dead, To really think that the master is no longer available.

Down-after-milliseconds
Sentinel sends a heartbeat ping to master to confirm that Master is alive, and if Master does not respond to pong within a "certain timeframe" or if it responds to an error message, then this Sentinel will subjectively (unilaterally) Think this master is no longer available (subjectively down, also referred to as Sdown). And this down-after-milliseconds is used to specify this "time range", in milliseconds.

Can-failover

No indicates that the current Sentinel is an observer and only participates in voting not to implement failover

At least one of the global is Yes

Parallel-syncs

When the new master is generated, the number of slave "slaveof" to the new master and "SYNC"

Default is 1, we recommend keeping the default value

Client requests are terminated when Salve performs salveof and synchronization
Failover-timeout

Failover expiration time, when failover starts, no failover action is triggered during this time

Current Sentinel will consider this failoer to be a failure

Looking at the queue technology in Redis cluster mode with publish/subscribe function (i)