Redis master-slave cluster construction and disaster recovery deployment (sentinel)

Redis master-slave cluster construction and disaster recovery deployment (sentinel)

Redis has also been used for a while. Record the cluster construction and configuration details for later use.

Outline

• Install Redis
• Overall Architecture
• Redis master-slave structure construction
• Redis disaster recovery deployment (sentinel)
• Redis FAQs

Install Redis

Released: CentOS-6.6 64bit

Kernel: 2.6.32-504. el6.x86 _ 64

CPU: intel-i7 3.6G

Memory: 2 GB

Download redis and select an appropriate version

[Root @ rocket software] # wget http://download.redis.io/releases/redis-2.8.17.tar.gz

[Root @ rocket software] # cd redis-2.8.17

[Root @ rocket redis-2.8.17] # make

[Root @ rocket redis-2.8.17] # make test

Cd src & make test

Make [1]: Entering directory '/home/software/redis-2.8.17/src'

You need tcl 8.5 or newer in order to run the Redis test

Make [1]: *** [test] Error 1

Make [1]: Leaving directory '/home/software/redis-2.8.17/src'

Make: *** [test] Error 2

Make test reports an error. Install tcl

[Root @ rocket software] # wget http://prdownloads.sourceforge.net/tcl/tcl8.5.18-src.tar.gz

[Root @ rocket software] # tar-zxvf tcl8.5.18-src.tar.gz

[Root @ rocket software] # cd tcl8.5.18

[Root @ rocket tcl8.5.18] # cd unix/

[Root @ rocket unix] #./configure; make test; make install

Tcl is successfully installed. Continue to test the redis installation.

[Root @ rocket redis-2.8.17] # make test

......

Cleanup: may take some time... OK

Make [1]: Leaving directory '/home/software/redis-2.8.17/src'

Redis is installed normally.

Overall architecture

Here is the overall architecture of the cluster set up in this article, using the master-slave structure + sentinel for Disaster Tolerance.

Directory structure Redis master-slave structure construction redis master copy executable files

[Root @ rocket master] # pwd

/Usr/local/redisDB/master

[Root @ rocket master] # cp/home/software/redis-2.8.17/src/redis-cli.

[Root @ rocket master] # cp/home/software/redis-2.8.17/src/redis-server.

Configuration File redis. conf

# Daemonize yes in daemonize mode # pid filepidfile/var/run/redis. pid # listening port 7003 # TCP receiving queue length, affected by kernel parameters/proc/sys/net/core/somaxconn and tcp_max_syn_backlog, tcp-backlog 511 # How many seconds a client is idle before closing the connection (0 indicates disabling and never disabling) timeout 0 # if it is not zero, set the SO_KEEPALIVE option to send ACKtcp-keepalive 60 # specify the server debugging level # possible value: # debug (a large amount of information, useful for Development/Testing) # verbose (a lot of concise and useful information, but not as much as the debug level) # notice (a proper amount of information is basically required in your production environment) # warning (only important/serious information is recorded) loglevel notice # Specify the log file name logfile ". /redis7003.log "# set the number of databases 16 # after the specified number of seconds and the number of data changes, the database will be written to the disk # After 900 seconds (15 minutes, and at least one change #300 seconds (5 minutes), and at least 10 Changes #60 seconds later, save 10000 1 save 900 10 save 60 300 # by default, if RDB snapshot is enabled (at least one save command) and the latest background storage fails, redis will stop accepting write operations # This will make the user know that the data is not correctly persisted to the hard disk, otherwise, no one may notice and cause some disasters. stop-writes-on-bgsave-error yes # when exported. whether to use LZF to compress the string object rdbcompression yes # version 5 rdb has a CRC64 algorithm checksum placed at the end of the file. This makes the file format more reliable. Rdbchecksum yes # Name of the persistent database file dbfilename dump. rdb # working directory dir. /# When password protection is set for the master service, the slav service connects the master password masterauth 0234kz9 * l # When a slave loses its connection to the master or the synchronization is in progress, there are two types of slave behavior: #1) If slave-serve-stale-data is set to "yes" (default), slave will continue to respond to client requests, # It may be normal data, outdated data, or empty data that has not yet been obtained. #2) If slave-serve-stale-data is set to "no", slave will reply "synchronizing from master # (SYNC with master in progress)" to process various requests, besides the INFO and SLAVEOF commands. Slave-serve-stale-data yes # You can configure whether the salve instance accepts write operations. Writable slave instances may be useful for storing temporary data (because the data written to salve # will be easily deleted after synchronization with the master node. slave-read-only yes # whether to send SYNC on the slave socket disable TCP_NODELAY? # If you choose "yes" Redis, you will use less TCP packets and bandwidth to send data to slaves. However, this will delay data transmission to slave, the default configuration of the Linux kernel will reach 40 ms # If you select "no" for data transmission to salve, the latency will be reduced, but you need to use more bandwidth repl-disable-tcp-nodelay no # the priority of slave is an integer displayed in the Info output of Redis. If the master node no longer works normally, the Sentinel will use it to # select a Server Load balancer instance to be upgraded to the master node. # Salve with a smaller priority number will give priority to master, so for example, three slave priorities are 10,100, 25, and # The Sentinel will select the slave with a minimum priority number of 10. #0 is a special priority, and the slave cannot be used as the master. Therefore, a server Load balancer with a priority of 0 will never be selected by # The Sentinel and upgraded to masterslave-priority 100 # password verification # warning: because Redis is too fast, people outside the company can try a K password every second to crack the password. This means you need a high-strength password. Otherwise, it will be too easy to crack. requirepass 0234kz9 * l # The maximum memory occupied by the redis instance. Do not use more memory than the configured upper limit. Once the memory usage reaches the upper limit, Redis will delete the keymaxmemory 3 gb Based on the selected recycle policy (see # maxmemmory-policy) # maximum Memory policy: if the memory limit is reached, how does Redis choose to delete a key. You can choose # volatile-lru-> delete a key with an expiration time based on the LRU algorithm in the following five actions. # Allkeys-lru-> delete any key based on the LRU algorithm. # Volatile-random-> the key is randomly deleted based on the expiration settings and has an expiration time. # Allkeys-> random deletion without difference, any key. # Volatile-ttl-> Delete (supplemented by TTL) based on the latest expiration time. This is the key with an expiration time # noeviction-> no one deletes the key, an error is returned directly during the write operation. Maxmemory-policy volatile-lru # by default, Redis asynchronously exports data to the disk. This mode is good enough in many applications, but Redis process # a problem or power failure may cause a loss of write operations for a period of time (depending on the configured save command ). # AOF is a more reliable alternative to persistence mode. For example, you can use the default data writing policy (see the configuration below) # Redis # can only lose 1 second of write operations in case of Redis's own process problems, such as server power failure or single write failure, but the operating system is still running normally. # AOF and RDB persistence can be started at the same time without any problems. # If AOF is enabled, Redis will load the AOF file at startup, which guarantees data reliability. Appendonly no # aof file name appendfilename "appendonly. aof" # fsync () System Call tells the operating system to write data to the disk, instead of waiting for more data to enter the output buffer. # Some operating systems will actually fl data to the disk immediately; some will try to do so as soon as possible. # Apsaradb for Redis supports three different modes: # no: Do not fl immediately. You can only fl it when the operating system needs to fl it. Fast. # Always: Every write operation is immediately written to the aof file. Slow, but safest. # Everysec: Write once per second. Compromise. Appendfsync everysec # If the AOF synchronization policy is set to "always" or "everysec", and the background storage process (the background storage or write AOF # logs) it will produce a lot of disk I/O overhead. Some Linux configurations may cause Redis to be blocked for a long time due to fsync () system calls. # Note: the current situation has not been perfectly corrected, and even fsync () of different threads will block the synchronous write (2) Call. # To alleviate this problem, use the following option. It can prevent the main process from performing fsync () during BGSAVE or BGREWRITEAOF processing (). # This means that if a sub-process is performing the save operation, Redis will be in the "non-synchronous" state. # This means that in the worst case, 30 seconds of log data may be lost. (Default Linux setting) # If you have latency issues, set this to "yes"; otherwise, keep "no", which is the safest way to save persistent data. No-appendfsync-on-rewrite yes # automatically override the AOF File auto-aof-rewrite-percentage 100auto-aof-rewrite-min-size 64 mb # The AOF file may be incomplete at the end (this is a problem with system shutdown), in particular, when mounting the ext4 file system, the # data = ordered option is not added. Only when the OS is dead, redis itself is not completely dead ). # There is a problem when redis is restarted and loaded into the memory. # When an error occurs, you can select redis to start an error and notify users and write logs, or load as much normal data as possible. # If aof-load-truncated is yes, a log is automatically published to the client and then loaded (default ). # If no, you must manually fix the aof file in redis-check-AOF. # Note: If the aof is damaged during the read process, the server will also quit, # This option is only used when the server tries to read more data but cannot find the corresponding data. Aof-load-truncated yes # maximum Lua script execution time, in milliseconds, lua-time-limit 5000 # Redis slow query logs can record the length of slowlog-log-slower-than 10000 # queries that have exceeded the specified time. It only consumes memory. You can use slowlog reset to recycle memory. Slowlog-max-len 128 # The redis latency monitoring system samples some operations during operation to collect the data source that may cause latency. # Use the LATENCY command to print some drawings and obtain some reports for easy monitoring # The system only records the operation whose execution time is greater than or equal to the specified time (milliseconds, # This reservation time is specified through latency-monitor-threshold configuration. # when it is set to 0, this monitoring system is in the stopped status latency-monitor-threshold 0 # Redis can notify the Pub/Sub client of events related to the key space, notify-keyspace-events "" is disabled by default. # when there are only a few entries in hash and the maximum entry space does not exceed the specified limit, it is encoded using a memory-saving # data structure. The following command can be used to set the limit hash-max-ziplist-entries 512hash-max-ziplist-value 64 # similar to the hash list with fewer data elements, you can use another encoding method to save a lot of space. # This special method is used only when the following restrictions are met. Use list-max-ziplist-entries 512list-max-ziplist-value 64 # set has a special encoding condition: when the set data is a string consisting of 64-bit signed integer digits in decimal format. # The following configuration item is used to set the maximum length of the set to use this encoding to save memory. Set-max-intset-entries 512 # similar to hash and list, ordered sets can also be encoded in a special way to save a lot of space. # This encoding method is only applicable to the zset-max-ziplist-entries 128zset-max-ziplist-value 64 # HyperLogLog sparse structure that is less than the limit below. This limit consists of #16 bytes of headers. When HyperLogLog uses sparse Structure Representation # these restrictions, it is converted to density representation. # It is useless if the value is greater than 16000, because at this point # intensive representation is more memory efficiency. # The recommended value is about 3000, so as to have memory benefits, reduce memory consumption hll-sparse-max-bytes 3000 # enable hash refresh, every 100 CPU milliseconds takes one millisecond to refresh Redis's master hash table (top-level key-value ing table) activerehashing yes # client output buffer limit, it can be used to force the client-output-buffer-limit normal 0 0 0client-output-buffer-limit slave 256 mb 64 mb 60client-output-buffer-limit pubsub 32 mb 8 mb 60 # By default, "hz" is set to 10. Increasing this value will allow Redis to use more CPUs when it is idle, but when multiple keys # expire at the same time, it will make Redis more responsive, and timeout can more accurately process hz 10 # When a sub-process overrides the AOF file, if the following options are enabled, every 32 MB of data generated by the file will be synchronized with aof-rewrite-incremental-fsync yes

Start master

[Root @ rocket master] #./redis-server./redis. conf

[Root @ rocket master] # ps axu | grep redis

Root 24000 0.1 0.7 137356 7440? Ssl./redis-server *: 7003

Test the connection using a client

[Root @ rocket master] #./redis-cli-a 0234kz9 * l-p 7003

127.0.0.1: 7003> select 1

OK

127.0.0.1: 7003 [1]> set name zhangsan

OK

127.0.0.1: 7003 [1]> get name

"Zhangsan"

127.0.0.1: 7003 [1]> quit

We can see that redis is successfully started and data can be read and written.

Set up redis slave

The slave configuration is basically the same as that of the master. You only need to modify the corresponding pidfile, port, log file name, and the master Address and authentication password.

Configuration File redis_slave.conf (difference from redis master)

# Pid file

Pidfile/var/run/redis_slave.pid

# Listening port

Port 8003

# Specify the log file name

Logfile "./redis8003.log"

# Set the IP address and port of the master service when the local machine is slav service. When Redis is started, it will automatically synchronize data from the master.

Slaveof 127.0.0.1 7003

# When password protection is set for the master service, the slav service connects to the master password

Masterauth 0234kz9 * l

Start slave and view Data Synchronization status

[Root @ rocket slave] #./redis-server./redis_slave.conf

[Root @ rocket slave] #./redis-cli-a 0234kz9 * l-p 8003

127.0.0.1: 8003> select 1

OK

127.0.0.1: 8003 [1]> get name

"Zhangsan"

We can see that the key-value set in the master has been successfully synchronized.

Role of Sentinel in Redis disaster recovery deployment

1. Monitoring: monitor whether the master and slave nodes are normal
2. Notification: when a problem occurs, you can notify the relevant personnel.
3. failover: Automatic master-slave switchover
4. Unified Configuration Management: the connector asks sentinel to obtain the Master/Slave address.

Raft Distributed Algorithm

1. Main Application: used for Distributed Systems, fault tolerance of systems, and selection of leaders
2. Author: Diego Ongaro, graduated from Harvard
3. Currently, projects that use this algorithm include:
A. CoreOS: See the following
B. ectd: a distributed, consistent shared configuration
C. LogCabin: Distributed Storage System
D. redis sentinel: redis Monitoring System

Core Raft algorithms used by Sentinel: Principles

1. All sentinel have the right to lead the election
2. Each sentinel requires other sentinel to elect itself as the leader (The sentinel, which finds redis to go offline objectively, initiates the election first)
3. Each sentinel has only one chance of election.
4. Adopt the first come, first served principle
5. Once added to the system, it will not be automatically cleared (this is important, why ?)
6. Each sentinel has a unique uid and will not be changed due to restart.
7. The condition for reaching the leader is that N/2 + 1 sentinel chose himself
8. Use the configuration epoch. If a split-brain occurs in an election, the configuration epoch increases progressively and enters the next election. All sentinel instances are in the unified configuration epoch, which is based on the latest version.

Raft algorithm core: Visual chart

Raft Visualization (algorithm demonstration)

Raft Distributed Algorithm Application

Coreos: Docker, a cloud computing star, is developing at a rocket speed, and the ecosystem related to it is getting better. CoreOS is one of them. CoreOS is A brand new Linux operating system designed for data centers. In July 2014, CoreOS released its first stable version, which has completed A round financing of $8 million.

Sentinel implements Redis Disaster Tolerance deployment three-guard Architecture

[Root @ rocket sentinel] # tree

.

── Redis-cli

── Redis-sentinel

── Redis-server

── Sentinel1

│ ── Sentinel1.conf

│ ── Sentinel1.log

── Sentinel2

│ ── Sentinel2.conf

│ ── Sentinel2.log

── Sentinel3

── Sentinel3.conf

── Sentinel3.log

Configure sentinel1.conf

# Example sentinel. conf # port <sentinel-port> port 26371 # daemonize in daemonize mode yes # specify the log file name logfile ". /sentinel1.log "# working path, sentinel generally specifies/tmp is relatively simple dir. /# When the Sentinel monitors this master, when at least the quorum sentinel instances think that the master is down, mark the master as odown # (objective down; corresponding sdown, subjective down, subjective down) status. # Slaves is automatically discovered, so you do not need to specify slaves explicitly. Sentinel monitor TestMaster 127.0.0.1 7003 1 # How long does the master or slave Take (30 seconds by default) fail to use and mark it as s_down. Sentinel down-after-milliseconds TestMaster 1500 # If sentinel fails to complete the failover operation within this configuration value (that is, master/slave automatic switch upon failure), the current failover fails. Sentinel failover-timeout TestMaster 10000 # Set the master and slaves verification password sentinel auth-pass TestMaster 0234kz9 * lsentinel config-epoch TestMaster 15 sentinel leader-epoch TestMaster 8394 # in addition to the current sentinel, what other sentinel known-sentinel TestMaster 127.0.0.1 26372 slave known-sentinel TestMaster 127.0.0.1 26373 slave current-epoch 8394

Configure sentinel2.conf in sentinel2.conf

# Example sentinel. conf # port <sentinel-port> port 26372 # daemonize in daemonize mode yes # specify the log file name logfile ". /sentinel2.log "# working path, sentinel generally specifies/tmp is relatively simple dir. /# When the Sentinel monitors this master, when at least the quorum sentinel instances think that the master is down, mark the master as odown # (objective down; corresponding sdown, subjective down, subjective down) status. # Slaves is automatically discovered, so you do not need to specify slaves explicitly. Sentinel monitor TestMaster 127.0.0.1 7003 1 # How long does the master or slave Take (30 seconds by default) fail to use and mark it as s_down. Sentinel down-after-milliseconds TestMaster 1500 # If sentinel fails to complete the failover operation within this configuration value (that is, master/slave automatic switch upon failure), the current failover fails. Sentinel failover-timeout TestMaster 10000 # Set the master and slaves verification password sentinel auth-pass TestMaster 0234kz9 * lsentinel config-epoch TestMaster 15 sentinel leader-epoch TestMaster 8394 # in addition to the current sentinel, what other sentinel known-sentinel TestMaster 127.0.0.1 26371 slave known-sentinel TestMaster 127.0.0.1 26373 slave current-epoch 8394

Configure sentinel3.conf on sentinel3.conf

# Example sentinel. conf # port <sentinel-port> port 26373 # daemonize in daemonize mode yes # specify the log file name logfile ". /sentinel3.log "# working path, sentinel generally specifies/tmp is relatively simple dir. /# When the Sentinel monitors this master, when at least the quorum sentinel instances think that the master is down, mark the master as odown # (objective down; corresponding sdown, subjective down, subjective down) status. # Slaves is automatically discovered, so you do not need to specify slaves explicitly. Sentinel monitor TestMaster 127.0.0.1 7003 1 # How long does the master or slave Take (30 seconds by default) fail to use and mark it as s_down. Sentinel down-after-milliseconds TestMaster 1500 # If sentinel fails to complete the failover operation within this configuration value (that is, master/slave automatic switch upon failure), the current failover fails. Sentinel failover-timeout TestMaster 10000 # Set the master and slaves verification password sentinel auth-pass TestMaster 0234kz9 * lsentinel config-epoch TestMaster 15 sentinel leader-epoch TestMaster 8394 # in addition to the current sentinel, what other sentinel known-sentinel TestMaster 127.0.0.1 26371 slave known-sentinel TestMaster 127.0.0.1 26372 slave current-epoch 8394

View the monitored master and slave in sentinel

[Root @ rocket sentinel] #./redis-cli-p 26371

127.0.0.1: 26371> SENTINEL masters

1) 1) "name"

2) "TestMaster"

3) "ip"

4) "127.0.0.1"

5) "port"

6) "7003"

7) "runid"

8) "de0896e3799706bda49cb92048776e233841e25d"

9) "flags"

10) "master"

127.0.0.1: 26371> SENTINEL slaves TestMaster

1) 1) "name"

2) "127.0.0.1: 8003"

3) "ip"

4) "127.0.0.1"

5) "port"

6) "8003"

7) "runid"

8) "9b2a75596c828d6d605cc8529e96edcf951de25d"

9) "flags"

10) "slave"

View the current master

127.0.0.1: 26371> SENTINEL get-master-addr-by-name TestMaster

1) "127.0.0.1"

2) "7003"

Stop the master and check the disaster recovery switchover.

[Root @ rocket master] # ps axu | grep redis

Root 24000 0.2 0.9 137356 9556? Ssl Jan12./redis-server *: 7003

Root 24240 0.2 0.7 137356 7504? Ssl Jan12./redis-server *: 8003

Root 24873 0.3 0.7 137356 7524? Ssl ../redis-sentinel *: 26371

Root 24971 0.3 0.7 137356 7524? Ssl ../redis-sentinel *: 26372

Root 24981 0.3 0.7 137356 7520? Ssl ../redis-sentinel *: 26373

Root 24995 0.0 0.5 19404 5080 pts/2 S +./redis-cli-p 26371

Root 25969 0.0 0.0 103252 844 pts/0 S + grep redis

[Root @ rocket master] # kill-QUIT 24000

Check the master and find that the master has been switched to the original slave.

127.0.0.1: 26371> SENTINEL get-master-addr-by-name TestMaster

1) "127.0.0.1"

2) "8003"

View sentinel logs

Start the original master and find that it has become a Server Load balancer instance.

[Root @ rocket master] #./redis-server./redis. conf

127.0.0.1: 26371> SENTINEL slaves TestMaster

1) 1) "name"

2) "127.0.0.1: 7003"

3) "ip"

4) "127.0.0.1"

5) "port"

6) "7003"

It is found that the master and slave have been reversed.

Sentinel automatic discovery

Each Sentinel subscribes to all the master servers monitored by it and the _ sentinel __: hello channel of the slave server, and finds the sentinel (looking for unknown sentinels) that has not previously appeared ). When a Sentinel discovers a new Sentinel, it adds the new Sentinel to a list, which stores all the other Sentinel that Sentinel knows and monitors the same master server.

Wagner. 0.0.1: 7003 [1]> SUBSCRIBE _ sentinel __: hello

Reading messages... (press Ctrl-C to quit)

1) "subscribe"

2) "_ sentinel __: hello"

3) (integer) 1

1) "message"

2) "_ sentinel __: hello"

3) "127.0.0.1, 26373, 7d919ccfb5752caf6812da2d0dba4ed0a528ceda, 8436, TestMaster, 127.0.0.1, 7003,8436"

1) "message"

2) "_ sentinel __: hello"

3) "127.0.0.1, 26372, 9eda79e93e6d1aa4541564ac28e3dc899d39e43b, 8436, TestMaster, 127.0.0.1, 7003,8436"

1) "message"

2) "_ sentinel __: hello"

3) "127.0.0.1, 26371, 8d63bebfbca9e1205a43bc13b52079de6015758e, 8436, TestMaster, 127.0.0.1, 7003,8436"

Redis FAQs

Maximum memory problem: You need to set the maximum memory to avoid the need to continuously apply for memory, causing system memory to be used up.

Fork process problems: the 'vm. overcommit_memory = 1' option should be added to the system configuration to prevent fork from failure due to insufficient memory.

Password problems: complicated settings are required to prevent brute force cracking.

You may also like the following articles about Redis. For details, refer:

Install and test Redis in Ubuntu 14.04

Basic configuration of Redis master-slave Replication

Redis cluster details

Install Redis in Ubuntu 12.10 (graphic explanation) + Jedis to connect to Redis

Redis series-installation, deployment, and maintenance

Install Redis in CentOS 6.3

Learning notes on Redis installation and deployment

Redis. conf

Redis details: click here
Redis: click here

This article permanently updates the link address: