REDIS3 cache cluster (cluster) setup

Redis3 later support cluster features: 1. Node auto-discovery; 2.slave->master election, cluster fault tolerance; 3. Hot resharding: Online sharding; 4. Inbound Management: Cluster xxx;5. Cluster management based on configuration (nodes-port.conf); 6. ASK steering/moved steering mechanism.

Technical knowledge point of cluster construction

• Redis Service port: The port used to receive client connections or client request commands, such as: 9579.
• Redis Cluster Communication port: The port used for Redis internal update AC status, which is the current server port +10000, such as 19579.
• The Slot:redis of Redis divides all keys into 16,284 parts, from 0 to 16283.
• Redis Server Relationship: Master-Slave, from the main backup; Master: The main owner is responsible for the data of keys in different slots, and the 0-16383 slots need to be assigned to the master server.
• Redis profile: Redis starts with a configuration file to initialize the server's information, using the command Redis-server redis.conf.
• The Redis cluster is built with a minimum of 3 Redis instances, 3 primary instances, and 0-16383 slots to three instances. The Redis instance from can be any, used as a backup switch.
• In addition to using IP plus port to mark a Redis real exception in a cluster of Node_id:redis in a Redis cluster, a Redis instance is marked with something called node_id.

Redis Cluster architecture

Redis-cluster Frame composition

1. All Redis nodes are interconnected (the ping-pong mechanism), using binary protocols internally to optimize transmission speed and bandwidth.
2. The fail of a node takes effect only when the detection of more than half of the nodes in the cluster fails.
3. The client is directly connected to the Redis node and does not require an intermediate proxy layer. The client does not need to connect to all nodes in the cluster to connect to any of the available nodes in the cluster.

4. Redis-cluster maps all the physical nodes to the [0-16383]slot, cluster is responsible for maintaining node<->slot<->value.

   Redis-cluster election fault tolerance
   

5. Lead the election process is all master in the cluster, if more than half of the master node communicates with the master node (cluster-node-timeout), the current master node is considered to be dead.

6. When the entire cluster is unavailable (Cluster_state:fail), when the cluster is unavailable, all operations on the cluster are unavailable, and the error is received ((error) Clusterdown the cluster is down).
   A: If any master of the cluster is hung, and the current master is not slave. The cluster enters the fail state, it can also be understood that the slot mapping into the group [0-16383] does not complete when entering the fail state.
   B: If more than half of the master hangs, regardless of whether there is a slave cluster into the fail state.

REDIS3 Installation

1. Downloading and unpacking

cd /data/crm/crmwebwget http:////download.redis.io/releases/redis-3.2.0.tar.gztar -zxvf /redis-3.2.1.tar.gz

2. Compiling and installing

cd redis-3.2.0makemake install

3. Create a Redis node

This time we select a machine for verification, general production on the cluster is not recommended on a machine host, to avoid a single point of failure of the system, can not ensure high availability. This time on a host to build a cluster (3 main nodes, 6 slave nodes).

CD/Data/crm/crmweb/redis-3.2. 0mkdir Redis_cluster//Create cluster directoryMkdir9579 9679 9779  //represents three primary nodes for their respective server ports 9579 9679 9779 //Create 9579 node For example, copy to 9579 directoryCP/Data/crm/crmweb/redis-3.2. 0/redis.Conf./redis_cluster/9579/    //Copy to 9679 directoryCP/Data/crm/crmweb/redis-3.2. 0/redis.Conf./redis_cluster/9679/    //Copy to 9779 directoryCP/Data/crm/crmweb/redis-3.2. 0/redis.Conf./redis_cluster/9779/   

Modify the corresponding configuration for the 3 files in the 9579,9679, 9779 folder respectively

Daemonize Yes//redis Background RunPidfile/var/run/redis_9579.Pid//pidfile file corresponds to 9579,9679,9779Port9579                                //Port 9579,9679,9779Cluster-enabledYes//Open cluster Remove comment #Cluster-config-filenodes_9579.Conf//Cluster configuration profile is first started automatically generated 9579,9679,9779Cluster-node-timeout   the                //Request Timeout setting 5 seconds enough.AppendOnly Yes//aof Log on if necessary, it will log each write operation

4. Start the above node

cd /data/crm/crmweb/redis-3.2.0/srcredis-server  ../redis_cluster/9579/redis.confredis-server  ../redis_cluster/9679/redis.confredis-server  ../redis_cluster/9779/redis.conf

Repeat 3, 42 steps to create the other slave nodes sequentially 8689,8699,9689,9699,9789,9799

5. View Startup results Ps-ef|grep Redis

Create a cluster

We are ready to build the Redis node for the cluster, then we will concatenate the nodes together to build the cluster. The official provides a tool: Redis-trib.rb (/DATA/CRM/CRMWEB/REDIS-3.2.0/SRC/REDIS-TRIB.RB) look at the suffix to know that the bird can not be directly executed, it is a program written in Ruby, So we have to install Ruby.

-y install ruby ruby-devel rubygems rpm-build

Then use the Gem This command to install the Redis interface Gem is a toolkit for Ruby.

gem install redis    //等一会儿就好了

The above steps are done, then run the REDIS-TRIB.RB

/usr/Local/redis-3.2. 1/SRC/REDIS-TRIB.RB Usage:redis-trib < Command> <options> <arguments ...>Reshard Host:port--to <arg>                  --yes                  --slots <arg>                  --from <arg>Check Host:port call Host:port command arg arg . Arg   Set-timeout Host:portmilliseconds  Add-node New_host:new_port Existing_host:existing_port--master-id <arg>                  --slaveDel-node host:port node_id fix host:port import host:port--from <arg>Help (show this help)CreateHost1:port1 ... Hostn:portn--replicas <arg>For check, fix, Reshard, Del-node,Set-timeout you can specify theHost andPort of  anyWorking nodeinch  theCluster.

Confirm that the above 9 nodes are started, then you can use the above command to create.

create--replicas 0 132.126.2.235:9579 132.126.2.235:9679 132.126.2.235:9779

Replicas 0 indicates that 0 nodes are created for the above three nodes, and of course 1, 3 nodes are created by default for the 1 node above. Since I want to specify that the previous 6 nodes are hung under different main nodes, it is not automatically assigned here.

Next, for the above 3 main nodes, hang from the node in turn.

./redis-trib.**add-node**----master-id132.126.2.235:8689132.126.2.235:9579

The above command is to use the 132.126.2.235:8689 node as the slave node of the 132.126.2.235:9579. Master-id EF131DA7722C9451B7F111A83C19DDBA8FAA3DF4 This can be used in a lot of ways./REDIS-TRIB.RB Check 132.126.2.235:9579 will also be displayed. Other nodes are also attached to the corresponding master node in turn. It's not written here.

Take a look at/data/crm/crmweb/redis-3.2.0/src/redis-trib.rb check 132.126.2.235:9579
Here the cluster has been initially built.

Next, you can log in to the next node to verify access to a value, cluster information, etc.
Login command:

redis-cli-h132.126.2.235-p9579

The above verification basically can demonstrate that the cluster has been successfully built.

REDIS3 configuration file

The above steps have been completed, the following description of the relevant configuration parameter meaning:
1.REDIS3 Cluster Configuration Items

#开启集群模式cluster-enabled yes#运行过程中集群信息保存的文件名，不能冲突，需要个性化cluster-config-file nodes.conf#集群节点通信内限定的超时时间5000#以追加的方式写数据appendonly yes#在某个主节点挂死的情况下，其他主节点仍然可以工作cluster-require-full-coverage no

2.redis3 Other Configurations

#绑定ip, you need to personalizeBind127.0. 0. 1#启用保护模式protected-mode Yes#绑定的服务端口, you need to personalizePort6379Tcp-backlog511#日志级别debug # verbose# notice# warningLogLevel Warning#redis连接多久不连接就关闭, units per second, 0 means continuous openTimeout0#用于检测连接是否挂死, send an ACK per number of secondsTcp-keepalive -Daemonize Nostop-writes-on-bgsave-error nordbcompression No#本地文件存储的数据, you need to personalizeDbfilename Dump.rdb#本地数据存储文件dbfilename的目录Dir./#开启日志文件LogFile Redis.log#pidfilePidfile/var/run/redis.pid#配置连接数MaxClients10000#配置使用内存MaxMemory4Gb#内存过期策略Maxmemory-policy VOLATILE-LRU#需要个性化Appendfilename"Appendonly.aof"Appendfsync Everysecno-appendfsync-on-rewrite Noauto-aof-rewrite-percentage -Auto-aof-rewrite-min-size -Mb

Summary of issues

Installation Issue 1:

command not found　　make[1127　　make[1directory `/data/crm/crmweb/redis-3.2.02

Workaround: GCC is not installed or the version is incorrect, install

yum  install  gcc

Installation Issue 2:

zmalloc.h:50:31 :   error : jemalloc/ Jemalloc.h:no such file or directory  zmalloc.h:55:2 : error:   #error   "newer version of Jemalloc required"  make[1 ]: * * * [ ADLIST.O] error  1  make[ 1 ]: Leaving directory '/data/crm/crmweb/redis-3.2  .0 /src Make: * * [ALL] error  2    

Workaround: The reason is that the jemalloc memory allocator is not installed, you can install Jemalloc or directly enter make MALLOC=LIBC && makes install

Cluster Connectivity Issue 1:
After the cluster was successfully built, a local tomcat application was introduced to access the cache cluster, and the following errors were always prompted: No reachable node in cluster

This problem actually trapped me all morning, the old thought is the cluster has what place improper configuration, but check repeatedly, online also looked up a lot of information, also finally did not solve. Later, I tried to authenticate a network connection to a node in the cluster locally:

cmd132.126.2.2359579

The results found that the bad luck, because I tried to telnet 132.126.2.235 8806 is a pass, originally only for this port alone opened.
WORKAROUND: Contact the integration colleague to release the firewall restrictions.

Other use problems, follow-up supplement ...

REDIS3 cache cluster (cluster) setup