[Research and use of]redis-cluster-pending study

Transferred from: http://hot66hot.iteye.com/blog/2050676

One: The current status of Redis cluster 1:redis cluster

Reids-cluster plan in the redis3.0, can be regarded as a Antirez statement: HTTP://ANTIREZ.COM/NEWS/49 (PS: Skip the ticket for a long time, this year seems to speed up), the latest version of the current see: https:// Raw.githubusercontent.com/antirez/redis/3.0/00-releasenotes

The author's goal: Redis Cluster would support up to ~1000 nodes. Praise...

Current Redis-supported cluster features (tested):

1. Node Auto discovery
2. Slave->master elections, cluster fault tolerance
3. Hot resharding: Online sharding
4. Cluster Management: Cluster XXX
5. Configuration-based (nodes-port.conf) cluster management
6. ASK steering/moved steering mechanism.

2:redis cluster architecture 1) redis-cluster frame composition

Architectural Details:

1. All Redis nodes are interconnected (the ping-pong mechanism), using binary protocols internally to optimize transmission speed and bandwidth.
2. The fail of a node takes effect only when the detection of more than half of the nodes in the cluster fails.
3. The client is directly connected to the Redis node and does not require an intermediate proxy layer. The client does not need to connect to all nodes in the cluster to connect to any of the available nodes in the cluster
4. Redis-cluster maps all the physical nodes to the [0-16383]slot, cluster is responsible for maintaining Node<->slot<->value

2) Redis-cluster election: Fault tolerance

1. Lead the election process is all master in the cluster, if more than half of the master node communicates with the master node (cluster-node-timeout), the current master node is considered to be dead.
2. When does the entire cluster not be available (Cluster_state:fail)?
1. If the cluster any master hangs, and the current master does not have slave. The cluster enters the fail state, it can also be understood that the cluster's slot mapping [0-16383] does not complete when it enters the fail state. Ps:redis-3.0.0.rc1 joins the Cluster-require-full-coverage parameter, which is turned off by default and the cluster compatibility section fails to open.
2. If the cluster exceeds more than half the master hangs, regardless of whether there is a slave cluster into the fail state.

PS: When the cluster is unavailable, all operations on the cluster are not available, received ((error) Clusterdown the cluster is down) error

II: Use of Redis cluster 1: Installing Redis cluster

1): Install Redis-cluster dependency: Redis-cluster dependent Library when using a compatibility problem, in Reshard encountered a variety of errors, please install according to the specified version.

(1) Ensure that the system is installed zlib, otherwise gem install will report (no such file to load--zlib)

1. #download: Zlib-1.2.6.tar
2. ./configure
3. Make
4. Make install

(2) Installing Ruby:version (1.9.2)

1. # ruby1.9.2
2. Cd/path/ruby
3. ./configure-prefix=/usr/local/ruby
4. Make
5. Make install
6. sudo cp Ruby/usr/local/bin

(3) Installing Rubygem:version (1.8.16)

1. # rubygems-1.8.16.tgz
2. Cd/path/gem
3. sudo ruby setup.rb
4. sudo cp Bin/gem/usr/local/bin

(4) Installing Gem-redis:version (3.0.0)

1. Gem Install Redis--version 3.0.0
2. #由于源的原因, may download fails, download it manually and install it
3. #download地址: http://rubygems.org/gems/redis/versions/3.0.0
4. Gem Install-l/data/soft/redis-3.0.0.gem

(5) Installation Redis-cluster

1. Cd/path/redis
2. Make
3. sudo cp/opt/redis/src/redis-server/usr/local/bin
4. sudo cp/opt/redis/src/redis-cli/usr/local/bin
5. sudo cp/opt/redis/src/redis-trib.rb/usr/local/bin

2: Configure Redis cluster 1) redis configuration file structure:

Use inclusion (include) to separate common configurations and special configurations for easy maintenance.

2) Redis Universal configuration.

2. #GENERAL
4. Daemonize No
6. Tcp-backlog 511
8. Timeout 0
10. Tcp-keepalive 0
12. LogLevel Notice
14. Databases 16
16. Dir/opt/redis/data
18. Slave-serve-stale-data Yes
20. #slave只读
22. Slave-read-only Yes
24. #not Use default
26. Repl-disable-tcp-nodelay Yes
28. Slave-priority 100
30. #打开aof持久化
32. AppendOnly Yes
34. #每秒一次aof写
36. Appendfsync everysec
38. The new write operation is fsync when the #关闭在aof rewrite
40. No-appendfsync-on-rewrite Yes
42. Auto-aof-rewrite-min-size 64MB
44. Lua-time-limit 5000
46. #打开redis集群
48. cluster-enabled Yes
50. #节点互连超时的阀值
52. Cluster-node-timeout 15000
54. Cluster-migration-barrier 1
56. Slowlog-log-slower-than 10000
58. Slowlog-max-len 128
60. Notify-keyspace-events ""
62. Hash-max-ziplist-entries 512
64. Hash-max-ziplist-value 64
66. List-max-ziplist-entries 512
68. List-max-ziplist-value 64
70. Set-max-intset-entries 512
72. Zset-max-ziplist-entries 128
74. Zset-max-ziplist-value 64
76. activerehashing Yes
78. Client-output-buffer-limit Normal 0 0 0
80. Client-output-buffer-limit slave 256MB 64MB 60
82. Client-output-buffer-limit pubsub 32MB 8MB 60
84. Hz 10
86. Aof-rewrite-incremental-fsync Yes

3) Redis special configuration.

2. #包含通用配置
4. Include/opt/redis/redis-common.conf
6. #监听tcp端口
8. Port 6379
10. #最大可用内存
12. MaxMemory 100m
14. #内存耗尽时采用的淘汰策略:
16. # VOLATILE-LRU, remove the key with a expire set using an LRU algorithm
18. # ALLKEYS-LRU, remove any key accordingly to the LRU algorithm
20. # volatile-random, remove a random key with an expire set
22. # allkeys-random, remove a random key, any key
24. # Volatile-ttl, remove the key with the nearest Expire time (minor TTL)
26. # noeviction, don ' t expire at all, just return a error on write operations
28. Maxmemory-policy ALLKEYS-LRU
30. #aof存储文件
32. Appendfilename "Appendonly-6379.aof"
34. #不开启rdb存储, only for adding slave procedures
36. Dbfilename Dump-6379.rdb
38. #cluster配置文件 (Start auto-build)
40. Cluster-config-file nodes-6379.conf
42. #部署在同一机器的redis实例, twist the Auto-aof-rewrite, because the memory footprint in the cluster environment is basically the same.
44. #防止同意机器下瞬间fork所有redis进程做aof rewrite, takes up a lot of memory (Ps:cluster must turn on AOF)
46. Auto-aof-rewrite-percentage 80-100

3:cluster operation

Cluster cluster-related commands, more Redis-related commands See document: http://redis.readthedocs.org/en/latest/

2. Cluster
4. CLUSTER Info Print cluster information
6. CLUSTER NODES Lists all nodes currently known to the cluster, as well as information about those nodes.
8. Node
10. CLUSTER MEET <ip> <port> Add the node specified by IP and port to the cluster, making it part of the cluster.
12. CLUSTER Forget <node_id> removes node_id specified node from the cluster.
14. CLUSTER REPLICATE <node_id> Sets the current node to node_id node from the specified node.
16. CLUSTER Saveconfig Save the configuration file of the node to the hard disk.
18. Slot (slot)
20. CLUSTER addslots <slot> [slots ...] assigns one or more slots (slots) (assign) to the current node.
22. CLUSTER delslots <slot> [slots ...] removes the assignment of one or more slots to the current node.
24. CLUSTER Flushslots removes all slots assigned to the current node, making the current node a node without any slots assigned.
26. CLUSTER setslot <slot> Node <node_id> assigns the slot slot to the node specified by node_id, and if the slot is already assigned to another node, let the other node delete the slot before assigning it.
28. CLUSTER setslot <slot> Migrating <node_id> migrates the slot slots of this node to the nodes specified in the node_id.
30. CLUSTER setslot <slot> importing <node_id> import slot slots from node_id specified node to this node.
32. CLUSTER Setslot <slot> STABLE cancels the import or migration (migrate) of the slot slots.
34. Key
36. CLUSTER keyslot <key> Calculate the key should be placed on which slot.
38. CLUSTER Countkeysinslot <slot> Returns the number of key-value pairs currently contained in the slot slot.
40. CLUSTER Getkeysinslot <slot> <count> returns the key in count slot slots.

4:redis Cluster operation 1) Initialize and build the cluster

(1) Start the cluster-related node (must be an empty node, beta3 can be a node with data), specify the configuration file and output log

1. redis-server/opt/redis/conf/redis-6380.conf >/opt/redis/logs/redis-6380.log 2>&1 &
2. redis-server/opt/redis/conf/redis-6381.conf >/opt/redis/logs/redis-6381.log 2>&1 &
3. redis-server/opt/redis/conf/redis-6382.conf >/opt/redis/logs/redis-6382.log 2>&1 &
4. redis-server/opt/redis/conf/redis-7380.conf >/opt/redis/logs/redis-7380.log 2>&1 &
5. redis-server/opt/redis/conf/redis-7381.conf >/opt/redis/logs/redis-7381.log 2>&1 &
6. redis-server/opt/redis/conf/redis-7382.conf >/opt/redis/logs/redis-7382.log 2>&1 &

(2): Build a cluster with your own Ruby tool (REDIS-TRIB.RB)

1. Create sub-command construction #redis-TRIB.RB
2. #--replicas specifies several slave nodes for each master node in the Redis cluster
3. #节点角色由顺序决定, first master is slave (for easy identification, the port of slave is 1000 larger than master)
4. REDIS-TRIB.RB Create--replicas 1 10.10.34.14:6380 10.10.34.14:6381 10.10.34.14:6382 10.10.34.14:7380 10.10.34.14:7381 10.10.34.14:7382

(3): Check cluster status

1. #redis-trib.rb Check subcommand build
2. #ip:p ort can be any node in the cluster
3. REDIS-TRIB.RB Check 10.10.34.14:6380

Finally output the following information, there is no warning or error, indicating that the cluster started successfully and is in OK State

1. [OK] All nodes agree about slots configuration.
2. >>> Check for open Slots ...
3. >>> Check Slots Coverage ...
4. [OK] All 16384 slots covered.

2): Add new Master Node

(1) Add a master node: Create an empty node, and then move some slots to this empty node, the process currently requires manual intervention

A): Generate configuration files based on port (ps:establish_config.sh is the output configuration script I wrote myself)

1. SH establish_config.sh 6386 > conf/redis-6386.conf

b): Start node

1. redis-server/opt/redis/conf/redis-6386.conf >/opt/redis/logs/redis-6386.log 2>&1 &

c): Join the empty node to the cluster
Add-node adds a node to the cluster, the first one is the new node Ip:port, and the second is any existing node Ip:port

1. REDIS-TRIB.RB Add-node 10.10.34.14:6386 10.10.34.14:6381

Node: The new node does not contain any data because it does not contain any slots. The newly added add-on is a master node that will not be selected when the cluster needs to upgrade a slave node to a new master node

D): Allocate slots for new nodes

1. REDIS-TRIB.RB Reshard 10.10.34.14:6386
2. #根据提示选择要迁移的slot数量 (PS: choose here 500)
3. How many slots does want to move (from 1 to 16384)? 500
4. #选择要接受这些slot的node-id
5. What is the receiving node ID? F51e26b5d5ff74f85341f06f28f125b7254e61bf
6. #选择slot来源:
7. #all表示从所有的master重新分配,
8. #或者数据要提取slot的master节点id, and end with done.
9. Please enter the source node IDs.
10. The Type ' All ' is the nodes as source nodes for the hash slots.
11. Type ' Done ' once your entered all the source nodes IDs.
12. Source node #1: All
13. #打印被移动的slot后, enter Yes to begin moving the slot and the corresponding data.
14. #Do want to proceed with the proposed Reshard plan (yes/no)? Yes
15. #结束

3): Add a new slave node

A): The first three steps are the same as adding master

b) Step Fourth: REDIS-CLI connect the new node shell, enter the command: cluster replicate corresponding to the Node-id of the master

1. Cluster replicate 2b9ebcbd627ff0fd7a7bbcc5332fb09e72788835

Note: When adding slave online, the entire master process needs to be dump and passed to slave, and the Rdb file is loaded into memory by slave, and the RDB transfer process Master may not be able to provide the service, the whole process consumes a lot of Io, careful operation.

For example, this time adding an RDB file generated by the slave operation

1. -rw-r--r--1 root root 34946 Apr 18:23 Dump-6386.rdb
2. -rw-r--r--1 root root 34946 Apr 18:23 Dump-7386.rdb

4): Online Reshard data:

For the load/data uneven situation, can be resolved online reshard slot, the same method as adding a new master Reshard, just need to reshard the master node is the old node.

5): Delete a Slave node

1. #redis-trib del-node ip:port ' <node-id> '
2. REDIS-TRIB.RB del-node 10.10.34.14:7386 ' c7ee2fca17cb79fe3c9822ced1d4f6c5e169e378 '

6): Delete a master node

A): Before deleting the master node, first use Reshard to remove all of the master slots and then delete the current node (currently only deleted

Master's slots are migrated to a single node)

1. #把10.10.34.14:6386 Current Master migrated to 10.10.34.14:6380
2. REDIS-TRIB.RB Reshard 10.10.34.14:6380
3. #根据提示选择要迁移的slot数量 (PS: choose here 500)
4. How many slots does want to move (from 1 to 16384)? 500 (total number of slots deleted master)
5. #选择要接受这些slot的node-id (10.10.34.14:6380)
6. What is the receiving node ID? C4A31C852F81686F6ED8BCD6D1B13ACCDC947FD2 (ps:10.10.34.14:6380 's Node-id)
7. Please enter the source node IDs.
8. The Type ' All ' is the nodes as source nodes for the hash slots.
9. Type ' Done ' once your entered all the source nodes IDs.
10. Source node #1: F51E26B5D5FF74F85341F06F28F125B7254E61BF (node-id deleted from master)
11. Source node #2:d One
12. #打印被移动的slot后, enter Yes to begin moving the slot and the corresponding data.
13. #Do want to proceed with the proposed Reshard plan (yes/no)? Yes

b): Delete Empty master node

1. REDIS-TRIB.RB del-node 10.10.34.14:6386 ' F51E26B5D5FF74F85341F06F28F125B7254E61BF '

Three: Redis cluster client (Jedis) 1: Client basic operation using

2. <span style= "color: #333333; Font-family:arial, Sans-serif; " > private static Binaryjediscluster JC;
4. static {
6. Only one instance of the cluster can be
8. set
10. Jedisclusternodes.add (New Hostandport ("10.10.34.14", 6380));
12. Jedisclusternodes.add (New Hostandport ("10.10.34.14", 6381));
14. Jedisclusternodes.add (New Hostandport ("10.10.34.14", 6382));
16. Jedisclusternodes.add (New Hostandport ("10.10.34.14", 6383));
18. Jedisclusternodes.add (New Hostandport ("10.10.34.14", 6384));
20. Jedisclusternodes.add (New Hostandport ("10.10.34.14", 7380));
22. Jedisclusternodes.add (New Hostandport ("10.10.34.14", 7381));
24. Jedisclusternodes.add (New Hostandport ("10.10.34.14", 7382));
26. Jedisclusternodes.add (New Hostandport ("10.10.34.14", 7383));
28. Jedisclusternodes.add (New Hostandport ("10.10.34.14", 7384));
30. JC = new Binaryjediscluster (jedisclusternodes);
32. }
34. @Test
36. public void Testbenchredisset () throws Exception {
38. Final Stopwatch Stopwatch = new Stopwatch ();
40. List List = Buildblogvideos ();
42. for (int i = 0; i <; i++) {
44. String key = "key:" + i;
46. Stopwatch.start ();
48. byte[] bytes1 = protostuffserializer.serialize (list);
50. Jc.setex (Key, *, bytes1);
52. Stopwatch.stop ();
54. }
56. System.out.println ("time=" + stopwatch.tostring ());
58. }</span>

2:jedis the client's pit.

1) Redis Slave does not accept any read/write operations under the cluster environment.

2) The client side does not support the keys batch operation, does not support the Select Dbnum operation, only one db:select 0

3) Jediscluster info () and other standalone functions cannot be called, return (No way to dispatch the This command to Redis Cluster) error.

4) Jediscluster does not have an API for byte[] and needs to expand itself (attachment is my add-on byte[-based Binaryjediscluster API)

Reference Documentation:

Http://redis.io/topics/cluster-spec

Http://redis.io/topics/cluster-tutorial

[Research and use of]redis-cluster-pending study