Deployment of codis Clusters

I. Summary 1. It has been almost two months since codis cluster tossing. Thanks to Liu Qi, author of codis, and Huang Dongxu for their patience and support. Here, I would like to give you a thumbs up, now we have a business running on the codis cluster. Currently, we only cut 10% of the total business volume. We are expected to switch the entire business volume to codis next week. At this time, we are sure to want to know about codis.

I. Summary 1. It has been almost two months since codis cluster tossing. Thanks to Liu Qi, author of codis, and Huang Dongxu for their patience and support. Here, I would like to give you a thumbs up, now we have a business running on the codis cluster. Currently, we only cut 10% of the total business volume. We are expected to switch the entire business volume to codis next week. At this time, we are sure to want to know about codis.

I. Summary

1. We have been tossing the codis cluster for almost two months. Thanks to Liu Qi, author of codis, and Huang Dongxu for their patience and support. Here, I would like to give you a thumbs up, now we have a business running on the codis cluster. At present, we only cut 10% of the total business volume. We expect to switch the entire business to codis next week, at this time, you must particularly want to know that codis is stable? Are there any bugs? I want to know what you want to know. You just need to know it when you get started. At present, we have not found any problems, some small questions have been promptly removed from the contact with the author. Okay, don't talk nonsense, the purpose of this article is to help beginners who want to know about codis to quickly deploy it (the official deployment documentation is still a little effort-consuming for children's shoes with weak O & M knowledge) you can also prepare a record for later cluster deployment.

2. Codis is a distributed Redis solution. For upper-layer applications, there is no obvious difference between connecting to Codis Proxy and connecting to the native Redis Server (some commands are not supported ), upper-layer applications can be used like single-host Redis. Codis processes request forwarding and data migration without stopping services, the client is transparent. You can simply think that the backend connection is a Redis service with infinite memory. Of course, redis released a stable version of the official 3.0 version some time ago, 3.0 supports cluster functions. The implementation principle of codis is similar to that of cluster functions of 3.0. I understand that Meituan and Alibaba have already used 3.0 of cluster functions. Our business here is mainly php, the sdk for the 3.0 cluster does not currently support the php language. If your php application has a 3.0 cluster, please contact me and I will learn it. related to common cluster technologies of redis, please move to the infoq column Redis cluster technology and Codis practices of @ Xiao tianguo Xiao

Ii. Architecture

650) this. width = 650; "src =" http://www.68idc.cn/help/uploads/allimg/151111/1212341447-0.jpg "title =" 11111111111.png" alt = "wKiom1U4wNbil8RQAAIeNjYHmcE992.jpg"/>

Iii. Batch roles

Zookeeper cluster: zookeeper-config, codis-ha: 10.10.32.10: small: 10.10.32.10: small: 10.10.32.42: 6379, 10.10.32.43: 6380 (master, slave) 10.32.43: 6379, 10.10.32.44: 6380 (master and slave) 10.10.32.44: 6379, 10.10.32.42: 6380 (master and slave)

Iv. Deployment

1. Install zookeeper

Yum-y install zookeeper jdk # install the service

Vim/etc/hosts # Add host10.10.0.47 ZooKeeper-node110.10.0.48 ZooKeeper-node210.10.1.76 ZooKeeper-node3

Vim/etc/zookeeper/conf/zoo. cfg # Write zk configuration file maxClientCnxns = 50 tickTime = 2000 initLimit = 10 syncLimit = 5 dataDir =/data/zookeeper/clientPort = 2181server. 1 = ZooKeeper-node1: 2888: 3888server. 2 = ZooKeeper-node2: 2888: 3888server. 3 = ZooKeeper-node3: 2888: 3888

Mkdir/data/zookeeper/# create the zk datadir directory echo "2">/data/zookeeper/myid # generate an ID. Note that the corresponding Zo ID is. cfg server. ID, for example, the ZooKeeper-node2 corresponding to myid should be 2/usr/lib/zookeeper/bin/zkServer. sh start # start a service

2. go installation (codis is written in the go language, so you need to install it on those machines)

wget https://storage.googleapis.com/golang/go1.4.1.linux-amd64.tar.gztar -zxvf go1.4.1.linux-amd64.tar.gzmv go /usr/local/cd /usr/local/go/src/bash all.bashcat >> ~/.bashrc << _bashrc_exportexport GOROOT=/usr/local/goexport PATH=\$PATH:\$GOROOT/binexport GOARCH=amd64export GOOS=linux_bashrc_exportsource ~/.bashrc

3. Download and compile codis (the machine where codis-config, codis-proxy, and codis-server are located)

mkdir /data/goexport GOPATH=/data/go/usr/local/go/bin/go get github.com/wandoulabs/codiscd  /data/go/src/github.com/wandoulabs/codis/./bootstrap.shmake gotest

5. Start services and initialize Clusters

1. Start dashboard (operations on codis-config)

Cat/etc/codis/config_10.ini # Write configuration file zk = 10.10.0.47: 2181, 10.10.0.48: 2181, 10.10.1.76: 2181 product = zh_newsproxy_id = codis-proxy_10net_timeout = 5000 proto = route = 10.10.32.10: 18087

cd /data/go/src/github.com/wandoulabs/codis/ &&  ./bin/codis-config -c /etc/codis/config_10.ini  dashboard &

2. initialize slots (operations on codis-config)

cd /data/go/src/github.com/wandoulabs/codis/ &&  ./bin/codis-config -c /etc/codis/config_10.ini slot init

3. Start Codis Redis, which is the same as the official Redis Server parameter (Operation on codis-server)

cd /data/go/src/github.com/wandoulabs/codis/ && ./bin/codis-server /etc/redis_6379.conf &

4. Add a Redis Server Group. Each Server Group exists as a Redis Server Group. Only one master is allowed and multiple slave instances are allowed, group id only supports integers greater than or equal to 1 (operations on codis-config)

cd /data/go/src/github.com/wandoulabs/codis/./bin/codis-config -c /etc/codis/config_10.ini server add 1 10.10.32.42:6379 master./bin/codis-config -c /etc/codis/config_10.ini server add 1 10.10.32.43:6380 slave./bin/codis-config -c /etc/codis/config_10.ini server add 2 10.10.32.43:6379 master./bin/codis-config -c /etc/codis/config_10.ini server add 2 10.10.32.44:6380 slave./bin/codis-config -c /etc/codis/config_10.ini server add 3 10.10.32.44:6379 master./bin/codis-config -c /etc/codis/config_10.ini server add 3 10.10.32.42:6380 slave

5. Set the slot range of the server group service Codis to implement data sharding using the Pre-sharding technology. By default, it is divided into 1024 slots (0-1023). For each key, use the following formula to determine the Slot Id: SlotId = crc32 (key) % 1024 each slot will have a specific server group id to indicate which server group the slot data is provided. (Operation on codis-config)

cd /data/go/src/github.com/wandoulabs/codis/./bin/codis-config -c /etc/codis/config_10.ini slot range-set 0 300 1 online./bin/codis-config -c /etc/codis/config_10.ini slot range-set 301 700 2 online./bin/codis-config -c /etc/codis/config_10.ini slot range-set 701 1023 3 online

6. Start codis-proxy (Operation on codis-proxy)

Cat/etc/codis/config_10.ini # Write configuration file zk = 10.10.0.47: 2181, 10.10.0.48: 2181, 10.10.1.76: 2181 product = zh_newsproxy_id = codis-proxy_10 #10.10.32.49 on change to codis-proxy_49, multiple proxies. The proxy_id must be a unique net_timeout = 5000 proto = tcp4dashboard_addr = 10.10.32.10: 18087

cd /data/go/src/github.com/wandoulabs/codis/ &&  ./bin/codis-proxy  -c /etc/codis/config_10.ini -L /data/log/codis-proxy_10.log  --cpu=4 --addr=0.0.0.0:19000 --http-addr=0.0.0.0:11000 &cd /data/go/src/github.com/wandoulabs/codis/ &&  ./bin/codis-proxy  -c /etc/codis/config_49.ini -L /data/log/codis-proxy_49.log  --cpu=4 --addr=0.0.0.0:19000 --http-addr=0.0.0.0:11000 &

OK. The entire cluster has been set up successfully. show it to you.

650) this. width = 650; "src =" http://www.68idc.cn/help/uploads/allimg/151111/1212343524-1.jpg "title =" 222222222222.png" alt = "wKiom1U4woPj8_kiAAIq1SRxwxo029.jpg"/>

6. codis-server HA

Codis-ha implements the master-slave switchover of codis-server. If the master database of codis-server crashes, a slave database is upgraded to the master database. If the slave database crashes, the slave database is set to go offline from the cluster.

1. Installation

Export GOPATH =/data/go/usr/local/go/bin/go get github.com/ngaut/codis-hacd/data/go/src/export buildcp codis-ha/data/go/src/export -- codis-config = dashboard address: 18087 -- productName = cluster project name

2. Use supervisord to manage the codis-ha Process

yum -y install supervisord

/Etc/supervisord. add the following content to conf: [program: codis-ha] autorestart = Truestopwaitsecs = 10 startsecs = 1 stopsignal = QUITcommand =/data/go/src/sources -- codis-config = 10.10.32.17: 18087 -- productName = zh_newsuser = rootstartretries = 3 autostart = Trueexitcodes =

3. Start the supervisord Service

/etc/init.d/supervisord startchkconfig supervisord  on

In this case, ps-ef | grep codis-ha. You will find that the codis-ha process has been started. At this time, stop a codis-server master and check whether the slave will be upgraded to a master.

VII. Monitoring

We use zabbix for the monitoring of the entire codis cluster. The monitoring metrics are relatively simple. Therefore, I would like to provide more suggestions for you.

Zookeeper: monitors port connectivity of each node (You may want to monitor the process in the future)

Codis-proxy: monitors port connectivity, which is far from enough.

Codis-server: monitors memory usage, connections, and connectivity

Codis-ha: monitoring process

Dashboard: monitoring port connectivity

8. Problems Encountered during use

1. Log cutting of codis-proxy. The default Log Level of codis-proxy is info. The log volume is large. We generate over 50 GB logs every day, currently, codis-proxy does not support hot restart. It is troublesome to modify the startup parameters. logrotate is recommended for log splitting.

2. The default listening address of codis-proxy does not have a specific ipv4 address, that is, no listening like 0.0.0.0: 19000 after codis-proxy is started, this will cause the problem that the front-end lvs cannot handle Server Load balancer codis-proxy and cannot forward requests. The problem has been handled by the author, add proto = tcp4 to the configuration file started by codis-proxy to support listening to ipv4 addresses.

3. When you add a Redis Server Group, non-codis-server (native redis) can also be added to the codis cluster. There is a clear physical machine between redis and codis-server, it is easy to add an error and you want to have a verification. Non-codis-server cannot be added to the codis cluster.

4. The internal communication of the codis cluster is through the host name. If the host name does not have domain name resolution, the dashboard cannot access the proxy's http-addr address through the host name, this will cause the OP/s data to be invisible from the web interface. There are no other problems. I have not found any problem yet. We recommend that you directly use the Intranet IP address for internal communication.

PS: I have established a QQ Group for managing codis. If you are interested in codis, join the group number 183613045.