Redis之codis3.0.1叢集簡單實現步驟

Codis 由四部分組成:
Codis Proxy (codis-proxy) Proxy 伺服器做請求轉寄，codis-proxy 本身實現了 Redis 協議，codis-proxy 本身是無狀態的.可以執行多個
Codis Dashboard (codis-config) codis工具，包括添加刪除節點，並且內建一個dashboard，點一點即可
Codis Redis (codis-server) 基於2.8.13開發的分支，只能使用codis-server
ZooKeeper/Etcd Codis 依賴 ZooKeeper 來存放資料路由表和 codis-proxy 節點的元資訊, codis-config 發起的命令都會通過 ZooKeeper 同步到各個存活的 codis-proxy.
architecture (1).png
Codis 支援按照 Namespace 區分不同的產品, 擁有不同的 product name 的產品, 各項配置都不會衝突

1，codis由go語言寫的，所以需要安裝go語言套件

[root@yum-down local]# wget https://storage.googleapis.com/golang/go1.6.linux-amd64.tar.gz
[root@yum-down local]# tar -zxvf go1.6.linux-amd64.tar.gz
[root@yum-down local]# cd /usr/local/go/src/
[root@yum-down src]# go version
go version go1.6 linux/amd64
[root@yum-down src]# mkdir /gopath/src/golang.org/x
[root@yum-down src]# go get github.com/tools/godep
[root@yum-down src]# vim /etc/profile
export PATH=$PATH:/usr/local/go/bin
export GOROOT=/usr/local/go
export PATH=$PATH:/gopath/bin
[root@yum-down local]# source /etc/profile
也可以yum安裝：

[root@yum-down local]# yum -y install gcc gcc-c++ make git wget go
[root@yum-down codis]# cat /etc/profile.d/go.sh
export GOPATH=/gopath
export PATH=$GOPATH/bin:$JAVA_HOME/bin:$PATH
2，安裝java

[root@yum-down local]# wget http://download.oracle.com/otn-pub/java/jdk/8u73-b02/jdk-8u73-linux-x64.rpm
[root@yum-down local]# rpm -ivh jdk-8u73-linux-x64.rpm
Preparing...                ########################################### [100%]
   1:jdk1.8.0_73            ########################################### [100%]
Unpacking JAR files...
    tools.jar...
    plugin.jar...
    javaws.jar...
    deploy.jar...
    rt.jar...
    jsse.jar...
    charsets.jar...
    localedata.jar...
    jfxrt.jar...
[root@yum-down opt]# cat /etc/profile.d/java
export JAVA_HOME=/usr/java/latest
export PATH=$JAVA_HOME/bin/:$PATH
export PATH=$PATH:/gopath/bin
[root@yum-down opt]# . /etc/profile.d/java
[root@yum-down local]# java -version
java version "1.8.0_73"
Java(TM) SE Runtime Environment (build 1.8.0_73-b02)
Java HotSpot(TM) 64-Bit Server VM (build 25.73-b02, mixed mode)
[root@yum-down local]#
3，zookeeper下載地址

[root@yum-down local]# wget http://mirrors.cnnic.cn/apache/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz
[root@yum-down local]# tar xf zookeeper-3.4.6.tar.gz
[root@yum-down local]# ln -s  zookeeper-3.4.6 zookeeper
[root@yum-down zookeeper]# mkdir /opt/{zk1,zk2,zk3}
複製設定檔：

[root@yum-down local]# cp /usr/local/zookeeper/conf/zoo_sample.cfg /opt/zoo.cfg
編輯設定檔，修改datadir.並且添加server（由於是一台機器，連接埠則不能相同），如下：

dataDir=/opt/zk1
clientPort=2181
server.1=10.10.0.250:2887:3887
server.2=10.10.0.250:2888:3888
server.3=10.10.0.250:2889:3889
將設定檔複製到每個目錄下：

[root@yum-down local]# cp /opt/zoo.cfg /opt/zk1/zk1.cfg
[root@yum-down local]# cp /opt/zoo.cfg /opt/zk2/zk2.cfg
[root@yum-down local]# cp /opt/zoo.cfg /opt/zk3/zk3.cfg
修改完後如下：

[root@yum-down local]# grep '^[a-z]' /opt/zk1/zk1.cfg
tickTime=2000     心跳時間
initLimit=10      心跳探測間隔數
syncLimit=5    
dataDir=/opt/zk1
clientPort=2181
server.1=10.10.0.250:2887:3887  叢集1
server.2=10.10.0.250:2888:3888  叢集2
server.3=10.10.0.250:2889:3889  叢集3
修改後如下：

[root@yum-down local]# grep '^[a-z]' /opt/zk2/zk2.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/opt/zk2
clientPort=2182
server.1=10.10.0.250:2887:3887
server.2=10.10.0.250:2888:3888
server.3=10.10.0.250:2889:3889
[root@yum-down local]# grep '^[a-z]' /opt/zk3/zk3.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/opt/zk3
clientPort=2183
server.1=10.10.0.250:2887:3887
server.2=10.10.0.250:2888:3888
server.3=10.10.0.250:2889:3889
[root@yum-down local]#
4，在一台機器上跑的話，在三個目錄，並且建立id

[root@yum-down local]# cd /opt/
[root@yum-down opt]# echo "1" > zk1/myid
[root@yum-down opt]# echo "2" > zk2/myid
[root@yum-down opt]# echo "3" > zk3/myid
＊＊＊＊＊＊＊＊＊＊＊＊zookeeper環境變數＊＊＊＊＊＊＊＊＊

[root@yum-down opt]#　echo -e "export ZOOKEEPER=/usr/local/zookeeper \nexport  PATH=\$ZOOKEEPER/bin:\$PATH" >> /etc/profile &&　source /etc/profile
＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊＊
啟動三個zookeeper

[root@yum-down local]# /usr/local/zookeeper/bin/zkServer.sh start /opt/zk1/zk1.cfg
ZooKeeper JMX enabled by default
Using config: /opt/zk1/zk1.cfg
Starting zookeeper ... STARTED
[root@yum-down local]# /usr/local/zookeeper/bin/zkServer.sh start /opt/zk2/zk2.cfg
ZooKeeper JMX enabled by default
Using config: /opt/zk2/zk2.cfg
Starting zookeeper ... STARTED
[root@yum-down local]# /usr/local/zookeeper/bin/zkServer.sh start /opt/zk3/zk3.cfg
ZooKeeper JMX enabled by default
Using config: /opt/zk3/zk3.cfg
Starting zookeeper ... STARTED
[root@yum-down local]#
查看是否已經自動選舉：

[root@yum-down opt]# /usr/local/zookeeper/bin/zkServer.sh status /opt/zk3/zk3.cfg
JMX enabled by default
Using config: /opt/zk3/zk3.cfg
Mode: follower  跟隨者
[root@yum-down opt]# /usr/local/zookeeper/bin/zkServer.sh status /opt/zk2/zk2.cfg
JMX enabled by default
Using config: /opt/zk2/zk2.cfg
Mode: leader  領導
[root@yum-down opt]# /usr/local/zookeeper/bin/zkServer.sh status /opt/zk1/zk1.cfg
JMX enabled by default
Using config: /opt/zk1/zk1.cfg
Mode: follower  跟隨者
[root@yum-down opt]#
串連測試：

[root@yum-down opt]# /usr/local/zookeeper/bin/zkCli.sh -server 10.10.0.250:2181
[zk: 10.10.0.250:2181(CONNECTED) 0] ls /
[zookeeper]
[zk: 10.10.0.250:2181(CONNECTED) 1] ls /zookeeper
[quota]
[zk: 10.10.0.250:2181(CONNECTED) 2]
下載codis

[root@yum-down ~]# go get -u -d github.com/CodisLabs/codis
package github.com/CodisLabs/codis: no buildable Go source files in /opt/gopath/src/github.com/CodisLabs/codis
[root@yum-down local]# cd /gopath/src/github.com/CodisLabs/codis/
[root@yum-down codis]# make
[root@yum-down codis]# make gotest
codis-config、codis-proxy、codis-server三個可執行檔已經產生
[root@yum-down bin]# ls
assets  codis-config  codis-proxy  codis-server
執行bin/codis-config dashboard，將會看到一個頁面
codis.png

[root@yum-down codis]# bin/codis-config dashboard
2016/03/04 07:22:03 dashboard.go:160: [INFO] dashboard listening on addr: :18087
2016/03/04 07:22:03 dashboard.go:143: [INFO] dashboard node created: /zk/codis/db_test/dashboard, {"addr": "10.10.0.250:18087", "pid": 33434}
2016/03/04 07:22:03 dashboard.go:144: [WARN] ********** Attention **********
2016/03/04 07:22:03 dashboard.go:145: [WARN] You should use `kill {pid}` rather than `kill -9 {pid}` to stop me,
2016/03/04 07:22:03 dashboard.go:146: [WARN] or the node resisted on zk will not be cleaned when I'm quiting and you must remove it manually
2016/03/04 07:22:03 dashboard.go:147: [WARN] *******************************
2016/03/04 07:23:07 dashboard_apis.go:88: [ERROR] get server groups failed
[error]: zk: node does not exist
    1   /gopath/src/github.com/CodisLabs/codis/pkg/models/server_group.go:110
            github.com/CodisLabs/codis/pkg/models.ServerGroups
    0   /gopath/src/github.com/CodisLabs/codis/cmd/cconfig/dashboard_apis.go:86
            main.apiGetServerGroupList
        ... ...
[stack]:
    0   /gopath/src/github.com/CodisLabs/codis/cmd/cconfig/dashboard_apis.go:88
            main.apiGetServerGroupList
        ... ...
啟動兩個redis（確保redis已經安裝）
建立兩個redis

[root@yum-down opt]# mkdir date1
[root@yum-down opt]# mkdir date2
[root@yum-down opt]# cp /etc/redis/6379.conf ./
[root@yum-down opt]# vim 6379.conf
將連接埠，pid修改，如：8001

[root@yum-down opt]# grep "8001" 6379.conf
pidfile /var/run/redis_8001.pid
# Accept connections on the specified port, default is 8001.
port 8001
dir /opt/date1
# cluster-config-file nodes-8001.conf
[root@yum-down opt]# cp 6379.conf date1/8001.conf
8002

[root@yum-down opt]# grep "8002" 6379.conf
pidfile /var/run/redis_8002.pid
# Accept connections on the specified port, default is 8002.
port 8002
dir /opt/date2
# cluster-config-file nodes-8002.conf
[root@yum-down opt]#
啟動

[root@yum-down opt]# /usr/local/redis/bin/redis-server /opt/date1/8001.conf
[root@yum-down opt]# /usr/local/redis/bin/redis-server /opt/date2/8002.conf
[root@yum-down opt]# ps aux|grep redis
root      36449  0.1  0.7 137448  7484 ?        Ssl  07:38   0:00 /usr/local/redis/bin/redis-server *:6379             
root      36485  0.0  0.7 137448  7436 ?        Ssl  07:48   0:00 /usr/local/redis/bin/redis-server *:8001             
root      36491  0.1  0.7 137448  7436 ?        Ssl  07:48   0:00 /usr/local/redis/bin/redis-server *:8002
啟動後手動添加即可
cordis-crete-1.png
串連後查看

[root@yum-down opt]# /usr/local/redis/bin/redis-cli -h 10.10.0.250 -p 8002
role:slave
master_host:10.10.0.250
master_port:8001
master_link_status:up
master_last_io_seconds_ago:8
master_sync_in_progress:0
slave_repl_offset:85
slave_priority:100
slave_read_only:1
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
手動提交提升主後，將會被孤立，主從將消失offline：
codis-提升主從.png

10.10.0.250:8002>
# Replication
role:master
connected_slaves:0
master_repl_offset:239
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
和之前的版本不同的是codis3.0.1的指令碼放在CodisLabs/codis/bin下：

[root@yum-down bin]# ls
assets  codis-config  codis-proxy  codis-server
[root@yum-down bin]# ll
total 37520
drwxr-xr-x 4 root root     4096 Mar  4 06:30 assets
-rwxr-xr-x 1 root root 16141960 Mar  4 06:30 codis-config
-rwxr-xr-x 1 root root 15961984 Mar  4 06:29 codis-proxy
-rwxr-xr-x 1 root root  6311107 Mar  4 06:30 codis-server
[root@yum-down bin]# pwd
/gopath/src/github.com/CodisLabs/codis/bin
設定檔仍然是config.ini

coordinator=zookeeper
zk=10.10.0.250:2181
product=test
dashboard_addr=10.10.0.250:18087
並且他還可以添加服務的slot範圍
資料移轉：
用gitlab作者的原話是：資料移轉的最小單位是 key, 我們在 codis redis 中添加了一些指令, 實現基於key的遷移, 如 SLOTSMGRT等 (命令列表), 每次會將特定 slot 一個隨機的 key 發送給另外一個 codis redis 執行個體, 這個命令會確認對方已經接收, 同時刪除本地的這個 k-v 索引值, 返回這個 slot 的剩餘 key 的數量, 整個操作是原子的.
支援動態根據執行個體記憶體, 自動對slot進行遷移, 以均衡資料分布.

他有很多命令不支援：https://github.com/CodisLabs/codis/blob/3.0.1/doc/unsupported_cmds.md
至於codis-proxy設計到zookeeper，如果對zookeeper不熟悉的話，有很多問題是無法解決的

初始化 slots

[root@yum-down codis]# bin/codis-config slot init
{
  "msg": "OK",
  "ret": 0
}
[root@yum-down codis]#
設定 server group 服務的 slot 範圍
添加group2

[root@yum-down codis]# bin/codis-config server add 2 10.10.0.250:6379 master
{
  "msg": "OK",
  "ret": 0
}
[root@yum-down codis]# bin/codis-config server add 2 10.10.0.250:8001 slave
{
  "msg": "OK",
  "ret": 0
}
添加group3
[root@yum-down codis]# bin/codis-config server add 3 10.10.0.250:8001 master
{
  "msg": "OK",
  "ret": 0
}
[root@yum-down codis]# bin/codis-config server add 3 10.10.0.250:8002 slave
{
  "msg": "OK",
  "ret": 0
}
設定server group伺服器的slot範圍

[root@yum-down codis]# bin/codis-config slot range-set 0 511 1 online
{
  "msg": "OK",
  "ret": 0
}
[root@yum-down codis]# bin/codis-config slot range-set 512 1023 2 online
{
  "msg": "OK",
  "ret": 0
}
codis-三個.png
啟動codis-proxy

[root@yum-down codis]# bin/codis-proxy -c config.ini -L ./log/proxy.log  --cpu=8 --addr=0.0.0.0:19000 --http-addr=0.0.0.0:11000
  _____  ____    ____/ /  (_)  _____
 / ___/ / __ \  / __  /  / /  / ___/
/ /__  / /_/ / / /_/ /  / /  (__  )
\___/  \____/  \__,_/  /_/  /____/

[root@yum-down codis]#
資料移轉：

[root@yum-down codis]# bin/codis-config slot migrate 0 511 2 --delay=10
{
  "msg": "OK",
  "ret": 0
}
[root@yum-down codis]#
codis到此為止，瞭解即可！