Redis Note-taking (ii): Java API usage and Redis distributed cluster environment setup

[TOC]

Redis Note-taking (ii): Java API use with Redis distributed cluster environment using Redis Java API (i): Standalone version of Redis API usage

The Redis Java API operates through Jedis, so the first Jedis third-party libraries are needed, because the MAVEN project is used, so the Jedis dependency is given first:

<dependency>    <groupId>redis.clients</groupId>    <artifactId>jedis</artifactId></dependency>

Basic code example

The commands that Redis can provide, Jedis are also provided, and are very similar in use, so here's just some code for the operation.

Package Com.uplooking.bigdata;import Org.junit.after;import Org.junit.before;import org.junit.test;import  Redis.clients.jedis.jedis;import java.util.list;import Java.util.map;import java.util.set;/** * Redis operation Java API *    Jedis is the gateway to our Redis Java API * A Jedis object that represents a redis connection * CRUD */public class Redistest {private Jedis Jedis;    Private String host;    private int port;        @Before public void SetUp () {host = "uplooking01";        Port = 6379;    Jedis = new Jedis (host, Port); } @Test public void Testcrud () {//after going to the collection of all keys set<string> keys = Jedis.keys ("*");//Jedi S.select (index);        Specifies the database to perform the operation by default, number No. 0 data System.out.println (keys);        String System.out.println ("**************string**************");        Delete key nam1 Long del = Jedis.del ("nam1") in Redis; System.out.println (del = 1L?)        "Delete succeeded ~": "Delete Failed ~");        list<string> mget = jedis.mget ("name", "Age");  System.out.println (Mget);      Hash System.out.println ("**************hash**************");        map<string, string> person = jedis.hgetall ("person");  Keyset//entryset for (map.entry<string, string> me:person.entrySet ()) {String field =            Me.getkey ();            String value = Me.getvalue ();        System.out.println (field + "---" + value);        }//list System.out.println ("**************list**************");        list<string> seasons = Jedis.lrange ("season", 0,-1);        for (String season:seasons) {System.out.println (season);        }//set System.out.println ("**************set**************");        set<string> NoSQL = jedis.smembers ("NoSQL");        for (String db:nosql) {System.out.println (db);        }//zset System.out.println ("**************zset**************");        set<string> website = jedis.zrange ("website", 0,-1); for (String ws:wEbsite) {System.out.println (WS);    }} @After public void CleanUp () {jedis.close (); }}

Development of Jedispool Serialization tool class

The previous code is to establish a Jedis connection each time, so that the consumption of resources, you can use Jedispool to solve this problem, and in order to improve the later development efficiency, can be based on jedispool to develop a tool class.

Jedisutil.java

Package Com.uplooking.bigdata.common.util.redis;import com.uplooking.bigdata.constants.redis.JedisConstants; Import Redis.clients.jedis.jedis;import Redis.clients.jedis.jedispool;import redis.clients.jedis.JedisPoolConfig; Import Java.io.ioexception;import java.util.properties;/** * Redis Java API Operation Tool class * mainly for us to provide Java Operations Redis object Jedis imitate a similar database    Connection Pool * * Jedispool */public class Jedisutil {private Jedisutil () {} private static Jedispool Jedispool;        Static {Properties prop = new Properties ();            try {prop.load (JedisUtil.class.getClassLoader (). getResourceAsStream ("redis/redis.properties"));            Jedispoolconfig poolconfig = new Jedispoolconfig ();            The maximum number of connections in the Jedis connection pool poolconfig.setmaxtotal (integer.valueof (Prop.getproperty (jedisconstants.jedis_max_total))); Jedis the maximum number of idle connections in the connection pool Poolconfig.setmaxidle (integer.valueof (Prop.getproperty (Jedisconstants.jedis_max            (_idle))); Jedis the minimum number of idle connections in the connection pool poolconfIg.setminidle (Integer.valueof (Prop.getproperty (Jedisconstants.jedis_min_idle))); Jedis Connection Pool Maximum Wait connection time MS value Poolconfig.setmaxwaitmillis (long.valueof (Prop.getproperty (jedisconstants.jedis_max_wait            (_millis)));            Represents the server hostname of Jedis, String host = Prop.getproperty (Jedisconstants.jedis_host);            String jedis_port = "Jedis.port";            int port = integer.valueof (Prop.getproperty (Jedisconstants.jedis_port));            The service password that represents Jedis String password = prop.getproperty (Jedisconstants.jedis_password);        Jedispool = new Jedispool (poolconfig, host, Port, 10000);        } catch (IOException e) {e.printstacktrace (); }}/** * provides Jedis object * @return * */public static Jedis Getjedis () {return jedispool.getresour    CE ();    }/** * Resource release * @param Jedis */public static void Returnjedis (Jedis Jedis) {jedis.close (); }}

Jedisconstants.java

package com.uplooking.bigdata.constants.redis;/** * 专门用于存放Jedis的常量类 */public interface JedisConstants {    //表示jedis的服务器主机名    String JEDIS_HOST = "jedis.host";    //表示jedis的服务的端口    String JEDIS_PORT = "jedis.port";    //表示jedis的服务密码    String JEDIS_PASSWORD = "jedis.password";    //jedis连接池中最大的连接个数    String JEDIS_MAX_TOTAL = "jedis.max.total";    //jedis连接池中最大的空闲连接个数    String JEDIS_MAX_IDLE = "jedis.max.idle";    //jedis连接池中最小的空闲连接个数    String JEDIS_MIN_IDLE = "jedis.min.idle";     //jedis连接池最大的等待连接时间 ms值    String JEDIS_MAX_WAIT_MILLIS = "jedis.max.wait.millis";}

Redis.properties

################################################ redis的配置文件################################################表示jedis的服务器主机名jedis.host=uplooking01###表示jedis的服务的端口jedis.port=6379###表示jedis的服务密码jedis.password=uplooking###jedis连接池中最大的连接个数jedis.max.total=60###jedis连接池中最大的空闲连接个数jedis.max.idle=30###jedis连接池中最小的空闲连接个数jedis.min.idle=5###jedis连接池最大的等待连接时间 ms值jedis.max.wait.millis=30000

This tool class can be easily used for REDIS operations in the following way:

// 获得Jedis连接对象Jedis jedis = JedisUtil.getJedis();// 释放Jedis对象资源JedisUtil.returnJedis(jedis);

Redis distributed cluster environment to build Redis cluster theory knowledge

   Redis集群是一个分布式Redis存储架构，可以在多个节点之间进行数据共享，解决Redis高可用、可扩展等问题。   Redis集群提供了以下两个好处   1.将数据自动切分(split)到多个节点   2.当集群中的某一个节点故障时，redis还可继续处理客户端的请求   一个Redis集群包含16384个哈希槽(hash slot)，数据库中的每个数据都属于这16384个哈希槽中的一个。   集群使用公式CRC16(key)%16384来计算key属于哪一个槽。集群中的每一个节点负责处理一部分哈希槽。集群中的主从复制   集群中的每个节点都有1个到N个复制品，其中一个为主节点，其余为从节点，如果主节点下线了，   集群就会把这个主节点的一个从节点设置为新的主节点，继续工作。这个集群就不会因为一个主节点的下线而无法正常工作。   如果某一个主节点和它所有的从节点都下线的话，redis集群就停止工作了。   Redis集群不保证数据的强一致性，在特定的情况下，redis集群会丢失已经执行过的命令。   使用异步复制(asynchronous replication)是Redis集群可能会丢失写命令的其中一个原因，   有时候由于网络原因，如果网络断开时间太长，redis集群就会启用新的主节点，之前发给主节点的数据聚会丢失。

The above theoretical knowledge, in the completion of the following Reids master-slave replication environment and distributed environment after the building, I believe there will be very intuitive understanding.

Redis Master-slave replication cluster installation

Three devices are used here, the environment is described as follows:

uplooking01 masteruplooking02 slaveuplooking03 slave即uplooking01为主从节点，02和03为从节点，主从节点主要负责写，从节点主要负责读，不能写。另外在两台从服务器上还会配置使用密码，测试一下使用密码时的连接方式（注意主服务器没有设置密码）。下面的配置会对这些需求有所体现。

The UPLOOKING01 redis.conf is configured as follows:

bind uplooking01daemonize yes（后台运行）logfile /opt/redis-3.2.0/logs/redis.log(日志文件，目录必须存在)

The UPLOOKING02 redis.conf is configured as follows:

bind uplooking02daemonize yes（后台运行）logfile /opt/redis-3.2.0/logs/redis.log(日志文件，目录必须存在)slave-read-only yesrequirepass uplookingslaveof uplooking01 6379

The UPLOOKING03 redis.conf is configured as follows:

bind uplooking03daemonize yes（后台运行）logfile /opt/redis-3.2.0/logs/redis.log(日志文件，目录必须存在)slave-read-only yesrequirepass uplookingslaveof uplooking01 6379

Once the above configuration is complete, the configuration of the master-slave replication cluster is completed by starting Redis on each of the two slave servers, with the following issues to note:

1.认证的问题    如果需要连接uplooking02或者uplooking03，那么需要加上密码，否则无法完成认证。    有两种方式：        可以在连接时就指定密码：redis-cli -h uplooking03 -a uplooking        也可以先连接，到终端后再认证：auth uplooking2.数据读写的问题    读：三台服务器上都能完成数据的读    写：只能在主节点上完成数据的写入3.Java API的使用问题    在前面使用的代码中，如果连接的是从服务器，则还需要配置密码    以我们开发的JedisPool工具类为例，在创建JedisPool时需要指定密码：    jedisPool = new JedisPool(poolConfig, host, port, 10000, password);

Redis Distributed Installation Deployment

Cluster Description:

1.前面搭建的只是主从复制的集群，这意味着，数据在三台机器上都是一样的，其目的只是为了读写分离，提高读的效率    同时也可以起到冗余的作用，主节点一旦出现故障，从节点可以替换，但显然，这只是集群，而不是分布式。2.但是可能会出现一个问题，就是当数据量过大时，所有的数据都保存在同一个节点上    (虽然两台做了备份，但因为保存的数据都是一样的，所以看做一个节点)，    单台服务器的数据存储压力会很大，因此，可以考虑使用分布式的环境来保存，这就是Redis的分布式集群。    分布式：数据分成几份保存在不同的设备上    集群：对于相同的数据，都会有至少一个副本进行保存。    这可以类比hadoop中的hdfs或者是kafka中的partition(topic可以设置partition数量和副本因子)3.在Redis中，搭建分布式集群环境至少需要6个节点，因此出于设备的考虑，这里会在同一台设备上操作    也就是说，这里搭建的是伪分布式环境，3个为主节点，另外3个分别为其从节点，用来保存其副本数据。    根据前面的理论知识，在分布式环境中，key值会进行如下的计算：        CRC16(16) % 16384    来计算key值属于哪一个槽，而对于我们的环境，每个主节点的槽位数量大概是16384 / 3 = 5461

1. Unzip the installation package

[[email protected] ~]$ mkdir -p app/redis-cluster[[email protected] ~]$ tar -zxvf soft/redis-3.2.0.tar.gz -C app/redis-cluster/

2. Compiling the installation

[[email protected] ~]$ cd app/redis-cluster/redis-3.2.0/[[email protected] redis-3.2.0]$ pwd/home/uplooking/app/redis-cluster/redis-3.2.0[[email protected] redis-3.2.0]$ make[[email protected] redis-3.2.0]$ make install PREFIX=/home/uplooking/app/redis-cluster/redis-3.2.0

3. Create a Redis node

[[email protected] redis-cluster]$ mv redis-3.2.0/ 7000[[email protected] redis-cluster]$ cp -r 7000 7001[[email protected] redis-cluster]$ cp -r 7000 7002[[email protected] redis-cluster]$ cp -r 7000 7003[[email protected] redis-cluster]$ cp -r 7000 7004[[email protected] redis-cluster]$ cp -r 7000 7005

4. Modify the configuration of each node

Take 7000 as an example:

daemonize   yes                             //配置redis后台运行bind    uplooking01                         //绑定主机uplooking01logfile "/home/uplooking/app/redis-cluster/7000/redis-7000.log"     //注意目录要存在pidfile /var/run/redis-7000.pid             //pidfile文件对应7000,7002,7003port    7000                                //端口cluster-enabled yes                         //开启集群  把注释#去掉cluster-config-file nodes-7000.conf         //集群的配置  配置文件首次启动自动生成cluster-node-timeout    15000               //请求超时  设置15秒够了appendonly  yes                             //aof日志开启  有需要就开启，它会每次写操作都记录一条日志

On other nodes, just change to 7001,7002 ... Can.

Tip: After the configuration completes 7000, it can be copied directly to the other node, CP redis.conf. /7001, then make full use of vim in 1, $s///g to replace 7000 with other numbers, such as 7001.

5. Start each node

Create a batch-initiated script first:

[[email protected] redis-cluster]$ cat start-all.sh#!/bin/bashcd 7000bin/redis-server ./redis.confcd ..cd 7001bin/redis-server ./redis.confcd ..cd 7002bin/redis-server ./redis.confcd ..cd 7003bin/redis-server ./redis.confcd ..cd 7004bin/redis-server ./redis.confcd ..cd 7005bin/redis-server ./redis.confcd ..

Then execute the script to start.

6. View the service

[[email protected] redis-cluster]$ ps -ef | grep redis500       1460     1  0 01:17 ?        00:00:01 bin/redis-server uplooking01:7000 [cluster]500       1464     1  0 01:17 ?        00:00:01 bin/redis-server uplooking01:7001 [cluster]500       1468     1  0 01:17 ?        00:00:01 bin/redis-server uplooking01:7002 [cluster]500       1472     1  0 01:17 ?        00:00:01 bin/redis-server uplooking01:7003 [cluster]500       1474     1  0 01:17 ?        00:00:01 bin/redis-server uplooking01:7004 [cluster]500       1480     1  0 01:17 ?        00:00:01 bin/redis-server uplooking01:7005 [cluster]500       3233  1018  0 01:53 pts/0    00:00:00 grep redis

7. Create a cluster (CORE)

Now it's time to use the previously prepared Redis node to concatenate it together to build the cluster. The official provides a tool: Redis-trib.rb ($REDIS _home/src a program written using Ruby, so you need to install Ruby):

Then use the Gem This command to install the Redis interface (GEM is a toolkit for Ruby):

gem install redis [ -v 3.2.0] #[]中为可选项制定具体的软件版本# 在我安装时，提示ruby版本需要>=2.2.2，但是上面接上redis接口的版本后就没有问题了。

Next Run the REDIS-TRIB.RB:

[[email protected] 7000]$ src/redis-trib.rb create--replicas 1 192.168.56.101:7000 192.168.56.101:7001 192.168.56.101:7002 192.168.56.101:7003 192.168.56.101:7004 192.168.56.101:7005>>> Creating cluster> >> performing hash slots allocation on 6 nodes ... Using 3 masters:192.168.56.101:7000192.168.56.101:7001192.168.56.101:7002adding Replica 192.168.56.101:7003 to 192.168.56.101:7000adding replica 192.168.56.101:7004 to 192.168.56.101:7001adding replica 192.168.56.101:7005 to 192.168.56.101:7002m:497bce5118057198afb0511cc7b88479bb0c3938 192.168.56.101:7000 slots:0-5460 (5461 slots) MasterM : F0568474acad5c707f25843add2d68455d2cbbb2 192.168.56.101:7001 slots:5461-10922 (5462 slots) MasterM: EBE86EA74AF5612E6393C8E5C5B3363928A4B7B2 192.168.56.101:7002 slots:10923-16383 (5461 slots) MasterS: C99C55AB3FCEA2D65CA3BE5B4786390A6E463EA2 192.168.56.101:7003 replicates 497bce5118057198afb0511cc7b88479bb0c3938s : 0a847801493a45d32487d701cd0fe37790d4b2f9 192.168.56.101:7004 replicates F0568474ACAD5C707F25843ADD2D68455D2CBBB2S:7F9E4BEC579FDA23A574A62D362A04463140BBC2 192.168.56.101:7005 replicates Ebe86ea74af5612e6393c8e5c5b3363928a4b7b2can I set the above configuration? (Type ' yes ' to accept): yes>>> Nodes configuration updated>>> Assign a different config epoch to each no De>>> sending CLUSTER MEET messages to join the clusterwaiting for the CLUSTER to join......>>> Performi ng Cluster Check (using node 192.168.56.101:7000) m:497bce5118057198afb0511cc7b88479bb0c3938 192.168.56.101:7000 Slots : 0-5460 (5461 slots) masterm:f0568474acad5c707f25843add2d68455d2cbbb2 192.168.56.101:7001 slots:5461-10922 (5462 Slots) masterm:ebe86ea74af5612e6393c8e5c5b3363928a4b7b2 192.168.56.101:7002 slots:10923-16383 (5461 slots) MasterM: C99C55AB3FCEA2D65CA3BE5B4786390A6E463EA2 192.168.56.101:7003 Slots: (0 slots) Master replicates 497bce5118057198afb05 11cc7b88479bb0c3938m:0a847801493a45d32487d701cd0fe37790d4b2f9 192.168.56.101:7004 Slots: (0 slots) Master replicates f0568474acad5c707f25843add2d68455d2cbbb2m:7f9e4bec579fda23a574a62d362 A04463140BBC2 192.168.56.101:7005 Slots: (0 slots) master replicates Ebe86ea74af5612e6393c8e5c5b3363928a4b7b2[ok] All Nodes agree about slots configuration.>>> Check for open slots...>>> Check slots coverage ... [OK] All 16384 slots covered.

A closer look at its hints will have a clearer understanding of the Redis distributed cluster. It is also important to note that because REDIS-TRIB.RB does not support the domain name or hostname, it is necessary to use Ip:port when creating the cluster.

8. Testing

[[email protected] 7000]$ bin/redis-cli -h uplooking01 -p 7000 -cuplooking01:7000> set name xpleaf-> Redirected to slot [5798] located at 192.168.56.101:7001OK192.168.56.101:7001> get name"xpleaf"192.168.56.101:7001>[[email protected] 7000]$ bin/redis-cli -h uplooking01 -p 7004 -cuplooking01:7004> get name-> Redirected to slot [5798] located at 192.168.56.101:7001"xpleaf"192.168.56.101:7001>[[email protected] 7000]$ bin/redis-cli -h uplooking01 -p 7004 -cuplooking01:7004> set name yyh-> Redirected to slot [5798] located at 192.168.56.101:7001OK192.168.56.101:7001> get name"yyh"[[email protected] 7000]$ bin/redis-cli -h uplooking01 -p 7002 -cuplooking01:7002> keys *(empty list or set)uplooking01:7002> get name-> Redirected to slot [5798] located at 192.168.56.101:7001"yyh"

The above tests can adequately illustrate the following issues:

1.分布式    数据是分布式存储的，根据key的不同会保存到不同的主节点上。2.数据备份    从节点是作为备份节点的，跟前面的主从复制集群一样，只是用来读数据，当需要写或修改数据时，需要切换到主节点上。

Redis Java API use (ii): Cluster API Usage

The previous code is only suitable for operating a single version of Redis, if you are using a distributed Redis cluster, then you need to modify the code, here, we directly develop a tool class JedisClusterUtil , as follows:

Package Com.uplooking.bigdata.common.util.redis;import Redis.clients.jedis.*;import Java.io.ioexception;import Java.util.hashset;import java.util.properties;import java.util.set;/** * Redis Java API Operations Tool class * specifically responsible for Redis's cluster mode */p    Ublic class Jedisclusterutil {private Jedisclusterutil () {} private static jediscluster jediscluster;        static {set

It is important to note that when using API operations such as Mget, Jediscluster does not allow data to be fetched on multiple nodes at the same time, such as:list<string> mget = Jedis.mget ("name", "Age"); If name and age are on separate nodes, the exception is reported, so it is not recommended to use this method to get the data.

Redis Note-taking (ii): Java API usage and Redis distributed cluster environment setup