Using pipeline Bulk Insert in a Redis cluster

Because of the need to use the Bulk Insert feature in the project, it was found on the web that Redis Bulk Insert can be efficiently inserted using pipeline, the sample code is as follows:

String key = "key";
Jedis jedis = new Jedis ("xx.xx.xx.xx");
Pipeline p = jedis.pipelined ();
List <String> myData = .... // List of data to insert
for (String data: myData) {
     p.hset (key, data);
}
p.sync ();
jedis.close ();

However, the problem is that the Redis used on the project is a cluster, and the class used when initializing is jediscluster instead of Jedis. Checked the Jediscluster documentation and found no way to provide a pipelined () method to get pipeline objects like Jedis.

Google a bit and found a solution.

The Redis cluster specification says: The key space of a Redis cluster is divided into16384slots (slots), and the maximum number of nodes in a cluster is also16384one. Each master node is responsible for processing16384part of a hash slot. When we say that a cluster is in a "stable" state, it means that the cluster is not performing a reconfiguration (reconfiguration) operation, and each hash slot is processed by only one node.

So we can know the number of the slot corresponding to the key according to the key to be inserted, and then find the corresponding Jedis from the cluster by the number of the slot. The specific implementation is as follows

// Initialize the jedis cluster, how to get the HostAndPort collection code is not written

Set <HostAndPort> nodes = .....

JedisCluster jedisCluster = new JedisCluster (nodes);



Map <String, JedisPool> nodeMap = jedisCluster.getClusterNodes ();

String anyHost = nodeMap.keySet (). Iterator (). Next ();

// getSlotHostMap method has below

TreeMap <Long, String> slotHostMap = getSlotHostMap (anyHost); 

private static TreeMap <Long, String> getSlotHostMap (String anyHostAndPortStr) {
         TreeMap <Long, String> tree = new TreeMap <Long, String> ();
         String parts [] = anyHostAndPortStr.split (":");
         HostAndPort anyHostAndPort = new HostAndPort (parts [0], Integer.parseInt (parts [1]));
         try {
             Jedis jedis = new Jedis (anyHostAndPort.getHost (), anyHostAndPort.getPort ());
             List <Object> list = jedis.clusterSlots ();
             for (Object object: list) {
                 List <Object> list1 = (List <Object>) object;
                 List <Object> master = (List <Object>) list1.get (2);
                 String hostAndPort = new String ((byte []) master.get (0)) + ":" + master.get (1);
                 tree.put ((Long) list1.get (0), hostAndPort);
                 tree.put ((Long) list1.get (1), hostAndPort);
             }
             jedis.close ();
         } catch (Exception e) {
            
         }
         return tree;
     }

The above steps can be done at initialization time. Do not need to be called every time, Nodemap and slothostmap are defined as static variables.

// Get slot number

int slot = JedisClusterCRC16.getSlot (key);

// Get the corresponding Jedis object

Map.Entry <Long, String> entry = slotHostMap.lowerEntry (Long.valueOf (slot));

Jedis jedis = nodeMap.get (entry.getValue ()). GetResource (); 

It is recommended that this step be encapsulated as a static method, such as the public static Jedis Getjedisbykey (String key). This means that in the cluster, the Jedis object corresponding to this key is obtained through key.

This is done through the above jedis.pipelined () and can be bulk inserted.

Note: This method was searched from Google, until now I haven't found any problems with it. If any of the great gods find something wrong, welcome the offer.

Using pipeline Bulk Insert in a Redis cluster