Ubuntu 15.10 under Scala operation Redis Cluster

1 Preface

The installation of Redis Standalone,redis cluster is described earlier, address: http://www.cnblogs.com/liuchangchun/p/5063477.html, here is not introduced.

2 Use of Redis

For a variety of programming languages, almost all have redis drivers. For Scala programs, there are several available drivers on GitHub:

Scala-redis:https://github.com/liuchchc/scala-redis

Rediscala:https://github.com/etaty/rediscala

Jedis:https://github.com/liuchchc/jedis

I use is Jedis, feel still good use, support cluster

2.1 Jedissentinelpool Way

In standalone mode, cluster provides sentinel to monitor the operation status of Redis server and elect master, which requires configuration of sentinel.conf in this mode, which is used in Jedis as follows

Val sentinelclustername = "MyMaster"Val sentinelserverhosths=Newhashset[string] Sentinelserverhosths.add ("192.168.1.103:26379") Sentinelserverhosths.add ("192.168.1.104:26379") Sentinelserverhosths.add ("192.168.1.105:26379") Sentinelserverhosths.add ("192.168.1.106:26379") Sentinelserverhosths.add ("192.168.1.107:26379") Sentinelserverhosths.add ("192.168.1.108:26379")    //If the assignment is-1, it is unrestricted, and if the pool is already assigned maxactive Jedis instances, the pool has a status of exhausted (exhausted) and the default value is 8. Val max_active =-1; //controls the maximum number of Jedis instances in a pool that have an idle (idle) state, and the default value is 8. Val Max_idle =-1; //The maximum time to wait for an available connection, in milliseconds, and the default value is-1, which means that never times out. If the waiting time is exceeded, the jedisconnectionexception is thrown directly;Val max_wait =-1; //Timeout, 0 never time outVal TIMEOUT = 0; var poolconfig=Newgenericobjectpoolconfig poolconfig.setmaxtotal (max_active) poolconfig.setmaxidle (MAX_IDLE) Poolconfig.setmaxwaitmillis (max_wait) Poolconfig.settestonborrow (true) Poolconfig.settestonreturn (true) def Getpoolconfig:genericobjectpoolconfig= Poolconfig
Lazy val jedissentinelpool = new Jedissentinelpool ("MyMaster", Sentinelserverhosths, Poolconfig, TIMEOUT)
Lazy val sentinelconnection = Jedisshardedpool.getresource
Redis operations ...
Redis Operation complete
Sentinelconnection.close
Jedissentinelpool.destroy

2.2 Shardedjedispool Way

Val shardsserverhostlist =NewArraylist[jedisshardinfo] Val si1=NewJedisshardinfo ("192.168.1.103", 6379) Val Si2=NewJedisshardinfo ("192.168.1.104", 6379) Val Si3=NewJedisshardinfo ("192.168.1.105", 6379) Val Si4=NewJedisshardinfo ("192.168.1.106", 6379) Val si5=NewJedisshardinfo ("192.168.1.107", 6379) Val si6=NewJedisshardinfo ("192.168.1.108", 6379) Shardsserverhostlist.add (SI1) shardsserverhostlist.add (SI2) shardsserverhostlist.add (SI3) Shardsserverhostlist.add (SI4) shardsserverhostlist.add (si5) shardsserverhostlist.add (si6) lazy Val Jedisshardedpool =NewShardedjedispool (poolconfig, shardsserverhostlist) lazy Val shardedconnection= Jedisshardedpool.getresource

  Redis operations ...

  Redis Operation complete

  Shardedconnection.close

  Jedisshardedpool.destroy

2.3 Jediscluster Way

In Redis cluster mode, I have 3 master,6 slave,hostandport only 1, and Redis detects which master and slave are in the cluster. As to which cluster the data is stored in when the data is added, the user level does not need to be concerned and the data is read as well.

Val jedisclusternodes =NewJava.util.hashset[hostandport] () Jedisclusternodes.add (NewHostandport ("192.168.1.100", 6379)) Jedisclusternodes.add (NewHostandport ("192.168.1.101", 6379)) Jedisclusternodes.add (NewHostandport ("192.168.1.103", 6379)) Jedisclusternodes.add (NewHostandport ("192.168.1.104", 6379)) Jedisclusternodes.add (NewHostandport ("192.168.1.105", 6379)) Jedisclusternodes.add (NewHostandport ("192.168.1.106", 6379)) Jedisclusternodes.add (NewHostandport ("192.168.1.107", 6379)) Jedisclusternodes.add (NewHostandport ("192.168.1.108", 6379)) Jedisclusternodes.add (NewHostandport ("192.168.1.109", 6379))  //Redis Operations
Lazy val jediscluster = new Jediscluster (jedisclusternodes)

3 Usage Scenarios

Here is the spark processing biological data, because the intermediate results are more, so added a Redis server storage intermediate results, a total of more than 100 million, about 3.5G, probably in minutes can be stored in, so redis performance or leverage drops, but to bulk write, read, otherwise it will be slow.

4 References: 4.1 http://my.oschina.net/zhuguowei/blog/4110774.2 http://ju.outofmemory.cn/entry/141317

Ubuntu 15.10 under Scala operation Redis Cluster