Redis Agent Service Twemproxy

1, Twemproxy explore

When we have a large number of Redis or Memcached, it is usually only through some data allocation algorithms (such as a consistent hash) of the client that the characteristics of the cluster storage can be realized. Although the Redis version 2.6 has been released Redis Cluster, it is not mature enough to apply to the formal production environment. Redis's Cluster scheme has not yet been formally launched, we use Proxy to implement the cluster storage.

Twitter, one of the world's largest redis clusters, is deployed on Twitter to provide timeline data to users. The Twitter Open Source department provides Twemproxy.

Twemproxy, also known as Nutcraker. is a twtter open source of a Redis and Memcache proxy server. As an efficient caching server, Redis is of great application value. But when used more, it is hoped that the management can be unified in some way. Avoid the loose nature of each client management connection for each application. At the same time it becomes controllable to some extent.

Twemproxy is a fast single-threaded agent that supports the Memcached ASCII protocol and the updated Redis protocol:

It is all written in C, using the Apache 2.0 license authorization. The project works on Linux and cannot be compiled on OS X because it relies on the Epoll API.

Twemproxy by introducing a proxy layer, multiple Redis or Memcached instances at the back end can be managed and distributed uniformly so that the application only needs to operate on the twemproxy without caring for the number of actual Redis or Memcached Store.

2, twemproxy characteristics:

Support for failed node auto-deletion You can set the time to reconnect the node can set the number of connections after the deletion of the node this method is suitable for cache storage

Support settings hashtag can set up two keyhash to the same instance by hashtag.

Reduce the number of direct connections to Redis and keep Redis long connections to set the number of agents and backgrounds per Redis connection

Automatic fragmentation to multiple Redis instances on the back end: the ability to use different policies and hash functions to support a consistent hash. You can set the weight of a backend instance

Avoid a single point of problem to deploy multiple agent tiers in parallel. Client automatically chooses one of the available

Support Redis pipelining Request

Supports streaming and batching of requests, reducing back and forth consumption

Support State monitoring can set state monitoring IP and port, access IP and port can get a JSON format state information string can set monitoring information refresh interval time

High throughput connection multiplexing, memory reuse. Multiple connection requests are composed of Reids pipelining unified to Redis requests.

In addition, the source code of the Redis can be modified to extract the first half of the Redis as an intermediary agent layer. In the end, the Epoll event mechanism under Linux is used to improve concurrency efficiency, in which Nutcraker itself is a epoll event mechanism. and performed well on performance tests.

3, Twemproxy problems and deficiencies
Twemproxy due to its own principle limitations, there are some deficiencies, such as:
Operations for multiple values are not supported, such as sets orthogonal complement (Mget and DEL) do not support Redis transaction operation error prompts are not perfect or support select operations

4, installation and configuration of the specific installation steps available to view the installation of Github:https://github.com/twitter/twemproxy Twemproxy, the main commands are as follows:

Apt-get Install automake
apt-get install libtool
git clone git://github.com/twitter/twemproxy.git
CD Twemproxy
autoreconf-fvi
./configure--enable-debug=log
make
src/nutcracker-h

With the above command installed, and then the specific configuration, the following is a typical configuration

REDIS1:
  listen:127.0.0.1:6379 #使用哪个端口启动Twemproxy
  redis:true #是否是Redis的proxy
  hash:fnv1a_64 # Specify a specific hash function
  Distribution:ketama #具体的hash算法
  auto_eject_hosts:true #是否在结点无法响应的时候临时摘除结点
  timeout:400 # Timeout (ms)
  server_retry_timeout:2000 #重试的时间 (ms)
  server_failure_limit:1 #结点故障多少次就算摘除掉
  servers: # The following represents all Redis nodes (IP: Port number: Weight)
   -127.0.0.1:6380:1
   -127.0.0.1:6381:1
   -127.0.0.1:6382:1

Redis2:
  listen:0.0.0.0:10000
  redis:true
  hash:fnv1a_64
  distribution:ketama
  auto_eject_hosts: False
  timeout:400
  servers:
   -127.0.0.1:6379:1
   -127.0.0.1:6380:1
   -127.0.0.1:6381:1
   -127.0.0.1:6382:1

You can open multiple Twemproxy instances at the same time, both of which can be read and written so that your application can completely avoid the so-called single point of failure.