Redis Proxy Service Twemproxy

1, Twemproxy explore

when we have a large number of Redis or Memcached, we usually only pass some data distribution algorithms (such as a consistent hash) of the client, to realize the characteristics of cluster storage. Although redis Version 2.6 has been released for Redis Cluster, but it is not yet mature for a formal production environment.  redis's Cluster scheme has not yet been formally launched, we The cluster storage.

Twitter, one of the world's largest redis clusters, is deployed on Twitter to provide users with timeline data. The Twitter Open Source department provides Twemproxy.

Twemproxy, also known as Nutcraker. is a twtter open source Redis and Memcache proxy server. As an efficient cache server, Redis is of great application value. But when used more often, you want to be able to manage it in some way. Avoid the looseness of each client management connection for each application. At the same time, it becomes controllable to some extent.

Twemproxy is a fast single-threaded agent that supports the Memcached ASCII protocol and the updated Redis protocol:

It is all written in C and is licensed using Apache 2.0 license. The project can work on Linux and cannot be compiled on OSX because it relies on the Epoll API.

Twemproxy by introducing a proxy layer, multiple Redis or Memcached instances from the backend can be managed and distributed uniformly, so that the application only needs to operate on twemproxy, regardless of the number of actual Redis or Memcached that follow. Store.  

2,twemproxy characteristics:

• Supports automatic deletion of failed nodes

  • You can set the time to reconnect the node
  • You can set how many times to connect and then delete the node
  • This method is suitable as a cache storage

• Support Settings hashtag

  • By hashtag you can set up two keyhash to the same instance.

• Reduce the number of direct connections to Redis

  • Maintain a long connection to Redis
  • The number of each Redis connection can be set for the agent and background

• Auto-Shard to back-end on multiple Redis instances

  • Multiple hash algorithms: the ability to support consistent hashing using different strategies and hash functions.
  • You can set weights for back-end instances

• Avoid a single point of problem

  • Multiple agent tiers can be deployed in parallel. Client automatically selects an available

• Support for Redis pipelining request

  Supports streaming and batching of requests, reducing the consumption of back and forth
  

• Support for status monitoring

  • State monitoring IP and port can be set, Access IP and port can get a JSON-formatted status information string
  • can set monitoring information refresh interval time

• High throughput

  • Connection multiplexing, memory reuse.
  • Multiple connection requests, composed of Reids pipelining, are unified to redis requests.

Alternatively, you can modify the source code of Redis to extract the first half of Redis as an intermediary agent layer. Ultimately, the concurrency efficiency is improved through the Epoll event mechanism under Linux, where Nutcraker itself is the event mechanism using Epoll. And the performance tests performed very well.

3, Twemproxy problems and deficiencies

Twemproxy due to its own principle limitations, there are some shortcomings, such as:

• Also does not support select operations

 specific installation steps are available to view Github:https://github.com/twitter/twemproxyTwemproxy installation, the main command is as follows:

1. Apt-get Install Automake
2. Apt-get Install Libtool
3. git clone git://github.com/twitter/twemproxy.git
4. CD Twemproxy
5. Autoreconf-fvi
6. ./configure --enable-debug=log
7. Make
8. Src/nutcracker-h

The above command, even if installed, and then the specific configuration, below is a typical configuration

• listen:127.0.0.1:6379 #使用哪个端口启动Twemproxy
• Redis:true #是否是Redis的proxy
• Hash:fnv1a_64 #指定具体的hash函数
• Distribution:ketama #具体的hash算法
• Auto_eject_hosts:true #是否在结点无法响应的时候临时摘除结点
• timeout:400 #超时时间 (MS)
• server_retry_timeout:2000 #重试的时间 (MS)
• Server_failure_limit:1 #结点故障多少次就算摘除掉
• Servers: #下面表示所有的Redis节点 (IP: Port number: Weight)
• -127.0.0.1:6380:1
• -127.0.0.1:6381:1
• -127.0.0.1:6382:1
• REDIS2:
• listen:0.0.0.0:10000
• Redis:true
• Hash:fnv1a_64
• Distribution:ketama
• Auto_eject_hosts:false
• timeout:400
• Servers
• -127.0.0.1:6379:1
• -127.0.0.1:6380:1
• -127.0.0.1:6381:1
• -127.0.0.1:6382:1

You can open multiple Twemproxy instances at the same time, both of which can read and write, so your application can completely avoid the so-called single point of failure.

Redis Proxy Service Twemproxy