Redis Agent Service Twemproxy

1, Twemproxy Explore

&http://www.aliyun.com/zixun/aggregation/37954.html ">nbsp; When 11545.html "> We have a large number of Redis or Memcached, it is usually possible to implement the cluster storage characteristics only through some client data allocation algorithms (such as a consistent hash). Although the Redis 2.6 version has been released Redis Cluster, it is not very mature to apply to the formal production environment. Redis's Cluster scheme has not yet been formally launched, we use Proxy to achieve cluster storage.

Twitter, one of the world's largest redis clusters, is deployed on Twitter to provide timeline data to users. The Twitter Open Source department provides Twemproxy.

Twemproxy, also known as Nutcraker. is a twtter open source Redis and Memcache proxy server. As an efficient caching server, Redis is of great application value. But when used more, it is hoped that the management can be unified in some way. Avoid the loose nature of each client management connection for each application. At the same time it becomes controllable to some extent.

Twemproxy is a fast single-threaded agent that supports the Memcached ASCII protocol and the updated Redis protocol:

It is all written in C, using the Apache 2.0 license authorization. The project works on Linux and cannot be compiled on OS X because it relies on the Epoll API.

Twemproxy by introducing a proxy layer, multiple Redis or Memcached instances at the back end can be managed and distributed uniformly so that the application only needs to operate on the twemproxy without caring for the number of actual Redis or Memcached Storage。

2, Twemproxy characteristics:

Support failed node automatic deletion

can set the time to reconnect the node can set the number of connections after the deletion of the node this method is suitable for cache storage

Support Settings hashtag

can set up two keyhash to the same instance by hashtag.

Reduce the number of direct connections to Redis

maintain a long connection to the Redis to set the number of Redis connections between the agent and the background

Automatic fragmentation to back-end multiple Redis instances

multiple hash algorithms: the ability to use different policies and hash functions to support a consistent hash. You can set the weight of a backend instance

Avoid a single point of issue

can deploy multiple agent tiers in parallel. Client automatically chooses a usable

Support Redis pipelining Request

Supports streaming and batching of requests, reducing back and forth consumption

Support Status Monitoring

can set state monitoring IP and port, access IP and port can get a JSON format state information string can set monitoring information refresh interval

High throughput

connection multiplexing, memory reuse. Multiple connection requests are composed of Reids pipelining unified to Redis requests.

In addition, the source code of the Redis can be modified to extract the first half of the Redis as an intermediary agent layer. In the end, the Epoll event mechanism under Linux is used to improve concurrency efficiency, in which Nutcraker itself is a epoll event mechanism. and performed well on performance tests.

3, Twemproxy problems and deficiencies

Twemproxy due to its own principle limitations, there are some deficiencies, such as:

does not support operations for multiple values, such as sets disjoint complement (except Mget and DEL) do not support Redis transaction operation error prompts are not perfect or support select operation

4, installation and configuration of specific installation steps can be viewed Github:https://github.com/twitter/twemproxytwemproxy installation, the main commands are as follows: Apt Install Automakeapt-get Install Libtoolgit clone GIT://GITHUB.COM/TWITTER/TWEMPROXY.GITCD twemproxyautoreconf-fvi./configure--enable-debug =logmakesrc/nutcracker-h


through the above command even if installed, and then the specific configuration, the following is a typical configuration redis1:listen:127.0.0.1:6379 #使用哪个端口启动Twemproxy redis:true #是否是Redis的proxy hash:fnv1a_64 #指定具体的hash函数 Distribution:ketama #具体的hash算法 auto_eject_hosts:true #是否在结点无法响应的时候临时摘除结点 timeout:400 Timeout (ms) server_retry_timeout:2000 #重试的时间 (ms) server_failure_limit:1 #结点故障多少次就算摘除掉 servers: #下面表示所有的Redis节点 (IP: Port number: Weight)-127.0.0.1:6380:1-127.0.0.1:6381:1-127.0.0.1:6382:1redis2:listen:0.0.0.0:10000 redis:true hash:fnv1a_64 Distribution:ketama auto_eject_hosts:false timeout:400 Servers:-127.0.0.1:6379:1-127.0.0.1:6380:1- 127.0.0.1:6381:1-127.0.0.1:6382:1


You can open multiple Twemproxy instances at the same time, they can both read and write, so your application can completely avoid the so-called single point of failure.