Redis proxy service Twemproxy

1. twemproxy release E

When we have a large number of Redis or Memcached instances, we can only use some data distribution algorithms (such as consistent hashing) on the client side to implement Cluster Storage features. Redis Cluster has been released in Redis 2.6, but it is not very mature and applicable to the formal production environment. Before the Redis Cluster solution is officially launched, we use the Proxy method to implement Cluster Storage.

Twitter, one of the world's largest Redis clusters, is deployed on Twitter to provide timeline data for users. Twitter Open Source provides Twemproxy.

Twemproxy, also known as nutcraker. Is an open-source redis and memcache proxy server of twtter. As an Efficient Cache Server, redis is of great application value. However, when many applications are used, you may wish to manage them in a unified manner. Avoid loose management connections for each client of each application. At the same time, it becomes controllable to a certain extent.

Twemproxy is a fast single-thread agent that supports Memcached ASCII protocol and the updated Redis protocol:

It is all written in C and authorized using Apache 2.0 License. A project can work on Linux, but cannot be compiled on OSX because it depends on epoll API.

By introducing a proxy layer, Twemproxy can centrally manage and allocate multiple backend Redis or Memcached instances so that the application only needs to perform operations on Twemproxy, you don't have to worry about the actual Redis or Memcached storage.

2. twemproxy features:

Support automatic deletion of failed nodes

You can set the time for the node to be reconnected.
You can set the number of connections to delete the node.
This method is suitable for cache storage.
Support setting HashTag

With HashTag, you can set two KEYhash entries to the same instance.
Reduce the number of direct connections with redis

Maintain a persistent connection with redis
You can set the number of connections between the proxy and each redis in the background.
Auto-sharding to multiple backend redis instances

Multiple hash algorithms: different policies and hash functions can be used to support consistent hash.
You can set the weight of the backend instance.
Avoid single point of failure

Multiple proxy layers can be deployed in parallel. The client automatically selects one available
Support for redis pipelining request

Supports stream and batch processing of requests to reduce consumption

Support Status Monitoring

You can set the status monitoring ip address and port. The access ip address and port can get a status information string in json format.
Set monitoring information Refresh Interval
High Throughput

Connection multiplexing and memory multiplexing.
Combine multiple connection requests to form reids pipelining and send requests to redis in a unified manner.
In addition, you can modify the redis source code and extract the First Half of redis as an intermediate proxy layer. In the end, the epoll event mechanism in linux is used to improve the concurrency efficiency. Among them, nutcraker also uses the epoll event mechanism. In addition, the performance test is outstanding.

3. twemproxy problems and deficiencies

Due to its own principle limitations, Twemproxy has some shortcomings, such:

Operations on multiple values are not supported, for example, subgrouping and population of sets (except MGET and DEL)
Redis transaction operations are not supported
Error prompt is not complete yet
The select operation is not supported.

4. installation and configuration
Detailed installation steps available to view github: https://github.com/twitter/twemproxy
The main command for installing Twemproxy is as follows:

Apt-get install automake
Apt-get install libtool
Git clone git: // github.com/twitter/twemproxy.git
Cd twemproxy
Autoreconf-fvi
./Configure -- enable-debug = log
Make
Src/nutcracker-h

After the preceding command is installed, the specific configuration is displayed. The following is a typical configuration.

Redis1:
Listen: 127.0.0.1: 6379 # port used to start Twemproxy
Redis: true # whether it is Redis's proxy
Hash: fnv1a_64 # specify a specific hash Function
Distribution: ketama # specific hash Algorithm
Auto_eject_hosts: true # Whether to temporarily remove the node when the node cannot respond
Timeout: 400 # timeout (MS)
Server_retry_timeout: 2000 # retry time (MS)
Server_failure_limit: 1 # The number of node failures is removed.
Servers: # All Redis nodes (IP: Port Number: weight)
-127.0.0.1: 6380: 1
-127.0.0.1: 6381: 1
-127.0.0.1: 6382: 1

Redis2:
Listen: 0.0.0.0: 10000
Redis: true
Hash: fnv1a_64
Distribution: ketama
Auto_eject_hosts: false
Timeout: 400
Servers:
-127.0.0.1: 6379: 1
-127.0.0.1: 6380: 1
-127.0.0.1: 6381: 1
-127.0.0.1: 6382: 1

You can enable multiple Twemproxy instances at the same time, which can be read and written, so that your application can completely avoid the so-called single point of failure.

Twemproxy: Redis proxy service released by Twitter

Install Redis in Ubuntu 12.10 (graphic explanation) + Jedis to connect to Redis

Redis series-installation, deployment, and maintenance

Install Redis in CentOS 6.3

Redis details: click here
Redis: click here