Redis proxy service twemproxy

1. twemproxy release E

When we have a large number of redis or memcached instances, we can only use some data distribution algorithms (such as consistent hashing) on the client side to implement Cluster Storage features. Although redis
Redis cluster has been released in version 2.6, but it is not very mature and applicable to the formal production environment. Before the redis cluster solution is officially launched, we use
Proxy to achieve Cluster Storage.

Twitter, one of the world's largest redis clusters, is deployed on Twitter to provide timeline data for users. Twitter open source provides twemproxy.

Twemproxy, also known as nutcraker. Is an open-source redis and memcache proxy server of twtter. As an Efficient Cache Server, redis is of great application value. However, when many applications are used, you may wish to manage them in a unified manner. Avoid loose management connections for each client of each application. At the same time, it becomes controllable to a certain extent.

Twemproxy is a fast single-thread agent that supports memcached ASCII protocol and the updated redis protocol:

It is all written in C and authorized using Apache 2.0 license. A project can work on Linux, but cannot be compiled on OSX because it depends on epoll API.

By introducing a proxy layer, twemproxy can centrally manage and allocate multiple backend redis or memcached instances so that the application only needs
Operate on Twemproxy without worrying about the actual Redis or Memcached storage.

2. twemproxy features:

• Support automatic deletion of failed nodes

  • You can set the time for the node to be reconnected.
  • You can set the number of connections to delete the node.
  • This method is suitable for cache storage.

• Support setting HashTag

  • With hashtag, you can set two keyhash entries to the same instance.

• Reduce the number of direct connections with redis

  • Maintain a persistent connection with redis
  • You can set the number of connections between the proxy and each redis in the background.

• Auto-sharding to multiple backend redis instances

  • Multiple hash algorithms: different policies and hash functions can be used to support consistent hash.
  • You can set the weight of the backend instance.

• Avoid single point of failure

  • Multiple proxy layers can be deployed in parallel. The client automatically selects one available

• Support for redis pipelining request

  Supports stream and batch processing of requests to reduce consumption

• Support Status Monitoring

  • You can set the status monitoring IP address and port. The access IP address and port can get a status information string in JSON format.
  • Set monitoring information Refresh Interval

• High Throughput

  • Connection multiplexing and memory multiplexing.
  • Combine multiple connection requests to form reids pipelining and send requests to redis in a unified manner.

In addition, you can modify the redis source code and extract the First Half of redis as an intermediate proxy layer. In the end, the epoll event mechanism in linux is used to improve the concurrency efficiency. Among them, nutcraker also uses the epoll event mechanism. In addition, the performance test is outstanding.

3. twemproxy problems and deficiencies

Due to its own principle limitations, Twemproxy has some shortcomings, such:

• Operations on multiple values are not supported, for example, subgrouping and population of sets (except MGET and DEL)
• Redis transaction operations are not supported
• Error prompt is not complete yet
• The select operation is not supported.

4. installation and configuration

Detailed installation steps available to view GitHub: https://github.com/twitter/twemproxy

The main command for installing Twemproxy is as follows:

apt-get install automakeapt-get install libtoolgit clone git://github.com/twitter/twemproxy.gitcd twemproxyautoreconf -fvi./configure --enable-debug=logmakesrc/nutcracker -h

After the preceding command is installed, the specific configuration is displayed. The following is a typical configuration.

Redis1: listen: 127.0.0.1: 6379 # port used to start Twemproxy redis: true # whether it is Redis's proxy hash: fnv1a_64 # specify the specific hash function distribution: ketama # specific hash algorithm auto_eject_hosts: true # Whether to temporarily remove the node timeout when the node cannot respond: 400 # timeout (milliseconds) server_retry_timeout: 2000 # retry time (milliseconds) server_failure_limit: 1 # How many node faults are removed even if servers: # The following indicates all Redis nodes (IP: Port Number: weight)-127.0.0.1: 6380: 1-127.0.0.1: 6381: 1-127.0.0.1: 6382: 1redis2: listen: 0.0.0.0: 10000 redis: true hash: fnv1a_64 distribution: ketama auto_eject_hosts: false timeout: 400 servers:-127.0.0.1: 6379: 1-127.0.0.1: 6380: 1-127.0.0.1: 6381: 1-127.0.0.1: 6382: 1

You can enable multiple Twemproxy instances at the same time, which can be read and written, so that your application can completely avoid the so-called single point of failure.