Twemproxy-proxy for MemCached and Redis.

At last year's QConLondon2012 conference, Twitter published the article titled Timelines @ Twitter.

At last year's QCon London2012 conference, Twitter gave a speech entitled "Timelines @ Twitter", which mentioned that Redis is the main storage of its timeline. Currently, it is globally oriented, twitter may be the biggest user of Redis (or Sina Weibo ?).

Today, we will talk about the Twemproxy, an open-source Redis and Memcached proxy from Twitter.

Features

We know that both Memcached and current Redis do not have distributed cluster features. when we have a large number of Redis or Memcached instances, generally, you can only use some client data distribution algorithms (such as consistent hash) to implement cluster storage features.

By introducing a proxy layer, Twemproxy can centrally manage and allocate multiple backend Redis or Memcached instances so that the application only needs to operate on Twemproxy, you don't have to worry about the actual Redis or Memcached storage.

Before the Redis Cluster solution is officially launched, it may be the best choice to implement a storage Cluster through Proxy. Besides, Twemproxy is a product that has been fully tested by Twitter itself.

Performance

According to the test results of Redis, in most cases, Twemproxy has a very good performance. compared with direct Redis operations, it has a maximum performance loss of 20%. This is really insignificant for the benefits it brings. The only thing that needs to be improved is the efficiency of MGET operations. its performance is only 50% of the direct operations on Redis.

Installation and configuration

The installation of Twemproxy is a little troublesome. The main commands are as follows:

apt-get install automakeapt-get install libtoolgit clone git://github.com/twitter/twemproxy.gitcd twemproxyautoreconf -fvi./configure --enable-debug=logmakesrc/nutcracker -h

After the preceding command is installed, the specific configuration is displayed. The following is a typical configuration.

Redis1: listen: 0.0.0.0: 9999 # port used to start Twemproxy redis: true # whether it is Redis's proxy hash: fnv1a_64 # specify the specific hash function distribution: ketama # specific hash algorithm auto_eject_hosts: true # whether to temporarily remove the node timeout when the node cannot respond: 400 # timeout (milliseconds) server_retry_timeout: 2000 # retry time (milliseconds) server_failure_limit: 1 # How many node faults are removed even if servers: # The following indicates all Redis nodes (IP: port number: weight)-127.0.0.1: 6379: 1-127.0.0.1: 6380: 1-127.0.0.1: 6381: 1-127.0.0.1: 6382: 1redis2: listen: 0.0.0.0: 10000 redis: true hash: fnv1a_64 distribution: ketama auto_eject_hosts: false timeout: 400 servers:-127.0.0.1: 6379: 1-127.0.0.1: 6380: 1-127.0.0.1: 6381: 1-127.0.0.1: 6382: 1

You can enable multiple Twemproxy instances at the same time, all of which can be read and written, so that your application can completely avoid the so-called single point of failure.

Problems and deficiencies

Due to its own principle limitations, Twemproxy has some shortcomings, such:

• Operations on multiple values are not supported, for example, Subgrouping and population of sets (except MGET and DEL)
• Redis transaction operations are not supported
• Error prompt is not complete yet