Twemproxy Introduction and use

Twemproxy is a proxy shard mechanism, open source by Twitter. Twemproxy as an agent, can accept access from multiple programs, according to routing rules, forwarded to the background of the various Redis servers, and then the original path back. This solution solves the problem of single Redis instance carrying capacity. Of course, Twemproxy itself is a single point, need to use keepalived to do high-availability programs. The twemproxy can be used to expand the Redis service horizontally by using multiple servers, which can effectively avoid the problem of single point of failure. While the use of twemproxy requires more hardware resources and a certain amount of loss in Redis performance (about 20% of the Twitter test), it is also fairly cost-effective to improve the overall system's ha. Unfamiliar Twemproxy classmate, if played Nginx reverse proxy or MySQL proxy, then you must also understand twemproxy. In fact, twemproxy not only realized the Redis protocol, but also realized the memcached protocol, what does it mean? In other words, twemproxy can not only proxy redis, but also proxy memcached, official note:

twemproxy   (pronounced "two-em-proxy"), Aka  Nutcracker  is A Fast and Lightweight proxy for memcachedand  redis protocol. it was built primarily to reduce the number of connections to the caching servers on the back End.  this, together with protocol pipeling and sharding enables your to horizontally Scale your distributed caching architecture.

Twemproxy Architecture:

But from the above we can see that since Twemproxy has become a single point, it is often combined with keepalived to achieve twemproxy high availability. The architecture diagram is as follows:

The above architecture usually only one twemproxy in the work, the other one in the standby machine, when one hangs, the VIP automatically drift, standby machine replacement work. Information about keepalived can be found in the relevant articles in front of me.

1. Compile and install:

Autoconf:http://ftp.gnu.org/gnu/autoconf/autoconf-2.69.tar.gz
Twemproxy:https://codeload.github.com/twitter/twemproxy/zip/master
Twemproxy installation requires Autoconf version above 2.64, otherwise prompt "error:autoconf version 2.64 or higher is required".

Find old version autoconf, and uninstall

rpm-qf/usr/bin/autoconf  -e--nodeps autoconf-2.63   

Install the latest version

tar zxvf autoconf-2.69. Tar . GZ CD autoconf-2.69  . /configure--prefix=/make doinstall

Compiling and installing Twemproxy

Unzip Twemproxy-master. Zip CD twemproxy--fvi. /configure--prefix=/usr/local/twemproxymake8makeinstall

Set Environment variables:

Echo " path= $PATH:/usr/local/twemproxy/sbin/ " >>/etc/ profile Source/etc/profile

2. Create the relevant directory (storing configuration files and PID files)

cd/usr/local/twemproxymkdir Run conf

3. Add Proxy configuration file

Vim/usr/local/twemproxy/conf/nutcracker.yml

The content is as follows (official profile reference)

Alpha:listen:127.0.0.1:22121hash:fnv1a_64 Distribution:ketama auto_eject_hosts:trueRedis:trueserver_retry_timeout: -Server_failure_limit:1Servers:-127.0.0.1:7000:1-127.0.0.1:7001:1-127.0.0.1:7002:1-127.0.0.1:7003:1-127.0.0.1:7004:1-127.0.0.1:7005:1

I installed 6 Redis instances locally, unclear to my classmates to refer to the Redis official website installation or my previous article Redis installation and master-slave implementation.

4. Start the Twemproxy service

NUTCRACKER-T Test configuration file

Start command:

nutcracker-d-c/usr/local/twemproxy/conf/nutcracker.yml-p/usr/local/twemproxy/run/redisproxy.pid-o/usr/local/ Twemproxy/run/redisproxy.log

Nutcracker Usage and command options

file ]                  [file] [-s stats port] [-a stats addr]                  [file] [-m mbuf size]

Options:-h,–help: Viewing the Help document, displaying command options-v,–version: View Nutcracker version-t,–test-conf: Testing the correctness of the configuration script-d,–daemonize: Run as daemon-d,–describe-Stats: Print status Description-v,–verbosity=n: Set the log level (default:5, min:0, Max: One)-o,–output=S: Set the log output path, default to standard error output (DEFAULT:STDERR)-c,–conf-file=s: Specifies the configuration file path (default:conf/nutcracker.yml)-s,–stats-port=n: Set the status monitoring port, default 2222222222)-a,–stats-addr=s: Set the Status monitoring IP, default 0.0.0.0(Default:0.0.0.0)-i,–stats-interval=n: Sets the state aggregation interval (default:30000msec)-p,–pid-file=S: Specify process PID file path, default off (Default:off)-m,–mbuf-size=n: Sets the mbuf block size to bytes units (default:16384bytes

Review the process and confirm the startup.

PS grep grep grep Root        809     1  0 The following :        XX:xx nutcracker-d -c/usr/local/twemproxy/conf/nutcracker.yml-p/usr/local/  Twemproxy/run/redisproxy.pid-o/usr/local/twemproxy/run/redisproxy.log[[email protected]

5. Simple test

[Email protected] ~]# NETSTAT-NLTP |grepnutcrackertcp0      0 0.0.0.0:22222               0.0.0.0:* LISTEN809/Nutcracker TCP0      0 127.0.0.1:22121             0.0.0.0:* LISTEN809/Nutcracker [[email protected]-server ~]# Redis-cli-p22121             127.0.0.1:22121>Set name Yaunok127.0.0.1:22121>Get name"Yaun"127.0.0.1:22121>

Summarize:

Twemproxy is still very reliable, although the performance of the loss (20%), but relatively well worth it, and proven, very extensive use. Perform the performance test later. For more detailed information, please refer to the official documentation.

Resources:

Https://github.com/twitter/twemproxy

Twemproxy Introduction and use