Redis/memcache Agent Service Twemproxy Introduction

zhanhailiang 日期：2014-12-14

Brief introduction

Twemproxy, also known as Nutcracker, is twtter Open source Redis and Memcache proxy server.

Function

• Fast.
• Lightweight.
• Maintains persistent server connections.
• Keeps connection count on the backend caching servers low.
• Enables pipelining of requests and responses.
• Supports proxying to multiple servers.
• Supports multiple server pools simultaneously.
• Shard data automatically across multiple servers.
• Implements the complete memcached ASCII and Redis protocol.
• Easy configuration of server pools through a YAML file.
• Supports multiple hashing modes including consistent hashing and distribution.
• Can is configured to disable nodes on failures.
• Observability through stats exposed on stats monitoring port.
• Works with Linux, *bsd, OS X and Solaris (Smartos)

Installation

Download the installation package from official website compile and install: Distribution Tarball:

[[email protected]~/software/nutcracker-0.3.0]# ./configure --prefix=/usr/local/nutcracker-0.3.0[[email protected]~/software/nutcracker-0.3.0]# make[[email protected]~/software/nutcracker-0.3.0]# make install[[email protected]~/software/nutcracker-0.3.0]# cp -R conf /usr/local/nutcracker-0.3.0[[email protected]~/software/nutcracker-0.3.0]# ln -s /usr/local/nutcracker-0.3.0 /usr/local/nutcracker[[email protected]~/software/nutcracker-0.3.0]# ln -s /usr/local/nutcracker/sbin/nutcracker /usr/local/bin/nutcracker

Configuration

Twemproxy support is configured with YAML syntax, which supports the following set of instructions:

• Listen: The Listening address and port (Name:port or Ip:port) for this server pool.
• Hash: The name of the hash function. Possible values are:
  • One_at_a_time
  • Md5
  • Crc16
  • CRC32 (CRC32 implementation compatible with libmemcached)
  • CRC32A (correct CRC32 implementation as per the spec)
  • Fnv1_64
  • Fnv1a_64
  • Fnv1_32
  • Fnv1a_32
  • Hsieh
  • Murmur
  • Jenkins
• Hash_tag: A character string that specifies the part of the key used for hashing. Eg "{}" or "$$". Hash tag enable mapping different keys to the same server as long as the part of the key within the tag is the same.
• distribution: The key distribution mode. Possible values are:
  • Ketama
  • Modula
  • Random
• Timeout: The timeout value in msec, we wait for establish a connection to the server or receive a response From a server. By default, we wait indefinitely.
• Backlog: The TCP backlog argument. Defaults to 512.
• preconnect: A boolean value that controls if Nutcracker should preconnect to all the servers in this pool on proc ESS start. Defaults to False.
• Redis: A Boolean value that controls if A server pool speaks Redis or memcached protocol. Defaults to False.
• server_connections: The maximum number of connections that can is opened to each server. By default, we open at the most 1 server connection.
• auto_eject_hosts: A boolean value that controls if server should is ejected temporarily when it fails consecutive Ly server_failure_limit times. See Liveness recommendations for information. Defaults to False.
• server_retry_timeout: The timeout value in msec to wait for before retrying on a temporarily ejected server when Auto_eject_host is set to true. Defaults to 30000 msec.
• server_failure_limit: The number of conseutive failures on a server, would leads to it being temporarily ejec Ted when Auto_eject_host are set to true. Defaults to 2.
• servers: A List of server address, port and weight (name:port:weight or ip:port:weight) for the This server pool.

For the example of this article, configure a Redis agent and a memcache proxy service as follows:

[[email protected]~/software/nutcracker-0.3.0]# cat/usr/local/nutcracker/conf/ nutcracker.ymlalpha:listen:127.0.0.1:22121 hash:fnv1a_64 Hash_tag: "{}" Distribution:ketama Auto_eject_hosts:fal SE timeout:400 redis:true servers:-127.0.0.1:6379:1beta:listen:127.0.0.1:22122 hash:fnv1a_64 Distribution: Ketama timeout:400 backlog:1024 preconnect:true auto_eject_hosts:true server_retry_timeout:2000 server_failure_ Limit:3 Servers:-127.0.0.1:11211:1-127.0.0.1:11212:1  

Among them, redis instance 127.0.0.1:6379 and memcache instance 127.0.0.1:11211 127.0.0.1:11212 please pre-boot well.

/etc/init.d/redis_6379 start/usr/local/memcached-1.4.20/bin/memcached -d -m 64 -l 127.0.0.1 -p 11211 -u root/usr/local/memcached-1.4.20/bin/memcached -d -m 64 -l 127.0.0.1 -p 11212 -u root

Test

https://github.com/billfeller/billfeller.github.io/blob/master/code/twenproxyTest.php

Start the service:

[[email protected]/usr/local/nutcracker/conf]# nutcracker-c nutcracker.yml[Sun Dec 14  20:59:04] nc.c:187 nutcracker-0.3.0 built for Linux 2.6.32-431.23.3.el6.x86_64 x86_64 started on PID 14359[sun Dec 14 20:59:04] nc.c:192 run, rabbit run/dig that hole, forget the sun/and when at last the work is Done/don ' t sit D Own/it ' s time to dig another one  

Execute the test script:

[[email protected]~/wade/git/billfeller.github.io/code]# /usr/local/php/bin/php twenproxyTest.php bool(true)string(1) "1"int(1)bool(false)bool(true)int(2)bool(true)int(3)bool(true)bool(false)

Redis/memcache Agent Service Twemproxy Introduction