Flare-install, deploy, and test a distributed key/value storage system compatible with the Memcached protocol

Flare is an open-source product developed by GreenLab. it uses TC at the underlying layer and is fully compatible with the Memcached protocol (ASCII, non-binary ). The primary features are as follows: Support for Master/slave replication and support for adding and deleting master partitions and support for failover persistent storage. now, everyone seems to be concerned about the key/value storage solution. Work

Flare, an open-source product developed by Green Lab, uses TC at the underlying layer, and is fully compatible with the Memcached protocol (ASCII, non-binary ). It has the following features:

Supports Master/slave replication and allows you to add or delete dynamic nodes in master partitions. you can use failover for persistent storage.

It seems that everyone is very concerned about the key/value storage solution. As a supplement to traditional databases. Because Memcached lacks the persistent storage function, it cannot be used as a reliable key/value solution.
I have been concerned about some solutions derived from Memcached, including

The memcachedb-Sina team, bdb + memcached protocol tokyo tyrant-TC + memcached network interface LightCloud implements multi-node distributed management on Tokyo Tyrant. Repcached-added the Copy function based on memcached source code redist-in addition to string, it also allows storage of list, set, and other types

In addition to the above software, there are some solutions using Erlang and Java, which I did not consider. First of all, my application scenario is not as big as that, and I hope to choose a cost-effective low-end solution
Among them, the most important thing is TT (Toyko tyrant ). However, what I'm not quite satisfied with TT is that memcached protocol compatibility is not good enough. for example, if it cannot properly handle expiration, it cannot handle flag. As a result, the client cannot be decompressed normally and unserialize can be processed on the client. However, it was intended to be used as a session storage, and it would be troublesome to automatically expire, the solution is to execute lua on the TT side for gc, and the overall perception is incomplete.
Memcachedb is good for storing fixed-length fields, but the storage of non-fixed-length records does not seem good enough, and there is no current example.
Repcached only supports mm replication, but it cannot solve persistent storage problems.
The redist lacked the APIs I needed during the initial test. However, php pecl APIs are available now. The redist replication mechanism has also been greatly improved. There are not many redist application instances. However, I think it deserves continuous attention like mongoDb.
Flare is the final choice. After testing, although the speed is not as fast as others, I am very satisfied with its excellent scalability. In addition, memcached protocol compatibility is also very good.
There is another feature. the flare key can exceed 256 bytes, and the value can exceed 1 mb. The traditional memcached memory allocation policy imposes the above restrictions.
Flare's underlying storage is scalable and currently uses tc (tokyo cabinet ).
From this structure, it is a good alternative to tt.
Flare's running performance is still good. it is currently used in GREE. it is taken from the project website description:

Flare is running at GREE (one of the major SNS services in Japan) w/ 6 masters and 6 slaves (over 2,000M keys), and 500-1,000 qps (load average is nearly 0.00…).

It looks attractive. Because the TT key exceeds M, the performance will decrease linearly. The lightclound of plurk solves this problem through multi-node load balancing. (Btw and plurk.com are currently in harmony. to get the lightclound source code, you still need to go through the wall. alas)
The following is an installation test on my osx.
1. first download the source code

http://labs.gree.jp/Top/OpenSource/Flare/Download-en.html

2. use port to install boost

$ portinstall boost

3. install tc

http://tokyocabinet.sourceforge.net/

4. compile flare

tar zxvf flare-1.0.8.tgz./configure –with-boost=/opt/local –prefix=/opt/flaremake install

For more detailed installation and configuration instructions, refer to this article: how to compile and install flare from the source code in Ubuntu and how to install it from the Debian package

Flare only has 2 files after Compilation
Flarei, flared
Index server and node server.
Very refreshing.
5. run
There is an etc directory under flare sources, with the index server and node server configurations in the middle.
Can be copied directly to/opt/flare/etc
Then use

flared –daemonize -fflarei –daemonize -f

You can run it.
6. concepts
Flare has several concepts that need to be understood
Index server
This is an index server used to control the node server status.
Note that the client does not directly interact with the index server.
Node server
There are three types of role for the actual storage node. node:
Master/slave/proxy
Master is the master node, slave is the shunt node, used for synchronous replication of master
Proxy forwards client requests to the appropriate node (including master/slave ).
Some people do not quite understand why proxy is used and whether or not to do so. the actual test shows that,
Requests forwarded through proxy are indeed much slower than those forwarded directly to the actual node.
This is because flare is a cluster in which node servers can be dynamically added or deleted.
When a node is down, the index server detects it and marks its state as down.
In addition, the master supports partition, so the client does not need to handle these complex problems through the proxy node.
According to the author's suggestion, the proxy node should run locally with the client to reduce unnecessary tcp requests.
The following is a script I wrote to build a typical test scenario:
1 index server
Two master nodes, enabling partition
1 slave, enabling balance
Because flare uses ip: port as a node, only one ip address is required, and different ports can be used to quickly implement the environment required for testing.

$. /Start_flare.sh # Start index server and listen to 127.0.0.1: 12120 flarei-daemonize-f/opt/flare/etc/flarei. conf # proxyflared-daemonize-data-dir/opt/flare/var/data/d0-index-server-name 127.0.0.1-index-server-port 12120-server-name 127.0.0.1-server -port 12121 # master1 nodeflared-daemonize-data-dir/opt/flare/var/data/d1-index-server-name 127.0.0.1-index-server-port 12120-server-name 127.0.0.1-server-po Rt 12122 # slave node # flared-daemonize-data-dir/opt/flare/var/data/d2-index-server-name 127.0.0.1-index-server-port 12120-server- name 127.0.0.1-server-port 12123 # master2 # flared-daemonize-data-dir/opt/flare/var/data/d3-index-server-name 127.0.0.1-index-server-port 12120-server-name 127.0.0.1-server-port 12124 echo "waiting dameon startup ..." Sleep 3 echo "set nodes role ...." # When a node is added, the default value is proxy role. Therefore, you need to modify the role # telnet to the index server to execute these commands, in the script, we can use # nc to automatically execute # node role command format: # node role server port master | slave | proxy balance partiionecho "node role 127.0.0.1 12122 master 1 0" | nc 127.0.0.1 12120 # Set to slave and balance to 2, give twice the read weight echo "node role 127.0.0.1 12123 slave 2 0" | nc 127.0.0.1 12120 # 2nd master nodes, set partition to 1, indicating this is 2nd partion, allow # synchronize part of the data from the original master to echo "node role 127.0.0.1 12124 master 1 1 1" | nc 127.0.0.1 12120 sleep 2 echo "stats nodes" | nc 127.0.0.1 12120 STAT 127.0.0.1: 12121: role proxySTAT 127.0.0.1: 12121: state downSTAT 127.0.0.1: 12121: partition-1 STAT 127.0.0.1: 12121: balance 0 STAT 127.0.0.1: 12121: thread_type 16 STAT 127.0.0.1: 12122: role masterSTAT 127.0.0.1: 12122: state downSTAT 127.0.0.1: 12122: partition 0 STAT 127.0.0.1: 12122: balance 1 STAT 127.0.0.1: 12122: thread_type 17 STAT 127.0.0.1: 12123: role slaveSTAT 127.0.0.1: 12123: state prepareSTAT 127.0.0.1: 12123: partition 0 STAT 127.0.0.1: 12123: balance 0 STAT 127.0.0.1: 12123: thread_type 18 STAT 127.0.0.1: 12124: role masterSTAT 127.0.0.1: 12124: state prepareSTAT fig: 12124: partition 1 STAT 127.0.0.1: 12124: balance 1 STAT 127.0.0.1: 12124: thread_type 19END

Stats nodes can list the status information of each node.
Note that the state of some nodes is prepare, indicating that synchronous replication is in progress.
I conducted a simple test to read and write 1 M records respectively,
Test Code

 addServer($host,$port);    $loop = 1000000;    echo "start set tests...\n";    $start =microtime(true);    for ($i=0; $i < $loop; $i++) {        $memcached->set(md5("k$i"),md5("k$i"));    }    $end = microtime(true);    echo "ok.\n";    $total1 = $end-$start;    echo "start get tests...\n";    $start =microtime(true);    for ($i=0; $i < $loop; $i++) {        $memcached->get(md5("k$i"));    }    $end = microtime(true);    $total2 = $end-$start;    echo "ok.\n";    echo "set time:$total1 ",$loop/$total1,' rps',"\n";    echo "get time:$total2 ",$loop/$total2, ' rps',"\n";?>

When a single node performs the set/get operation, the speed and memcached
The ascii mode is basically the same, and the rps is only a few hundred.
However, after the proxy node is used, it is only half of the original one.
================
Applicable scenarios of the recommended Flare:
1. the most orthodox is key/value storage. I use flare as the backend implementation of dhash in doggy.
2. Session Storage. For example, PHP can directly apply the session_handler of memcached. Because it is persistent, it solves the embarrassment and troubles of the original memcached.
3. other key/value-based extension schemes
Can Flare directly replace Memcached?
Answer. The advantage and essence of Flare is the persistent storage of distributed key/value, rather than as a cache. As a caching solution, it is much worse than memcached.
Especially after the binary protocol is enabled, memcached has obvious advantages.