Publish a reference Ssdb, a redis-like high-performance nosql:ledisdb with Go

Cause

Ledisdb is a reference ssdb, with go implementation, the underlying LEVELDB based, Redis-like high-performance NoSQL database, provides kv,list,hash and zset data structure support.

Our current application relies heavily on Redis, but as our users become more and more large, Redis's memory is becoming more and more scarce, and replication may also cause timeouts. Although we can do this later by adding multiple machines, we still want a single machine to carry more users under the existing machine configuration. In addition, because of the nature of the business, we do not actually need to put all the data into memory, only need to store the current active users.

After our research, we found that Ssdb has done a good job of helping us solve this problem by providing a consistent interface with Redis (some of which are slightly different), but the underlying is LEVELDB for storage. According to its official website, performance is nearing or even exceeding redis.

In the spirit of making wheels, I decided to use go to achieve a similar db, named Ledisdb, that is, level-redis-db, why do not use ready-made ssdb, I think there are several reasons:

• The rapid development of go language, this is undoubtedly, although the performance above the C + + code is definitely a gap, but I can quickly prototype and experiment. In fact, I developed the LEDISDB in a short time, so that I can continue to develop with confidence.
• LEVELDB research, I have always wanted to apply leveldb to our project, as the preferred data storage method of the local hotspot data, through the LEDISDB, let me have a lot of experience in the use of leveldb.
• Redis is familiar, although I used a long redis, but a lot of REDIS commands still need to check the manual, by implementing LEDISDB, I am more familiar with the Redis command, at the same time, because to understand how this command Redis implementation, a new analysis of Redis internal.

When I was preparing to develop LEDISDB, I was thinking about a question, do I need to develop another redis? In fact, this is a very clear question, I do not need another redis. Ledisdb Although reference to Redis, but in order to achieve simplicity, sometimes I do a lot of subtraction or change, for example, zset this data structure, I only support int64 type of score, and Redis's score is a double type, Specific reasons for follow-up explanation of Zset when the details.

So, we can assume that Ledisdb is a NoSQL database based on the Redis communication protocol that provides a variety of advanced data structures, and it's not another redis.

Compiling the installation

Because Ledisdb is written with go, you first need to install go and configure Goroot,gopath.

mkdir $WORKSPACEcd $WORKSPACEgit clone [Email protected]:siddontang/ledisdb.git src/github.com/siddontang/ LEDISDBCD src/github.com/siddontang/ledisdb# Build Leveldb, if already installed, can be ignored./build_leveldb.sh #安装ledisdb go dependency. ./bootstap.sh #配置GOPATH等环境变量. ./dev.sh go install./...  

For specific installation instructions, you can view the Readme under the code directory.

Example

Using Ledisdb is simple and only needs to run:

./ledis-server -config=/etc/ledis.json

LEDISDB configuration file in JSON format, why choose JSON, I use JSON as the main configuration format has been described.

We can use any Redis client connection Ledisdb, such as REDIS-CLI, as follows:

127.0.0.1:6380> set a 1OK127.0.0.1:6380> get a"1"127.0.0.1:6380> incr a(integer) 2127.0.0.1:6380> mset b 2 c 3OK127.0.0.1:6380> mget a b c1) "2"2) "2"3) "3"

Leveldb

Because Leveldb is written in C + +, CGO is a good way to use it in go. Here, I directly use the Levigo this library, and on the top of the package, see here. Although there is a go-leveldb, helpless still can not use.

The performance cost of CGO is still there, this is obvious when I do benchmark, but the subsequent optimization of the space is very large, such as the call logic of more than leveldb to use C rewrite, so that only need a cgo on it. But this follow-up is under consideration.

Leveldb some parameters in the build compile time is need to adjust, this I have no experience, only Google and reference Ssdb. For example, here are some of the following:

+ db/dbformat.h// static const int kL0_SlowdownWritesTrigger = 8;static const int kL0_SlowdownWritesTrigger = 16;// static const int kL0_StopWritesTrigger = 12;static const int kL0_StopWritesTrigger = 64;+ db/version_set.cc//static const int kTargetFileSize = 2 * 1048576;static const int kTargetFileSize = 32 * 1048576;//static const int64_t kMaxGrandParentOverlapBytes = 10 * kTargetFileSize;static const int64_t kMaxGrandParentOverlapBytes = 20 * kTargetFileSize;

The relevant parameters of the tuning, can only wait for my follow-up in-depth study leveldb in good consideration.

Performance testing

No performance test report for any service-side service that's bullying, I'm just using the Redis_benchmark to test the environment for a two-year-old master Mac Air machine.

Test statement:

redis-benchmark -n 10000 -t set,incr,get,lpush,lpop,lrange,mset -q

Redis-benchmark default no hash and Zset test, follow me on my own to join.

LEVELDB configuration:

compression       = falseblock_size        = 32KBwrite_buffer_size = 64MBcache_size        = 500MB

Redis

SET: 42735.04 requests per secondGET: 45871.56 requests per secondINCR: 45248.87 requests per secondLPUSH: 45045.04 requests per secondLPOP: 43103.45 requests per secondLPUSH (needed to benchmark LRANGE): 44843.05 requests per secondLRANGE_100 (first 100 elements): 14727.54 requests per secondLRANGE_300 (first 300 elements): 6915.63 requests per secondLRANGE_500 (first 450 elements): 5042.86 requests per secondLRANGE_600 (first 600 elements): 3960.40 requests per secondMSET (10 keys): 33003.30 requests per second

Ssdb

SET: 35971.22 requests per secondGET: 47393.37 requests per secondINCR: 36630.04 requests per secondLPUSH: 37174.72 requests per secondLPOP: 38167.94 requests per secondLPUSH (needed to benchmark LRANGE): 37593.98 requests per secondLRANGE_100 (first 100 elements): 905.55 requests per secondLRANGE_300 (first 300 elements): 327.78 requests per secondLRANGE_500 (first 450 elements): 222.36 requests per secondLRANGE_600 (first 600 elements): 165.30 requests per secondMSET (10 keys): 33112.59 requests per second

Ledisdb

SET: 38759.69 requests per secondGET: 40160.64 requests per secondINCR: 36101.08 requests per secondLPUSH: 33003.30 requests per secondLPOP: 27624.31 requests per secondLPUSH (needed to benchmark LRANGE): 32894.74 requests per secondLRANGE_100 (first 100 elements): 7352.94 requests per secondLRANGE_300 (first 300 elements): 2867.79 requests per secondLRANGE_500 (first 450 elements): 1778.41 requests per secondLRANGE_600 (first 600 elements): 1590.33 requests per secondMSET (10 keys): 21881.84 requests per second

You can see that the performance of Ledisdb to catch Redis and Ssdb still have a gap, but also not unusable, some differences are not big. As for why Lrange than Ssdb high, I am more confused.

Follow up the test report, I will constantly update in the benchmark file.

Todo......

Ledisdb is also a very new project, compared to SSDB has been in the production environment for a long time, there are many ways to go, there are some important functions need to be implemented, such as replication and so on.

Welcome to be interested in children's shoes to participate in, in the long process of development on the road there are some good friends but very lucky.