Netty-based Redis client-nedis

Recently brush up the Redis command, sad is a lot of things have been returned to the teacher, just in time to catch up with Antirez in April Fool's Day released Redis 3.0,redis finally have a support cluster of the official version, So the whim decided to implement a Redis client to soothe my sad heart.

Jedis is already strong enough, its network connection is based on the blocking IO, the implementation is very easy to understand, but Oio and NiO compared performance disadvantage, so decided to use NIO to achieve and Redis server network connection, now the industry's best NIO framework is not Netty, Just before also learned Netty framework, so decided to implement this Redis client based on Netty, so you can also be familiar with the Netty, so a tall name on the freshly baked-nedis. The implementation of the command is not a good struggle, full reference to the official Redis documentation, you can also refer to the Jedis code.

Because this code farming peacetime work is busy, in the company work is impossible to take time to do, a no time, the big boys all kinds of reminders, two is due to the company's information security policy, the company wrote code is not out. So only use the evening work time and weekend spare time to engage, working days have 2-3 hours of time, about 10 o'clock open, to 1 o'clock around, weekend due to take baby cooking, also can only squeeze out 3-4 hours out, so progress is relatively slow, from the beginning of April to now nearly 20 days of time finally completed key, String, hash, list, set, SortedSet all stand-alone commands and client-side shards (sharding), other transactions, LUA scripts, clustering and other functions have not been implemented, and left to the later version of the implementation.

Framework Components

The first priority is to determine the basic skeleton of the code, the skeleton is determined after the implementation of the various commands is purely a matter of workload, in fact, this work is very mechanical. After many modifications, the refactoring ultimately determines the skeleton of the code, the basic flow of the command, and the following are some of the basic components of Nedis:

• Nedisclient and Shardednedis: Two facades in the framework, processing single-machine commands and client key shards, and invoking interfaces that provide all the commands, because functions such as transactions and clustering are not yet available, All future may increase transationnedis and clusternedis and so on. All commands will eventually be Nedis forwarded, and Sharednedis will eventually call Nedisclient. Nedisclient build using Build mode, through the Nedisclientbuilder to build, about the client parameters, the current design is relatively coarse, the parameters are few, currently can give nedisclient set the following parameters:

Parameters	Around
C Onnecttimeoutmills	Connection time-out period
e ventloopgroupsize	Netty the thread pool size of the frame
tcpnodelay	whether to set tcp_no_delay identity
c onnectionpoolsize	connection pool size
m Axconnectionidletimeinmills	Idle connection time-out
Minidleconnections	Minimum number of idle connections

• ConnectionPool: In order to reduce the overhead of duplicate connections, the connection pool multiplexing connection is used, and the command is eventually sent through a connection in the connection pool through the Idlestatehandler monitors idle connections.
• protocoldecoder: Read command response, parse the response, inherit from Replayingdecoder handles packet transfers, and the parsing of command responses is ultimately delegated to redisprotocol for processing.
• responsereceiver: receive protocoldecoder Response analysis results, the results are given to responeadapter the adaptation, and the final adaptation of the results through the responsecallback callback is returned to the command caller, and eventually the connection is returned to the connection pool after processing is complete.
• Responeadapter: Put the results of Protocoldecoder parsing are adapted to the friendly type of the user.
• Responsecallback: Because Netty is an asynchronous processing framework, the command interface we provide does not block until the command returns (so that the benefits of NIO and Netty are not reflected), Instead, the caller is notified by the Responsecallback callback when the command response is reached.

Here is the timing of command send and response:

Test

In terms of the currently completed code, the Nedis in the single-Test code far beyond the framework code, the current single-test code of about 8000 lines and the framework code only about 4000, and the most critical is due to limited time, the test code does not cover all the paths, Some commands, especially the Shard-related command interface have not yet written a single test ( those who have not completed the unit testing code only after the slowly fill up ), so the coverage of 50% is less than, that is, the normal ratio unit test code: The framework code should be greater than 4:1, is not very shocked, But the test code must be written, this is a once and for all, with a single test code after the code to easily verify that the modified code is not a problem will not cause other problems, of course, if the test code to be reliable.

For our command test, there are two questions:

• When testing a command, you may want to rely on other commands to provide data, such as I want to test a GET command, then you need to construct the data through the SET command to support the test of the Get command, which requires that the get command must be executed before the set command, but because the Netty framework through the thread pool to perform the task , the set command and the Get command may be executed by different threads, so that the command execution may be out of order, even if the set command is called before the Get command is not fully guaranteed that the set command arrives at the server, and all calls to the SET command interface to make a small pause before invoking the Get command interface. To ensure that the set command executes before the GET command, the pause time is generally 100ms enough (I set 200ms).
• because the Netty framework is asynchronous, all calls to the command interface do not block, so in order to verify the test effect, it is necessary to ensure that the single measurement method does not end before the response is returned, we pass a synchronization controller Countdownlatch, the await method is called to block before the method ends, and is called when the last response returns countdown release, well, grey is often not elegant.

Here is a test method, the controller is said above Countdownlatch,cmd_pause_time is a pause time, it is a constant placed in the base class, can be arbitrarily modified, after the modification of all the test methods will take effect:

@Testpublic void Testset () {docmdtest (new testaction () {@Overridepublic void Dotest () throws Interruptedexception, nedisexception {client.flushall (null); Thread.Sleep (Cmd_pause_time); Client.set (new responsecallback<string> () {@Overridepublic void done (String Result) {assertequals ("OK", result);} @Overridepublic void failed (Throwable cause) {fail (cause);}, "Key1", "value1"); Thread.Sleep (Cmd_pause_time); Client.set (new responsecallback<string> () {@Overridepublic void done (String Result) {assertequals ("OK", result); Controller.countdown ();} @Overridepublic void failed (Throwable cause) {fail (cause); Controller.countdown ();}, "Key1", "value2");});

Share

In the spirit of everyone for me I for everyone, the framework code has been uploaded to Github:https://github.com/chenyihan/nedis, the code needs to have JDK7 to compile, the next will implement the remaining functions. Very ashamed to get a lot of valuable resources from GitHub, but so far only shared two code, must be a lot of contributions to GitHub, we make progress together.

Netty-based Redis client-nedis