Analysis of Redis network protocol with Netty

Analysis of Redis network protocol with Netty

According to the Redis Official document introduction, studied the Redis network communication protocol. Then accidentally found in the GitHub on the netty of the implementation of the Redis server, very interesting, so the hands-on realized a bit. 1.RESP Protocol

The Redis client and server use a network communication protocol called RESP (Redis serialization Protocol) to Exchange data. Resp's design weighed the three factors of simplicity, quick parsing, and human readable. Redis clients serialize integers, strings, data, and other data types by RESP, sending a string array representing the arguments to the server. The server responds to different data types according to different request commands. In addition to pipelines and subscriptions, the Redis client and server are communicating in this simple request-response model.

Specifically, RESP supports five types of data. The total length is identified with the "*" message header, and there may also be a "$" identity string length in the message, with each line ending in \ r \ n : simple string (simply string): Starting with "+" to represent the correct state information, "+" is the specific information. Many Redis commands use a simple string as a successful response, such as "+ok\r\n". But a simple string is not binary safe because it does not have length information like bulk string, and can only be determined by \ r \ n, so that the string cannot contain \ r \ n. error: "-" begins with "-" to indicate error status information, "-" is the specific information. Integer (integer): With ":" Beginning, like Setnx, DEL, EXISTS, INCR, Incrby, DECR, Decrby, Dbsize, Lastsave, Renamenx, move, Llen, Sadd, Srem, Sismember, and scard all return integers. Bulk String (Bulk string): Begins with "$", representing the string length of the next line, with the string reaching a maximum of 512MB in the next line. "$-1\r\n" is called a null Bulk String, indicating that no data exists. array: Begins with "*" to indicate how many rows (excluding the current row) the message total has, and "*" is the specific number of rows. The client uses a RESP array to represent the command to the server, and in turn the server can return the collection of data to the client with a RESP array. An array can be a mixed data type, such as an integer plus a string "*2\r\n:1\r\n$6\r\nfoobar\r\n". In addition, nested arrays are also possible.

For example, to observe the RESP of the following command, this group of Set/get is what we are going to achieve in Netty:

Set name HelloWorld
->
*3\r\n
$3\r\n
set\r\n
$4\r\n name\r\n $10\r\n helloworld\r\n
<-
: 1\r\n get

name
->
*2\r\n
$3\r\n get\r\n $4\r\n
name\r\n
<-
$10\r\n
helloworld\r\n

set name abc111
->
*3\r\n $3\r \
set\r\n
$4\r\n
name\r\n
$6\r\n
abc111\r\n
<-
: 0\r\n
get Age ->
*2\r\n
$3\r\n
get\r\n
$3\r\n
age\r\n
: <-

2. Resolution Protocol with Netty

The following is a high performance network communication framework Netty implement a simple Redis server backend, parse set and get commands, and save key value pairs. 2.1 Netty Version

The Netty version, 5.0, is still alpha, using the latest in the final edition. But even 4.0.25.Final is somewhat different from the first few versions of 4.0, and the APIs used in some online examples are simply not available. Netty API changed a bit too "self-willed" it. :)

        <dependency>
            <groupId>io.netty</groupId>
            <artifactid>netty-all</ artifactid>
            <version>4.0.25.Final</version>
        </dependency>

2.2 Start Service

Netty Server boot code, this code should be Netty 4 Standard template, the details are not in this article. The main concern is our registered several handler. Netty handler are divided into inbound and Outbound,rediscommanddecoder and Rediscommandhandler are inbound,rediscommanddecoder: rediscommanddecoder : resolves the Redis protocol to convert a byte array to a Command object. Redisreplyencoder : Writes the response to the output stream and returns to the client. Rediscommandhandler : Executes the commands in the command.

public class Main {public static void main (string[] args) throws Exception {new Main (). Start (6379);
        public void start (int port) throws Exception {Eventloopgroup group = new Nioeventloopgroup (); try {serverbootstrap b = new Serverbootstrap (). Group (group). Channel (Nioserversocketchannel.class). LocalAddress (Port). Childhandler (New Channelinitia Lizer<socketchannel> () {@Override public void Initchannel (Socketcha Nnel ch) throws Exception {Ch.pipeline (). AddLast (New Red
                                    Iscommanddecoder ()). AddLast (New Redisreplyencoder ())
                        . AddLast (New Rediscommandhandler ());

            }
                    }); Bind and start to accept incoming connections.
            Channelfuture f = b.bind (port). sync ();
            Wait until the "server socket is closed."
        F.channel (). Closefuture (). sync ();
            finally {//Shutdown the Eventloopgroup, which releases all resources.
        Group.shutdowngracefully (); }
    }

}

2.3 Protocol Resolution

When the Rediscommanddecoder begins, the CMDS is null and enters the Dodecodenumofargs to parse out the number of commands and parameters and initialize the CMDS. Then you will enter Dodecodeargs to parse the command name and parameters one by one. When the final completes, the Rediscommand object is created based on the result of the resolution and added to the Out list. So the next handler will be able to continue processing.

public class Rediscommanddecoder extends replayingdecoder<void> {/** decoded command and arguments */priv

    Ate byte[][] CMDS;

    /** Current Argument * * private int arg; /** Decode in Block-io style, rather than NIO.  * * @Override protected void decode (Channelhandlercontext ctx, bytebuf in, list<object> out) throws Exception
            {if (Cmds = = null) {if (in.readbyte () = = ' * ') {Dodecodenumofargs (in);
        } else {Dodecodeargs (in);
            } if (Iscomplete ()) {Dosendcmdtohandler (out);
        Docleanup ();  }/** Decode Number of arguments * * private void Dodecodenumofargs (Bytebuf in) {//Ignore negative
        case int Numofargs = ReadInt (in);
        System.out.println ("Rediscommanddecoder Numofargs:" + Numofargs);

        Cmds = new byte[numofargs][];
    Checkpoint (); }/** Decode Arguments * * private void doDecodeargs (Bytebuf in) {for (int i = arg; i < cmds.length; i++) {if (in.readbyte () = = ' $ ') {
                int lenofbulkstr = ReadInt (in);

                System.out.println ("Rediscommanddecoder lenofbulkstr[" + i + "]:" + lenofbulkstr);
                Cmds[i] = new BYTE[LENOFBULKSTR];

                In.readbytes (Cmds[i]);

                Skip CRLF (\ r \ n) in.skipbytes (2);
                arg++;
            Checkpoint ();
            else {throw new IllegalStateException ("Invalid argument"); }}/** * Cmds!= null means header decode complete * arg > 0 means arguments decode has B
     Egun * arg = = Cmds.length means complete!
                * Private Boolean Iscomplete () {return (Cmds!= null) && (Arg > 0)
    && (arg = = cmds.length); /** Send decoded command to next handler * * private void DosendcmdtohandlER (list<object> out) {System.out.println ("rediscommanddecoder:send command to Next handler");
        if (cmds.length = = 2) {Out.add (new Rediscommand (New String (cmds[0)), cmds[1]);
        else if (cmds.length = = 3) {Out.add (new Rediscommand (New String (cmds[0)), cmds[1], cmds[2));
        else {throw new illegalstateexception ("Unknown command");
        }/** Clean up state info */private void Docleanup () {this.cmds = null;
    This.arg = 0;
        private int readInt (bytebuf in) {int integer = 0;
        char c;
        while ((c = (char) in.readbyte ())!= ' \ r ') {integer = (integer *) + (C-' 0 ');
        } if (In.readbyte ()!= ' \ n ') {throw new IllegalStateException ("Invalid number");
    return integer; }

}

Because we simply implement the set and get commands, we can only have one parameter or two parameters:

public class Rediscommand {

    /** Command name *
    /private final String name;

    /** Optional Arguments * *
    private byte[] arg1;
    Private byte[] arg2;

    Public Rediscommand (String name, byte[] arg1) {
        this.name = name;
        THIS.ARG1 = arg1;
    }

    Public Rediscommand (String name, byte[] arg1, byte[] arg2) {
        this.name = name;
        THIS.ARG1 = arg1;
        THIS.ARG2 = arg2;
    }

    Public String GetName () {return
        name;
    }

    Public byte[] GetArg1 () {return
        arg1;
    }

    Public byte[] GetArg2 () {return
        arg2;
    }

    @Override public
    String toString () {return
        "command{" +
                "name= '" + name + ' \ ' +
                ", arg1=" + arrays.t Ostring (arg1) +
                ", arg2=" + arrays.tostring (arg2) +
                '} ';
    }

2.4 Command Execution

Rediscommandhandler Gets the Rediscommand, executes the command according to the command name. Here, a hashmap is used to simulate the database, set goes to map, and get is taken from inside. In addition to performing specific actions, you also return different reply objects based on the results of the execution: save successfully : 1\r\n. Successful modification : back: 0\r\n. Indicates that this key already exists in the map. Query succeeded : Returns a bulk String. Specifically see the back bulkreply. key does not exist : return: -1\r\n.

@ChannelHandler. sharable public class Rediscommandhandler extends simplechannelinboundhandler<rediscommand> {p

    Rivate hashmap<string, byte[]> database = new hashmap<string, byte[]> (); @Override protected void channelRead0 (Channelhandlercontext ctx, Rediscommand msg) throws Exception {SYSTEM.O

        Ut.println ("Rediscommandhandler:" + msg); if (Msg.getname (). Equalsignorecase ("set")) {if (Database.put (New String (MSG.GETARG1 ()), msg.getarg2 ()) = = nul
            L) {Ctx.writeandflush (new integerreply (1));
            else {Ctx.writeandflush (new integerreply (0)); } else if (Msg.getname (). Equalsignorecase ("get")) {byte[] value = Database.get (New String (M
            SG.GETARG1 ()));
            if (value!= null && value.length > 0) {ctx.writeandflush (new bulkreply (value));
      else {Ctx.writeandflush (bulkreply.nil_reply);      }
        }
    }

} 

2.5 Send a response

Redisreplyencoder implementation is relatively simple, get redisreply message, directly written to the BYTEBUF can be. The specific writing methods are implemented in each redisreply.

public class Redisreplyencoder extends messagetobyteencoder<redisreply> {

    @Override
    protected void Encode (Channelhandlercontext ctx, redisreply msg, bytebuf out) throws Exception {System.out.println
        (" Redisreplyencoder: "+ msg";
        Msg.write (out);
    }

Public interface redisreply<t> {byte[] CRLF = new byte[] {' \ r ', ' \ n '};

    T data ();

void Write (Bytebuf out) throws IOException;

    The public class Integerreply implements redisreply<integer> {private static final char MARKER = ': ';

    private final int data;
    public integerreply (int data) {this.data = data;
    @Override public Integer data () {return this.data;
        @Override public void Write (Bytebuf out) throws IOException {out.writebyte (MARKER);
        Out.writebytes (string.valueof (data). GetBytes ());
    Out.writebytes (CRLF);
                @Override public String toString () {return "integerreply{" + "data=" + Data +
    '}'; } public class Bulkreply implements redisreply<byte[]> {public static final bulkreply nil_reply = new Bulk

    Reply ();

    private static final char MARKER = ' $ ';

    Private final byte[] data; Private final int LEn
        Public bulkreply () {this.data = null;
    This.len =-1;
        Public bulkreply (byte[] data) {this.data = data;
    This.len = Data.length;
    @Override public byte[] data () {return this.data; @Override public void Write (Bytebuf out) throws IOException {//1.Write header Out.writebyte (M
        Arker);
        Out.writebytes (String.valueof (len). GetBytes ());

        Out.writebytes (CRLF);
            2.Write data if (Len > 0) {out.writebytes (data);
        Out.writebytes (CRLF); @Override public String toString () {return "bulkreply{" + "bytes=" + arrays.tost
    Ring (data) + '} '; }
}

2.6 Run Test

When the service is running, the official REDIS-CLI will be able to connect with our service and execute some commands to test it. See oneself realize Redis "pseudo service End" can "cheat" redis-cli, still very have a sense of achievement.

127.0.0.1:6379> Set name HelloWorld
(integer) 1
127.0.0.1:6379> get name
"HelloWorld"
127.0.0.1:6379> Set name abc123
(integer) 0
127.0.0.1:6379> get name
"abc123"
127.0.0.1:6379 > Get Age
(nil)

those "pits" in 3.Netty 4.

Because it is the first use of Netty 4, a lot of information on the Internet is Netty 3 or Netty 4 earlier versions of the API are different, so encountered a lot of problems, the official document did not find the answer, a little debugging, guess, look at the source code to touch a little "doorway": Handler Basic class : Netty 4 Use Simplechannelinboundhandler on it, the previous API is no longer applicable. data exchange between inbound and outbound processors : The context object is the interface for data interchange, and the difference is that inbound is made by Firechannelread () for data exchange, But from inbound to outbound the Writeandflush () was triggered. Inbound and outbound order : Firechannelread () will find the next inbound processor backwards, but Writeandflush () will look forward to the previous outbound processor. Therefore, in Channelinitializer, outbound should be placed in front of the Simplechannelinboundhandler in order to exchange data. @Sharable Annotation : If the handler is stateless, you can mark this annotation.