Added native thrift and avro support for dubbo/dubbox,

Added native thrift and avro support for dubbo/dubbox,

(Facebook) thrift/(hadoop) avro/(google) probuf (grpc) is a highly eye-catching and efficient serialization/rpc framework in recent years. Although the dubbo framework supports thrift, however, earlier versions of the dependency only support 0.8.0, and some extensions are also made to the Protocol, not the native thrift protocol.

Although some friends on github have extended dubbo to support native thrift, there are too many codes and only one class is needed:

Thrift2Protocal. java:

Package com. alibaba. dubbo. rpc. protocol. thrit2; import com. alibaba. dubbo. common. URL; import com. alibaba. dubbo. common. logger. logger; import com. alibaba. dubbo. common. logger. loggerFactory; import com. alibaba. dubbo. rpc. rpcException; import com. alibaba. dubbo. rpc. protocol. abstractProxyProtocol; import org. apache. thrift. TProcessor; import org. apache. thrift. protocol. TCompactProtocol; import org. apache. thrift. proto Col. TProtocol; import org. apache. thrift. server. TNonblockingServer; import org. apache. thrift. server. TServer; import org. apache. thrift. transport. TFramedTransport; import org. apache. thrift. transport. TNonblockingServerSocket; import org. apache. thrift. transport. TSocket; import org. apache. thrift. transport. TTransport; import java. lang. reflect. constructor;/*** Add "Native thrift" Support for dubbo-rpc * by Yang junming (http://yjmyzz.cnb Logs.com/) */public class Thrift2Protocol extends AbstractProxyProtocol {public static final int DEFAULT_PORT = 33208; private static final Logger logger = LoggerFactory. getLogger (Thrift2Protocol. class); public int getDefaultPort () {return DEFAULT_PORT;} @ Override protected <T> Runnable doExport (T impl, Class <T> type, URL url) throws RpcException {logger.info ("impl =>" + impl. getClass (); log Ger.info ("type =>" + type. getName (); logger.info ("url =>" + url); TProcessor tprocessor; TNonblockingServer. args tArgs = null; String iFace = "$ Iface"; String processor = "$ Processor"; String typeName = type. getName (); TNonblockingServerSocket transport; if (typeName. endsWith (iFace) {String processorClsName = typeName. substring (0, typeName. indexOf (iFace) + processor; try {Class <?> Clazz = Class. forName (processorClsName); Constructor constructor = clazz. getConstructor (type); try {tprocessor = (TProcessor) constructor. newInstance (impl); transport = new TNonblockingServerSocket (url. getPort (); tArgs = new TNonblockingServer. args (transport); tArgs. processor (tprocessor); tArgs. transportFactory (new TFramedTransport. factory (); tArgs. protocolFactory (new TCompactProtocol. factory () );} Catch (Exception e) {logger. error (e. getMessage (), e); throw new RpcException ("Fail to create thrift server (" + url + "):" + e. getMessage (), e) ;}} catch (Exception e) {logger. error (e. getMessage (), e); throw new RpcException ("Fail to create thrift server (" + url + "):" + e. getMessage (), e) ;}}if (tArgs = null) {logger. error ("Fail to create thrift server (" + url + ") due to null args "); Throw new RpcException (" Fail to create thrift server ("+ url +") due to null args ");} final TServer thriftServer = new TNonblockingServer (tArgs ); new Thread (new Runnable () {public void run () {logger.info ("Start Thrift Server"); thriftServer. serve (); logger.info ("Thrift server started. ");}}). start (); return new Runnable () {public void run () {try {logger.info ("Close Thrift Server"); thr IftServer. stop ();} catch (Throwable e) {logger. warn (e. getMessage (), e) ;}};}@ Override protected <T> T doRefer (Class <T> type, URL url) throws RpcException {logger.info ("type =>" + type. getName (); logger.info ("url =>" + url); try {TSocket tSocket; TTransport transport; TProtocol; T thritclient = null; String iFace = "$ Iface "; string client = "$ Client"; String typeName = type. ge TName (); if (typeName. endsWith (iFace) {String clientClsName = typeName. substring (0, typeName. indexOf (iFace) + client; Class <?> Clazz = Class. forName (clientClsName); Constructor constructor = clazz. getConstructor (TProtocol. class); try {tSocket = new TSocket (url. getHost (), url. getPort (); transport = new TFramedTransport (tSocket); protocol = new TCompactProtocol (transport); thritclient = (T) constructor. newInstance (protocol); transport. open (); logger.info ("thrift client opened for service (" + url + ")");} catch (Exception e) {logger. error (e. getMessage (), e); throw new RpcException ("Fail to create remoting client:" + e. getMessage (), e) ;}return thrifle tclient;} catch (Exception e) {logger. error (e. getMessage (), e); throw new RpcException ("Fail to create remoting client for service (" + url + "):" + e. getMessage (), e );}}}

Override the two Abstract METHODS doExport and doRefer of the parent class AbstractProxyProtocol. doExport is used to expose the RPC service. In this method, start thrift server. dubbo service provider calls this method at startup. DoRefer is used to obtain the corresponding rpc-client after dubbo service consumer discovers the service.

With reference to this idea, avro is also easily integrated:

AvroProtocol. java

Package com. alibaba. dubbo. rpc. protocol. avro; import com. alibaba. dubbo. common. URL; import com. alibaba. dubbo. common. logger. logger; import com. alibaba. dubbo. common. logger. loggerFactory; import com. alibaba. dubbo. rpc. rpcException; import com. alibaba. dubbo. rpc. protocol. abstractProxyProtocol; import org. apache. avro. ipc. nettyServer; import org. apache. avro. ipc. nettyTransceiver; import org. apache. avro. ipc. server; import org. apache. avro. ipc. reflect. reflectRequestor; import org. apache. avro. ipc. reflect. reflectResponder; import java.net. inetSocketAddress;/*** add avro support for dubbo-rpc * by Yang junming (http://yjmyzz.cnblogs.com/) */public class AvroProtocol extends actproxyprotocol {public static final int DEFAULT_PORT = 40881; private static final Logger logger = LoggerFactory. getLogger (AvroProtocol. class); public int getDefaultPort () {return DEFAULT_PORT;} @ Override protected <T> Runnable doExport (T impl, Class <T> type, URL url) throws RpcException {logger.info ("impl =>" + impl. getClass (); logger.info ("type =>" + type. getName (); logger.info ("url =>" + url); final Server server = new NettyServer (new ReflectResponder (type, impl), new InetSocketAddress (url. getHost (), url. getPort (); server. start (); return new Runnable () {public void run () {try {logger.info ("Close Avro Server"); server. close ();} catch (Throwable e) {logger. warn (e. getMessage (), e) ;}};}@ Override protected <T> T doRefer (Class <T> type, URL url) throws RpcException {logger.info ("type =>" + type. getName (); logger.info ("url =>" + url); try {NettyTransceiver client = new NettyTransceiver (new InetSocketAddress (url. getHost (), url. getPort (); T ref = ReflectRequestor. getClient (type, client); logger.info ("Create Avro Client"); return ref;} catch (Exception e) {logger. error (e. getMessage (), e); throw new RpcException ("Fail to create remoting client for service (" + url + "):" + e. getMessage (), e );}}}

Do not forget to add a file named com. alibaba. dubbo. rpc. Protocal under the META-INF/dubbo/internal with the content:

avro=com.alibaba.dubbo.rpc.protocol.avro.AvroProtocol

Next we will talk about how to package it into the dubbo jar:

Add two new projects in dubbo-rpc/pom. xml:

    <modules>        ...        <module>dubbo-rpc-avro</module>        ...        <module>dubbo-rpc-thrift2</module>        ...                </modules>

Then in dubbo/pom. xml:

   <artifactSet>       <includes>    ...           <include>com.alibaba:dubbo-rpc-api</include>           <include>com.alibaba:dubbo-rpc-avro</include>          ...           <include>com.alibaba:dubbo-rpc-thrift2</include>           ...       </includes>   </artifactSet>    

The dependencies Section also needs to be added:

<dependency>            <groupId>com.alibaba</groupId>            <artifactId>dubbo-rpc-thrift2</artifactId>            <version>${project.parent.version}</version>            <exclusions>                <exclusion>                    <groupId>org.apache.thrift</groupId>                    <artifactId>libthrift</artifactId>                </exclusion>            </exclusions>        </dependency>        <dependency>            <groupId>com.alibaba</groupId>            <artifactId>dubbo-rpc-avro</artifactId>            <version>${project.parent.version}</version>            <exclusions>                <exclusion>                    <groupId>org.apache.avro</groupId>                    <artifactId>avro</artifactId>                </exclusion>                <exclusion>                    <groupId>org.apache.avro</groupId>                    <artifactId>avro-ipc</artifactId>                </exclusion>            </exclusions>        </dependency>

The new Protocol is included in the packaged dubbo-xxx.jar. Google's protobuf is currently in the 3.x-beta stage, and will be available in the official version later.

The above Code has been submitted to github: https://github.com/yjmyzz/dubbox (Version: 2.8.4a)

Finally, the dubbo, thrift, avro, and rest protocols are compared and tested. The test case is simple:

  public String ping() {        return "pong";    }

The client calls the ping method, and the server returns the string "pong", which is called 50 thousand times on mac book pro. The result is as follows:

Dubbo RPC testing => 50000 RPC calls (dubbo protocol), a total of 14778 milliseconds, an average of 3383.407715/second avro RPC testing => 50000 RPC calls (avro Protocol ), A total of 10707 milliseconds, with an average of 4669.842285/second thrift RPC testing => 50000 RPC calls (thrift protocol), a total of 4667 milliseconds, average 10713.520508/second REST testing => 50000 REST calls, 112699 milliseconds in total, average 443.659668/second

This is as expected. REST adopts the http protocol, which is naturally the slowest. The underlying network communication between avro and dubbo is implemented by netty, which is in the same order of magnitude. However, avro's binary serialization efficiency is higher, so it's a little faster, while thrift is from the inside out, all implemented by facebook itself, with the best performance, winning other protocols.

Personal suggestion: for a service interface that provides both thrift and REST services, internal subsystems call thrift (because thrift is cross-language, in fact, the thrift-rpc method can also be used for external calls. In some scenarios where thrift-client calls are not convenient, the traditional REST is still adopted.