Microblogging lightweight RPC Framework Motan formal open Source: support Hundreds of billions of calls _ net Pick

The lightweight RPC framework that supports Weibo calls is Motan officially open source, with the project address Https://github.com/weibocom/motan. Weibo technical team hopes to have more excellent open source people in the future, and further improve the optimization. Microblogging lightweight RPC Framework Motan formal Open source

Motan, a lightweight, java-based RPC framework developed by the microblogging team, has been on the inside of Weibo for many years, stabilizing support for billions of internal calls on Weibo every day. Motan is a simple, easy-to-use, highly available RPC service framework based on Weibo's high concurrency and high load scenario optimization.

Motan features: Simple, easy-to-use, high availability

Non-invasive integration, ease of use, and the Spring configuration approach to integrate distributed invocation capabilities without additional code.

Integrated service discovery and service governance capabilities, flexible support for a variety of configuration management components, such as Consul, zookeeper, and so on.

Support the custom dynamic load balance, the Cross machine room flow adjustment and so on advanced service scheduling ability.

Based on high concurrency, high load scene optimization, with Failover and failfast capabilities, RPC service is guaranteed to be highly available.

Industry-typical RPC framework comparison

Currently, the industry RPC framework is broadly divided into two categories, one focusing on service governance and the other on Cross language invocation.

The service-governed RPC framework is typically Dubbo and Dubbox. Dubbo is Ali Open source distributed service framework, the implementation of High-performance RPC calls while providing a rich management capabilities, is a broad application of the excellent RPC framework, but now less maintenance updates. Dubbox is a new feature that is supported by the REST-style remote invocation, KRYO/FST serialization, when an RPC framework is expanded based on the Dubbo framework.

This type of RPC framework features rich functionality, high performance remote invocation, service discovery and service governance capabilities, service decoupling and service governance for large services, and transparent access for specific language (Java) projects. The disadvantage is the high degree of language coupling and the difficulty of cross language support.

The RPC framework for cross language invocation is Thrift, Grpc, Hessian, Hprose, and so on. This type of RPC framework focuses on the cross-language invocation of the service, supports language-independent calls in most languages, and is ideal for multilingual invocation scenarios. However, this kind of framework does not have service discovery mechanism, it needs the agent layer for request forwarding and load balancing strategy control in actual use.

Weibo's Motan RPC tends to be service-governed, and cross-language is trying to integrate with PHP calls. Compared with the Dubbo series, the functionality may not be as complete, and the extension implementation is not that much, but the Motan is more focused on simplicity, ease of use, and usage of high concurrency high availability scenarios.

Motan is a high performance lightweight RPC framework based on Java with practical service governance capabilities and RPC protocol scalability. Service discovery flexible support for a variety of configuration management components, based on high concurrency high load scenarios high availability strategy optimization, good SPI (service Provider Interface) expansion, detailed call statistics, flexible support for a variety of RPC transport protocols, in use, seamless support spring Configuration, with simple and flexible configuration can be quickly access to use.

Architecture and module design of Motan

Architecture design, divided into service provider (RPC Server), service caller (RPC Client), registry (Registry) Three roles, server to Registry registration declaration of services provided, Client to Registry subscription to the specified service, and Registry the server that returned the list of services to establish a connection and make RPC service calls. The Client perceives the Server's state change through Registry. The interactive relationship of the three is as follows:

The service modular design facilitates the flexible expansion, the Motan mainly includes register, transport, serialize, protocol, cluster and so on, each module supports the expansion through the SPI, each module interactive diagram is as follows:

Register module

Used to interact with the registry, including registration services, subscription services, service change notifications, service heartbeat, and so on; the Server will register the service through the Register module when the system is initialized, and the client side will subscribe to the service through the Register module during system initialization Server list, which is also notified to the Client by the Register module when the server list is changed.

Protocol Module

Used for the description of the RPC service and configuration management of the RPC service, this layer can also add different functions of filter to complete the statistics, concurrency restrictions and other functions.

The Serialize module serializes and deserializes objects such as parameters and results in RPC requests, that is, the conversion of objects to byte streams; The default is to serialize Java friendlier hessian2.

The transport module is used for remote communication and uses the TCP long link method of Netty NIO by default.

Cluster module

The client side uses the module, cluster is a set of available server in the logical encapsulation, contains a number of RPC services can provide the server, the actual request will be based on different high availability and load-balancing policies to select an available server to initiate a remote call.

In the case of RPC requests, the Client invokes the cluster module through the proxy mechanism, cluster selects an available Server based on the configured HA and LoadBalance, converts the RPC request into a byte stream through the Serialize module, and then passes the TRA The Nsport module is sent to the server side.

Service configuration enhances Motan Ease-of-use, and the Motan Framework abstracts functional modules into four configurable elements, respectively:

Protocol: Service communication protocol. The protocol that the service provider makes a remote call to the consumer by default is the Motan protocol, which is serialized using Hessian2, Netty as Endpoint and the Protocol encoding using Motan customization. Registry: Registration Center. The service provider registers the service information (including IP, port, service policy, etc.) into the registry, and the service consumer discovers the service through the registry. When the service changes, the Registry is responsible for notifying the various consumer parties. Service: Services provided by the service provider. The user will extract the core business as an independent service. The caller is invoked by exposing the service and registering the service to the registry. Referer: The service consumer's reference to the service, that is, the service caller.

Motan recommends using the Spring configuration RPC service, which currently Motan 6 custom spring XML tags motan:protocol motan:registry motan:basicservice

Motan:service Motan:basicreferer Motan:referer

High availability is a feature of Motan that supports multiple service governance and highly available mechanisms, including:

Flexible cluster load Balancing strategy, support activeweight/random/roundrobin/localfirst/consistent and other 6 kinds of policies, and support custom extension; Automatic integration Failover, failfast fault-tolerant strategy, Automatic removal of fault nodes, automatic detection and recovery, effective service fault isolation, away from service die and avalanche; connection pool custom control, according to the business scene flexible configuration, support the multi-machine room calls traffic compression, dynamic flow adjustment, to achieve a true cross-IDC high availability.

Based on high concurrency and high load scenario optimization, high availability under high pressure scenarios.

Benchmark test conditions are as follows:

Server side: A scenario with multiple Client connections, 50 concurrent numbers, and 100

Empty packet Request: Single Server TPS 18W

1 k String Request: Single Server TPS 8.4W

5K String Request: Single Server TPS 2W

Client side: (scene contrast diagram below)

Motan provides a basic performance testing framework, welcome to user performance evaluation, source code please refer to the Motan benchmark documentation:

Https://github.com/weibocom/motan/tree/master/motan-benchmark

Motan usage and ease of use, Motan uses Spring for configuration, business code needs no modification, engineering dependencies involve only the core 5 modules and can be relied upon on demand. For specific steps about using the Motan framework in your project, refer to the QuickStart documentation: HTTPS://GITHUB.COM/WEIBOCOM/MOTAN/BLOB/MASTER/DOCS/WIKI/ZH_QUICKSTART.MD

Whether to repeat the wheel

In the choice of RPC frameworks, as compared to the various RPC frameworks above, the current industry has few outstanding RPC frameworks to choose from and maintain. At the same time, in view of the very large number of internal calls, and there will be a lot of customized scenarios, to achieve smooth migration to these RPC frameworks also need to do a lot of customization of the transformation, and finally micro-Bo decided to independent research and development. Mainly from the following aspects to consider:

The performance and usability of the framework need to be customized;

Micro-Blog Internal call scale is very large, the industry rarely similar scenarios of the application experience can be used for reference to high concurrency and complex logic scene customization optimization. For example, the Failover and failfast mechanism of Motan as far as possible to achieve smooth migration;

Online business migration needs to protect business transformation as little as possible, support can be quickly rolled back, this must have effective mechanism protection. For example, Motan's INJVM mechanism and other future multi-language compatible access demands;

The overall technology system includes Java and PHP, as well as some Erlang, C + + and so on, in the future hope to be able to solve the whole internal dependency invocation problem through this service framework. Technology to accumulate reserves and control power.

Development and open source of Motan

Motan is now widely used within Weibo, supporting hundreds of billions of internal calls every day, a process that continues to improve optimization. From service discovery, service fault tolerance, fast failure, fault degradation and other aspects, the complex business architecture and high concurrent scenarios for continuous customization optimization improvement. With the rise of virtualization technology, flexible scheduling has become an indispensable capability of mature technology framework, and the new Motan Framework technology manager has added the functions of data traffic compression, dynamic flow adjustment, multiple registration center support and so on, so that Motan can adapt to the changing times.

In order to facilitate the reuse of other teams, for Motan core functions of the extraction and encapsulation, to eliminate micro-bo rely on their own, the formation of today's Motan open source version, hope to play the power of open source community, further development and play the value of Motan.

Motan's expectations

In this fast-breaking internet age, software development speed has reached an unprecedented level. This benefits from the large-scale reuse of the software's existing modules. In the past few years, open source software has undoubtedly made great contributions in this respect.

The microblogging technology team says it benefits from the open source community and also wants to contribute to the open source community. Motan is a lightweight RPC framework through micro-scale practice, hoping to further improve the optimization with more excellent open source people in the future. More companies are also expected to enjoy the Motan of this lightweight RPC framework.

Motan Project Address: Https://github.com/weibocom/motan

Q&a

1. Motan and the industry's existing RPC framework, such as Dubbo compared to what advantages.

Dubbo function is richer, compared with Dubbo layering, Motan module level to be simpler, no Exchange and directory. From the result of the test, the performance of Motan is better than Dubbo in the business scene of Weibo.

2. Can the current version be used in a production environment?

At present, Motan supports most of the underlying core business of Weibo, which seems to be more stable at present. However, there will be some use of environmental inconsistencies caused by problems, it is recommended that the best test to be applied to the production environment.

3. Does Motan support Cross-language calls?

Currently only Java applications are supported, and support for the PHP YAR Framework is in development and will support more languages in the future.

4. Open source version of the Motan and Weibo internal version of the same function.

The open source version Motan contains most of the functionality in the build, mainly by removing some of the internal dependencies related to the component.

5. Does Motan support asynchronous invocation? How to achieve it.

Motan requests are called asynchronously at the transport level, and no additional configuration is required.

6. I encountered problems in using Motan, where to ask questions.

can be submitted in Github [Issue]: https://github.com/weibocom/motan/issues