An overview of RPC service practice in micro-blogging platform

In the first minute of 2014, Sina Weibo released 808,298 Records again, and the first second was a 55% increase in the number of tweets. (Data Source: Sina Science and technology) This is the microblogging platform RPC framework "Motan" the first peak after the line, the overall performance is stable, basically reached the initial "application side no sense" goal.

In the case of RPC service, the microblogging platform is not the first to eat crabs: early there are foreign pioneers such as Amazon and ebay, near the finagle of Twitter, Taobao Dubbo and so on, all kinds of public information on the Internet is overwhelming. On the other hand, the simple RPC call function implementation, from the technical view is not complex: The client initiates the call, the framework intercepts the invocation information, the serialization, the transmission, the server side receives the invocation information, the deserialization, according to the call information initiates the actual call to obtain the result, then returns the original road Implementing these functions may also be 35 days of things, but in a complex business environment, stable and reliable application of it is the biggest challenge.

The RPC service splitting process of the microblogging platform began in July 2013. Prior to this, we spent a long time discussing the objectives of the service, primarily the scope of the project: which issues are not part of the problem that the service project needs to address. The actual framework code development cost Three engineers (Wang Yu @wangzhe_asdf9 Chen Bo @fishermen Machunsen @ Machunsen) for about one months, then spent nearly two months driving the first business online: Adjusting the engineer's development model, adjusting the test flow, modifying the on-line system, Add monitoring and alarm, small flow test, grayscale release, the last is the full amount of online. Then it took another one months to put it all online in the main business of the microblogging platform.

Some of the basic data metrics for micro-blogging RPC:

Motan Framework: 2w+ line of Java code, 1w Line test code, UT line coverage of more than 70% (currently Motan implementation, and micro-BO platform for multiple systems are functional binding, but also does not have open source conditions, but open source is that we set up from the beginning of a good goal)

Supports 2 ways of calling: INJVM and TCP remote invocation. The INJVM way is similar to the loopback network card: The data is processed by the protocol stack process, but it does not flow through the real network device. The INJVM approach is primarily used to support development debugging and testing, and to demote the RPC service as a fail-back at the beginning of the online operation

Typical business scene under the single instance of TPS limit 20k, micro-BO platform commonly used two-instance, the single machine limit 40k

Typical business scenario under average response time <3 MS, frame layer extra consumption < 0.01 ms

Maximum number of single core business days calls over 80 billion times

There are approximately two purposes of RPC service: to split a unification application into smaller RPC services, so that the goal is to understand decoupling and improve development efficiency; If you are transforming traditional HTTP or other remote calls into efficient RPC calls, it is to improve operational efficiency. Unfortunately, the RPC framework for the microblogging platform needs to be achieved at the same time for the two purposes: to split a unification application within the platform, and to consider the possibility of subsequent RPC access to the open platform's big customers. Therefore, the micro-blogging platform has made a lot of trade-offs and compromises in technology selection and project design:

First, do you choose an existing open source solution, or do you want to develop a new plan yourself? The basis of choice is sorted by importance: whether to meet their core needs, program maturity, cognitive costs (i.e. two development difficulties). Because it is split into an existing complex application, a core requirement of microblogging platform is: application developers want to smooth migration as far as possible, it is best to do the application without perception. Not one of the many open source solutions we've evaluated can be met, so we can only do it ourselves.

The possibility of flexibility and misuse: framework developers always have a bias and feel that the more flexible the framework is, the better it is that every step of the process can be configured or defined by the user itself. But for an internally enforced framework, the uniformity of usage is equally important, in other words, for most steps, it is necessary to ensure that all users within the team are configured in the same way to prevent misuse and to reduce learning and communication costs. Our experience is that once framework development is complete, there is a "framework consumer" role to limit all flexibility to the framework consumer side, to avoid exposing all details directly to the final business developer

Serialization mode selection: The microblogging platform introduced PB serialization from 2011 to replace JSON text in cache and DB. But in the process of the RPC framework line, we chose the Hessian2 that is more friendly to Java object. Because the previous PB serialization needs to define proto files and generate code, the platform only supports the necessary model classes, while RPC may involve more wrap classes, business logic classes, and so on, providing PB support for all classes is too heavy and later maintenance difficult. Of course, the Motan framework supports a variety of different serialization configurations.

Protocol selection: After evaluating several open source RPC Framework protocol designs, we finally chose to design our own RPC communication protocol based on TCP links, a simple binary protocol: the fixed-length header contains a length field, then the binary body payload, The RPC request or response after serialization.

Cluster Management: The microblogging platform Motan framework currently relies on the internally developed Config service (Code name vintage,based on zookeeper) for service registration, service discovery, and change notification.

Trace Systems: The microblogging platform Motan framework currently relies on internally developed trace systems such as Twitter Zipkin (Code name Watchman) to sample RPC requests and total trace.

Motan schema (RPC call Data flow diagram)