C ++ multi-process concurrency framework

I have been engaged in server program development for three years and have been busy. Not long ago, I ended my first job in my career. After a week of rest, I can finally write a summary. So we sorted out and optimized the previous open source code, which is FFLIB. Although the summary here looks like a diary, there are a lot of nonsense, but this article is still very targeted. Provide your own opinions on common problems in server development, such as multi-thread concurrency, message forwarding, Asynchronization, performance optimization, and unit testing.
Problems
I have encountered many problems in development projects. In many cases, due to time constraints, I have not used an elegant solution. When talking with some friends in the industry, I also realized that some problems existed. The following is a simple example:
● Multithreading and concurrency
● Asynchronous message/interface call
● Message serialization and Reflection
● Performance Optimization
● Unit test
Multithreading and concurrency
In the multi-core era, concurrency can achieve higher throughput and faster response, but it is also a double-edged sword. Summarize the following usage:
● Multithreading + display lock; the interface is called by multiple threads. When called, the interface displays the lock and then operates the object data. The tragedy is that engineers design multiple locks to reduce the granularity of the locks, or even use atomic operations in some places. This adds an additional design burden to the domain logic. The worst case is a deadlock.
● Multithreading + task queue; interfaces are called in multiple threads, but requests are saved to the task queue, while the task queue is continuously executed by a single thread. Typical producer and consumer mode. Its concurrency is that different interfaces can use different task queues. This is also my most commonly used concurrency method.
This is two of the most common multi-thread concurrency, and they have a natural defect-Scalability. There is always a bottleneck on the performance of a machine. Although the logic of the two scenarios is concurrent by multiple threads, the calculation workload is likely to be unable to be carried by one machine. For multi-process concurrency, you can deploy it on another machine in a distributed manner (or on one machine ). Therefore, multi-process concurrency is more Scalability than multi-thread concurrency. In addition, after multi-process is adopted, each process is designed with a single thread, and such a program is more Simplicity. Other advantages of multi-process, such as decoupling, modularization, convenient debugging, and convenient reuse, will not be repeated.
Asynchronous Message/interface call
When talking about distributed communication, we need to talk about distributed communication technology. The common methods are as follows:
● RPC-like, including WebService, RPC, and ICE, features remote synchronous calls. Remote interfaces are very similar to local interfaces. However, game server programs are generally very concerned about latency and throughput, so these blocking threads are not commonly used for Synchronous remote calls. However, we must be aware of its advantages, that is, it is very helpful for calling and testing.
● Full asynchronous message: when a remote interface is called, a request message is sent asynchronously. After the interface responds, a result message is returned. The caller's callback function continues the logical operation to process the result message. Therefore, some logics are cut into ServiceStart and ServiceCallback segments. Sometimes, the logic of the domain is broken down Asynchronously. In addition, the message processing function usually writes a batch of switches/cases to process different messages. The biggest problem is unit testing. In this case, traditional unit testing is helpless.
Message serialization and Reflection
There are many methods to achieve message serialization and deserialization. common applications such as Struct, json, and Protobuff are successful. I personally prefer to use lightweight binary serialization. The advantage is that it is transparent and efficient, and everything is under control. Bin_encoder_t and bin_decoder_t lightweight message serialization are implemented in FFLIB, with dozens of lines of code.
Performance Optimization
I have written a summary about the performance. For more information, see
Http://www.cnblogs.com/zhiranok/archive/2012/06/06/cpp_perf.html
Some netizens mentioned tools such as profiler, cpuprofiler, and callgrind. I have used these tools. To be honest, for me, I agree that they have high value. First, they can only be used in the development and testing phase, and some performance reference data can be obtained initially. Second, how they achieve tracing is unknown. Running it slows down the program and does not reflect real data. Third, can the performance be the same in the development and testing phase as that after the release? Impossible!
The principle of performance is to talk about data. For more information, see the blog.
Unit Test
We have already talked about unit testing. Game server programs are generally large, but what is incredible is that you have never seen a project (c ++ background architecture) with a complete unit test. Due to the existence of asynchronous and multithreading, the traditional unit testing framework is not competent, and Development supports asynchronous testing framework is unrealistic. We must see that the traditional unit test framework has achieved great success. As far as I know, the game background using the web architecture has been very mature in unit testing and achieved extremely good results. So my idea is to use the existing unit test framework to adjust the asynchronous messaging and multithreading architecture.
I have talked about unit testing for many times. In fact, the idea of developing FFLIB comes largely from this, otherwise it may be just a c ++ network library. I decided to locate FFLIB in the framework when trying to solve this problem.
Let's take a look at a very simple unit test code:
1
Assert (2 = Add (1, 1 ));
Please allow me to explain this line of code and input parameters for the Add function to verify whether the returned value is the expected result. Isn't that the nature of unit testing? When thinking about the asynchronous message sending process, if each input message has a result message package, each request is sent with a callback function to receive and verify the result message package. This satisfies the traditional unit test steps. Finally, we need to solve the problem. Assert cannot process asynchronous return values. Fortunately, the future mechanism can be converted to asynchronous synchronization. For more information about the future model, see here:
Http://blog.chinaunix.net/uid-23093301-id-190969.html
Http://msdn.microsoft.com/zh-cn/library/dd764564.aspx#Y300
Let's take a look at the example of remotely calling the echo service in the FFLIB framework:

Struct lambda_t
{
Static void callback (echo_t: out_t & msg _)
{
Echo_t: in_t in;
In. value = "XXX_echo_test_XXX ";
Singleton_t <msg_bus_t>: instance ()
. Get_service_group ("echo ")
-> Get_service (1)-> async_call (in, & lambda_t: callback );
}
};
Echo_t: in_t in;
In. value = "XXX_echo_test_XXX ";
Singleton_t <msg_bus_t>: instance (). get_service_group ("echo")-> get_service (1)-> async_call (in, & lambda_t: callback );
When you need to call a remote interface, async_call (in, & lambda_t: callback); An asynchronous call must be bound to a callback function. The callback function receives the result message and triggers subsequent operations. In this case, if you perform unit tests on the remote echo interface, you can do the following:

Rpc_future_t <echo_t: out_t> rpc_future;
Echo_t: in_t in;
In. value = "XXX_echo_test_XXX ";
Const echo_t: out_t & out = rpc_future.call (
Singleton_t <msg_bus_t>: instance ()
. Get_service_group ("echo")-> get_service (1), in );
Assert (in. value = out. value );
In this way, all remote interfaces can be overwritten by unit tests.
FFLIB Introduction
FFLIB Structure

The Client does not directly connect to the Service, but completes message transmission through the Broker intermediate layer. For more information about Broker mode, see: http://blog.chinaunix.net/uid-23093301-id-90459.html
Inter-process communication uses TPC instead of multi-thread shared memory. A Service is generally a single-threaded architecture. By starting a multi-process architecture, you can achieve concurrency relative to multithreading. The Broker mode is inherently sharded, so it has a good Scalability.
Message Sequence Diagram

How to register services and interfaces
Let's take a look at the implementation of the Echo service:

Struct echo_service_t
{
Public:
Void echo (echo_t: in_t & in_msg _, rpc_callcack_t <echo_t: out_t> & cb _)
{
Logtrace (FF, "echo_service_t: echo done value <% s>", in_msg _. value. c_str ()));
Echo_t: out_t out;
Out. value = in_msg _. value;
Cb _ (out );
}
};
 
Int main (int argc, char * argv [])
{
Int g_index = 1;
If (argc> 1)
{
G_index = atoi (argv [1]);
}
Char buff [128];
Snprintf (buff, sizeof (buff), "tcp: // % s: % s", "127.0.0.1", "10241 ");
 
Msg_bus_t msg_bus;
Assert (0 = singleton_t <msg_bus_t>: instance (). open ("tcp: // 127.0.0.1: 10241") & "can't connnect to broker ");
 
Echo_service_t f;
 
Singleton_t <msg_bus_t>: instance (). create_service_group ("echo ");
Singleton_t <msg_bus_t>: instance (). create_service ("echo", g_index)
. Bind_service (& f)
. Reg (& echo_service_t: echo );
 
Signal_helper_t: wait ();
 
Singleton_t <msg_bus_t>: instance (). close ();
// Usleep (1000 );
Cout <"\ noh end \ n ";
Return 0;
}
● Create_service_group: Creates a service group. A service group may have multiple parallel instances.
● Create_service creates a service instance with a specific id
● Reg registers interfaces for this service
● The interface definition standard is void echo (echo_t: in_t & in_msg _, rpc_callcack_t <echo_t: out_t> & cb _). The first parameter is the input message struct, the second parameter is the template exception of the callback function. The template parameter is the struct type of the returned message. The interface does not need to know details such as message sending, but only needs to call back the result.
● After the Broker is registered, all clients can obtain the service.
Message definition Specification
We agree that each interface (either remote or local) should contain an input message and a result message. Let's take a look at the message definition of the echo service:

Struct echo_t
{
Struct in_t: public msg_ I
{
In_t ():
Msg_ I ("echo_t: in_t ")
{}
Virtual string encode ()
{
Return (init_encoder () <value). get_buff ();
}
Virtual void decode (const string & src_buff _)
{
Init_decoder (src_buff _)> value;
}
 
String value;
};
Struct out_t: public msg_ I
{
Out_t ():
Msg_ I ("echo_t: out_t ")
{}
Virtual string encode ()
{
Return (init_encoder () <value). get_buff ();
}
Virtual void decode (const string & src_buff _)
{
Init_decoder (src_buff _)> value;
}
 
String value;
};
};
● Each interface must contain in_t messages and out_t messages, which are defined inside the Interface Name (such as echo _ t ).
● All messages are inherited from msg_ I, which encapsulates binary serialization and deserialization. The type name is assigned as the message name during construction.
● Each message must implement the encode and decode functions.
It should be noted that, in FFLIB, the corresponding CMD does not need to be defined for each message. When an API such as echo registers with the Broker, the reg API automatically registers the msg name to the Broker through the type inference of the C ++ template. The Broker assigns a unique msg_id to each msg name. Msg_bus automatically maintains the msing between msg_name and msg_id. Msg_ I is defined as follows:

Struct msg_ I: public codec_ I
{
Msg_ I (const char * msg_name _):
Cmd (0 ),
Uuid (0 ),
Service_group_id (0 ),
Service_id (0 ),
Msg_id (0 ),
Msg_name (msg_name _)
{}
 
Void set (uint16_t group_id, uint16_t id _, uint32_t uuid _, uint16_t msg_id _)
{
Service_group_id = group_id;
Service_id = id _;
Uuid = uuid _;
Msg_id = msg_id _;
}
 
Uint16_t cmd;
Uint16_t get_group_id () const {return service_group_id ;}
Uint16_t get_service_id () const {return service_id ;}
Uint32_t get_uuid () const {return uuid ;}
 
Uint16_t get_msg_id () const {return msg_id ;}
Const string & get_name () const
{
If (msg_name.empty () = false)
{
Return msg_name;
}
Return singleton_t <msg_name_store_t>: instance (). id_to_name (this-> get_msg_id ());
}
 
Void set_uuid (uint32_t id _) {uuid = id _;}
Void set_msg_id (uint16_t id _) {msg_id = id _;}
Void set_sgid (uint16_t sgid _) {service_group_id = sgid _;}
Void set_sid (uint16_t sid _) {service_id = sid _;}
Uint32_t uuid;
Uint16_t service_group_id;
Uint16_t service_id;
Uint16_t msg_id;
String msg_name;
 
Virtual string encode (uint16_t cmd _)
{
This-> cmd = cmd _;
Return encode ();
}
Virtual string encode () = 0;
Bin_encoder_t & init_encoder ()
{
Return encoder. init (cmd) <uuid <service_group_id <service_id <msg_id;
}
Bin_encoder_t & init_encoder (uint16_t cmd _)
{
Return encoder. init (cmd _) <uuid <service_group_id <service_id <msg_id;
}
Bin_decoder_t & init_decoder (const string & buff _)
{
Return decoder. init (buff _)> uuid> service_group_id> service_id> msg_id;
}
Bin_decoder_t decoder;
Bin_encoder_t encoder;
};
Performance
Since a remote interface must be called through a Broker, brokerwill generate statistical data for each interface and output data to the perf.txt file every 10 minutes. The file format is CSV. For more information, see:
Http://www.cnblogs.com/zhiranok/archive/2012/06/06/cpp_perf.html
Summary
The FFLIB framework has the following features:
● Use multi-process concurrency. The Broker makes the location of the Client and Service transparent.
● The Service interface must be registered with the Broker. All clients connected to the Broker can call (publisher/subscriber)
● A callback function must be bound to a remote call.
● Use the future mode for synchronization to Support Unit Testing
● Simple and efficient message definition specifications
● All service interface performance monitoring data is automatically generated for free lunch
● Single-threaded Service, more simplicity
Source code:
Svn co http://ffown.googlecode.com/svn/trunk/
Running example:
● Cd example/broker & make &./app_broker-l http: // 127.0.0.1: 10241
● Cd example/echo_server & make &./app_echo_server
● Cd example/echo_client & make &./app_echo_client