The skynet framework removes the message length limit of cluster rpc.

Why does the packet protocol provided by skynet only use two bytes to indicate the packet length?

Skynet provides a lua library netpack to parse the data in the tcp stream into a package with length + content. Even if skynet is used, however, I implemented another set of parsing protocols to parse external TCP data streams (for example, redis driver in skynet parses the redis server data streams by using line breaks to split packets), but many people still ask, can I customize the header length.

In the protocol defined in this library, the package length is expressed in two bytes of big-endian, that is, the length of a package cannot exceed 64 K. This makes many people very embarrassed. I have suggested several times to extend the length to 4 bytes, because most of the packages in their application environment will not exceed 64 KB, but a few of them will exceed this limit.

Historically, when skynet's gate was implemented using C (which version can still be used), it was indeed possible to customize whether to use 2 bytes or 4 bytes to indicate the length of the package. But after some consideration, I still removed this option.

A good library should be concise and guide users to do the right thing in the right way; instead of providing users with the opportunity to make mistakes. When communicating with the game client, if you only use one TCP connection, it is wrong to allow a long packet. Even 64 K is too large.

Games usually require fast response. If you allow inserting a large data block in a single TCP connection, such as 100 kB, therefore, it may take more than one minute to process the package in a relatively weak network condition (such as a mobile phone network. Such a large data block is not expected to be sent or received immediately in the business logic. A typical application scenario is that the user queries all the shelving items in the auction house. If all the returned data is put in a data packet, it will become very large. When you query a large number of such operations, the user is not expecting immediate response.

On a single TCP connection, such a big data block will block the entire channel, and all the data that needs to be delivered quickly will be delayed.

If you want to perform a Heartbeat packet on the business layer to check whether the network times out, you can easily block the Heartbeat packet. However, the network processing layer usually does not provide an interface to let the business layer know whether data is being accepted (skynet gateserver does not provide such an interface, although it is easy to provide, only a few lines of lua code need to be modified. Only after a complete data packet is received will it be handed over to the business layer for processing.

The correct method is to add a level to the protocol of length + content. Add a protocol called Big Data block, which allows sending a big data block in several segments. You can add the data block id in this protocol, and attach the data block id to the package that references the data block.

Why do we use this method instead of changing the header from 2 bytes to 4 bytes? When you do this design, it indicates that you have paid attention to the problems mentioned above.

When you divide data packets by hours, you can achieve the ability to carry multiple channels on a single TCP connection. For online games, not all data packets are contextual. You can think of multiple clues as hidden. For example, the information of the chat channel and the information synchronized in the scenario are independent of each other. Skynet also provides additional socket api support for this scenario. Socket. lwrite can write a string (a packet) to a low-priority channel. Only when all packages of the default channel (high-priority) are sent, the package on the low-priority channel will be sent at least one (a single package can ensure atomicity ).

For example, you can use it to send chat information, so that other important data packets will not be congested due to the flood of chat information. Likewise, you can use it to send split big data blocks. If you still have a lot of other important data to transmit to the client, these data blocks will be dispersed and interspersed.

Of course, you can also use lwrite to send all the data to the client, and only put the Heartbeat packet in the regular high-priority channel to ensure a more stable heartbeat frequency.

In addition, a four-byte packet length vulnerability may be exploited by attacks.

Generally, when receiving a packet header, the code of the split packet will be allocated a considerable amount of space in advance based on the length information, waiting for the subsequent data to be filled in. If attackers continuously send malicious length information on new connections, such as 2 GB, the server memory can be quickly consumed.

In the early days of skynet's gate implementation, the implementation of sharing a fixed-length ringbuffer was adopted to avoid such attacks. However, in the new version, no special processing is performed because the length of 4 bytes is no longer allowed.

If your application environment is very special, stick to a packet protocol that allows a larger length. So I suggest you implement a subcontracting module with caution, instead of simply changing 2 in the netpack library to 4.



Remove the message length limit of cluster rpc in skynet

The above explains why the packet protocol provided by skynet only uses two bytes to indicate the packet length. If there is a large message transfer requirement, it should be processed at the previous layer.

On the other hand, we should face up to the processing of long messages, instead of mixing them with the processing of common (short) messages, and the difference between the bottom layer.

Recently, the demand has come.

One of our new projects hopes to support large messages during inter-cluster communication. The originally asked student wanted to modify the skynet cluster module and modify the header length of the underlying protocol. Even if I stopped him, I made changes myself.

Simply put, the long message identifier is added to the cluster protocol at the previous layer. Because a byte was previously identified, the protocol was not significantly modified.

When a message (whether a request or a response) is too long, it is packaged separately and indicated on the identifier byte that this is only part of the complete message. Then merge the data in the receiver.

At first, I did not intend to modify the socket channel Module on which the cluster depends, so that each packet separately had an independent session. After half done, we found that it would be better to enhance the socket channel (instead of the previous interface. In addition, it is easy to use the low-priority queue feature of the previous socket.

I have submitted the new modification to an independent skynet branch named multipart. You are welcome to review it.

At the same time, I have redefined two internal macros with unclear meanings in skynet, removing the 16 M limitation on the internal message length.

After some simple tests, the current design goal can be achieved: within the cluster, rpc requests and response messages are no longer subject to the previous 64 K limit. In addition, once the message is too long (currently 32 kB), the transmission of long messages will be divided into multiple messages smaller than 32 kB for delivery, and other short messages can be interspersed in it. That is to say, if you use cluster to deliver a huge message of dozens of MB, the communication channel will not be blocked because the message is too large.


A memory leakage bug

The cause is an Issue of skynet. At the same time, a project we are developing over the past two days also seems to have leaked memory.

I think it is not normal for two things to happen at the same time, so I decided to check it out.

In fact, querying for memory leaks in skynet is much easier than normal projects. Because skynet is naturally divided into many small modules called services. The memory applied by the module is independent and has a high cohesion. The life cycle of a module is much shorter than that of the whole process. The module size is not too large and can be analyzed independently. Generally, if a memory application is not returned, it should be a bug in the C module. However, skynet rarely uses C modules. Once such a problem occurs, it can be quickly located.

Skynet implements a memory management hook to calculate memory overhead separately by service. The code is in the mallochook. c file under the skynetsrc directory.

I suggest you make the following changes:

When the memory is allocated and released each time, log the file name based on the handle of the current service. Here, we have considered a little multi-thread concurrency problem, but there is little probability of occurrence. If you only need to perform temporary debugging, you can ignore it temporarily. Because the allocation behavior of the same service is not concurrent, only a few cases are released in another thread. If this happens, it will lead to a bit of confusion during log writing.

The trouble is that the first write log in the hook to open the log file may lead to an endless loop caused by crt triggering malloc. To avoid this situation, you need to add a tls flag. After entering the hook, set the flag to avoid re-entry.

After more than 10 lines of code are added, the memory management log can work normally.

At first, I thought there was no problem with lua's memory management. Therefore, we can bypass the log (directly call je_malloc) of skynet_lalloc customized for lua in the malloc hook ). In this way, you can only record the memory management calls in the C module. The data volume is much smaller and it is easier to eliminate the problem.

However, after the log is added, the analysis log does not seem to find any leakage. Instead, I had to doubt whether the memory allocated through the lua distributor had been released.

There are two suspected points. One is to use lua_getallocf to obtain the call from the distributor. This was only found in lpeg and should have no problem. In addition, I modified lua myself.

To improve the memory usage of multiple lua VMS in skynet, I used to patch lua so that different lua VMS can share the same function prototype. The lua bytecode loaded for the first time will not be released. However, if the same code is loaded for 2nd times, the previous copy will be reused.

I carefully reviewed the patch and found no problems. Therefore, we still need to rely on new log analysis.

Next, I wrote the log again and changed it to record it in the lua custom distributor. This is much easier, just a few days ago. Some memory blocks with a length of 88 bytes are not released.

When the lua distributor allocates new memory, it will pass in the memory usage (for what type of data to use), which is an important clue. The type of the leaked 88 bytes is 0 (no record), but the type of the previous allocation record is 9, that is, LUA_TPROTO. This indicates that data in the function prototype is not released.

The 88 bytes is exactly the length of the topology proto structure added in my patch.

The final bug fix is simple: See this commit. A row is missing.

It's really a ghost.

Because the local code on my own hard disk has always been in this line: (I do not even remember to miss this line of code. (This patch was upgraded to lua 5.2 based on the patch of lua 5.3 earlier. The old version is correct .) Neither diff nor status of git noticed the differences between my local code and repository. This is why this memory leak has never happened on my own machine.

So I finally cloned a repository and submitted the missing code in this row. However, it is difficult to find a clue why this problem occurs when operating the git repository. Maybe it was caused by a forced push-f several months ago.