Goim Source Analysis 1-comet Module

This is a creation in Article, where the information may have evolved or changed.

Goim is Mao Jian classmate writes the IM service, Pure Golang realizes, currently applies in the Bilibili product line. Recently wrote two projects anti-spam and advertising systems, are internal services, and long-connected IM class is very unfamiliar to me. Spare time study Goim code, quite inspired, such as layered design, data merge, timer optimization, object optimization and RPC implementation optimization, basically commonly used components are GOIM rewritten or modified. Make a note, analyze the business code first, optimize the code to put in the latter few, and share with you, Pat, light abuse ...

Comet role Positioning

Goim Overall architecture

Throughout the architecture, the system is divided into Comet, Logic, Job, Router four modules, each module through RPC communication, refer to the official Chinese document, Comet program is the connection layer, exposed to the public network, all the business processing pushed to Logic module, through RPC communication. The advantage of this design is that the long connection logic is seldom changed, the stable public connection is maintained, the backend logic, the Router module changes frequently, and the restart does not affect the connection layer.

A few important structural bodies

As a typical code that is annotated open source project, Goim Basic not too much reading obstacles, a few logic points under the comb will soon understand.

Buckets: Each comet program has several buckets that can be understood as Session Management, preserving which rooms and Channel the current comet serves. Which Bucket is the long connection specifically distributed on? Choose according to subkey consistency Hash .

Room: Can be understood as rooms, groups or a group. This room maintains N Channel, which is a long connected user. Broadcasts a message within the room and sends it to all the Channel in the house.

Channel: Maintenance of a long connected user, can only correspond to a single. Messages that you push can be broadcast in the room or pushed to the specified Channel.

Proto: Message structure, store version number, operation type, message sequence number and message body.

Multi-protocol support

Goim supports TCP, Http, WebSocket, TLS WebSocket. Very powerful, the underlying principle is the same, the following analysis is based on the TCP protocol.

Buckets

Let's take a look at the definition of structure.

Bucket Structural body

The definition is clear, maintaining the current message channel and room information, the method is also very simple, plus reduce the channel and rooms. A Comet Server opens the Bucket.clock by default, which has the advantage of reducing lock contention, which is especially noticeable in large concurrent operations.

The

The housing structure

The rooms structure is slightly more complex, not only to maintain the message channel channels, but also the message broadcast merge write, that is, Batch write, if not merge write, each small message is written through a long connection, the system Syscall call overhead will be very large, Pprof When you see the network Syscall is the big head.

Message Broadcast

Logic Server sends the broadcast message to Room.push through RPC calls, the data is temporarily present Vers, OPS, msgs, each room initiates a groutine to process the staged message at initialization, up to batch Num number Volume or delay for a certain amount of time, the message is batched to the channel message path.

Channel

Channel Structural body

Always feel that the name of the Session is more intuitive, and not the language level of the "channel" conflict. Writer/reader is the encapsulation of the network Conn, Svrproto is a Ring Buffer, to save the radio or directly sent over the message body.

Message Flow

Only Comet code is analyzed here, so message generation is not mentioned

1. Client connects to Comet Server, handshake Authentication

2. Create a new channel for the current long connection, because the Comet service does not handle business logic, RPC is required to go to logic Server to obtain subscription information for that channel. At the same time the Channel opens a dispatchtcp groutine, blocking waits for the Ring Buffer data to be available and sent to the Client.

3. The Logic service passes the RPC, writes the message to the bedroom (broadcast) or writes directly to the specified Channel (unicast). Note here that the broadcast is a write merge BatchWrite, and unicast is not, and the message is sent immediately after it is generated.

4. The broadcast message in the room reaches a certain number of Batch Num, or the delay waits for a certain amount of time to write the message to Channel Ring Buffer.

Summary

Goim layering is a reasonable detail, which is just a business-level interpretation of Comet Server. There are also many performance optimizations, such as bytes. Pool, time. Timer, RPC and other optimizations to stay at the end of slowly sharing it.

We have fun!

Supplement-From Mao Jian

1. Bucket According to the cityhash of key to find the remainder hit, no consistent hash, because there is no migration involved

2. Private messages are actually merged, unlike the combination of the Ringbuffer, when the message is hungry, the real flush

3. There is also an optimization can be improved, because the rooms have a feature of the message may be the same, so in the rooms in advance to merge into byte buffer, and then broadcast everyone, to avoid everyone serialization once, and then use the GC to handle the release of the buffer, which can save a lot of CPU, This optimization hasn't been done yet.