MQ framework for multiple persistence scenarios based on GO+PROTOBUF: Kiteq

This is a creation in Article, where the information may have evolved or changed. MQ Framework for multiple persistence scenarios based on GO+PROTOBUF: Kiteq

Gitlab found a relatively good open source project, hereby introduced:

Brief introduction

* 基于zk维护发送方、订阅方、broker订阅发送关系、支持水平、垂直方面的扩展* 基于与topic以及第二级messageType订阅消息* 基于mysql、文件存储方式多重持久层消息存储* 保证可靠异步投递* 支持两阶段提交分布式事务

Engineering structure

kiteq/├──readme.md├──binding subscription Relationship management handles the interaction with ZK ├──build.sh installation script ├──client Kiteq Client ├──doc document ├──handler Kiteq required processing handler├──kite_benchmark.go KITEQ program benchmark Iteq.go Kiteq External Launch portal ├──pipe similar to Netty pipeline structure of the framework, organizing event and handler transfer ├──protocol KITEQ protocol package, base The packet├──remoting network layer for PB and defined includes components required for Remoting-server and client and re-connected logic, ├──server Kiteq server-side Assembly ├──sta T state information Statistics └──store KITEQ storage structure  

Concept:

* Binding:订阅关系，描述订阅某种消息类型的数据结构* Consumer : 消息的消费方* Producer : 消息的发送方* Topic: 消息的主题比如 Trade则为消息主题，一般可以定义为某种业务类型* MessageType: 第二级别的消息类型，比如Trade下存在支付成功的pay-succ-200的消息类型

Architecture diagram

Zookeeper data structure

    KiteServer : /kiteq/server/${topic}/ip:port    Producer   : /kiteq/pub/${topic}/${groupId}/ip:port    Consumer   : /kiteq/sub/${topic}/${groupId}-bind/#$data(bind)

Process:

1. KiteQ启动会将自己可以接受和投递的Topics列表给到zookeeper2. KiteQ拉取Zookeeper上的Topics下的订阅关系(Bingding:订阅方推送上来的订阅消息信息)。3. Consumer推送自己需要订阅的Topic+messageType的消息的订阅关系(Binding)到Zookeeper4. Consumer拉取当前提供推送Topics消息的KiteQ地址列表，并发起TCP长连接5. Producer推送自己可以发布消息Topics列表到Zookeeper6. Producer拉取当前提供接受Topics消息的KiteQ地址列表，并发起TCP长连接

How to subscribe:

Direct (直接订阅)： 明确的Topic+MessageType订阅消息Regx(正则式订阅):  Topic级别下，对MessageType进行正则匹配方式订阅消息Fanout(广播式订阅): Topic级别下，订阅所有的MessageType的消息

Two-phase commit:

 Because of the introduction of asynchronous delivery scenarios, this message can be consumed by subscribers when it is necessary to perform a local execution of a transaction successfully in some scenarios.    For example: A successful customer purchase member payment success needs to modify the local user account MySQL balance, and inform the member system for the user's membership extension. This time will be met, must ensure that the MySQL operation is successful, the member system can receive the member extension of the message. For the above problems, Kiteq is treated like Ali's notify system, 1. Send a UNCOMMIT message to Kiteq, Kiteq will not make a delivery operation for the Uncommite message 2. Kiteq periodically sends a Txack inquiry to producer for the UNCOMMIT message 3. Until producer explicitly tells the commit or rollback the message 4. Commit will go through the normal delivery process, and rollback will roll back the current message and delete the operation. 

1. Compile: SH build.sh 2. Install Zookeeper: Omit start Kiteq:./kiteq-bind=172.30.3.124:13800-pport=13801-db= "Mock://kiteq"-topics= Trade,feed-zkhost=localhost:2181-bind//Bind Ip:port-pport//pprof http port-db//Storage protocol address mock://start mock mode mysq l://mmap://-topics///native can handle the topics list of comma-delimited-zkhost-//zk addresses to start the client: for Kiteqclient to implement a message listener, we define the following interface: Type ILIs Tener interface {//Accept the callback OnMessage (msg *protocol) of the posted message. Stringmessage) bool//Receive transaction callback//unless the explicit commit succeeds, the remainder is unsuccessful//has an exception or the return value is False is not committed Onmessagecheck (TX *pro Tocol. Txresponse) Error} start Producer:producer: = client. Newkiteqclient (${zkhost}, ${groupid}, ${password}, &defualtlistener{}) producer. Settopics ([]string{"Trade"}) producer. Start ()//Build message msg: = &protocol. stringmessage{} msg. Header = &protocol. header{Messageid:proto. String (store. MessageId ()), Topic:proto. String ("Trade"), Messagetype:proto. String ("Pay-succ"), ExpirEdtime:proto. Int64 (time. Now (). Unix ()), Deliverylimit:proto. Int32 ( -1), Groupid:proto. String ("Go-kite-test"), Commit:proto. Bool (True)} msg. Body = Proto. String ("echo")//Send message producer. Sendstringmessage (msg) starts the consumer:consumer:= client. Newkiteqclient (${zkhost}, ${groupid}, ${password}, &defualtlistener{}) consumer. Setbindings ([]*binding. binding{binding. Bind_direct ("S-mts-test", "Trade", "PAY-SUCC", "$", "true"), consumer. Start () to complete the ability to publish and subscribe to messages .....

Project GitHub address: Https://github.com/blackbeans/kiteq