Go-implemented microblogging Message Queuing

This is a creation in Article, where the information may have evolved or changed.

Interested to see the implementation of Message Queuing principle, please visit https://github.com/YoungPioneers/mgq, thank you for your valuable comments

Introduction

Memcached Go queue, abbreviated as MGQ, is a message queue written in the go language, based on the Memcached protocol. Its Father McQ is the first basic message middleware applied to Weibo, which has the advantages of high performance and decoupling, which makes it widely used in Weibo

Features

MGQ is a FIFO message queue based on a NoSQL database BerkeleyDB, which currently supports the following features:

• Write multiple reads: For example, set Myqueue ' one message ', as long as get, myqueue start, #分隔, such as myqueue#1, read between multiple clients is independent of each other, is unaffected by

• The default get is the oldest message not read in the queue

• Supports GETC operation, which supports obtaining data for one of the cursor locations in the queue: for example, assuming that Myqueue already has 1000 data, getc myqueue 99, you can get the message with cursor 99 in the queue.

• Support for GETR operations, to obtain data from the start of a start cursor position in a queue, to end: For example, suppose Myqueue already has 20 data, Getr myqueue 1 10, you can get the message "cursor is [1-10] in the queue" In beta "

• GETN supports the timeout mechanism's blocking API to get the latest messages in the queue, for example: GETN queue 10 means that the data in 10s is returned immediately, otherwise it will return immediately after 10s, the default getn timeout is 0s, Never timeout (note that if the client has an GETN operation, another client of set needs to call SETN)

Benchmark

For the loss rate of the message, I did a single set and get test, the following is the result of the message number is 30w,50w,100w

Message total:30w

mgq message set total:300000, cost total time:123518222205 ns, 2428 per/s ,fail set total:0mgq message get total:300000, cost total time:51103619703 ns, 5870 per/s ,fail get total:0

Message total:50w

mgq message set total:500000, cost total time:210480212729 ns, 2375 per/s ,fail set total:0mgq message get total:500000, cost total time:87694742059 ns, 5701 per/s ,fail get total:0

Message total:100w

mgq message set total:1000000, cost total time:422339921379 ns, 2367 per/s ,fail set total:0mgq message get total:1000000, cost total time:173768683759 ns, 5754 per/s ,fail get total:0

At the same time, a simple pressure test, the following results are 1,2,3,4 routine, the relative average time for a single set and get, Benchmark code

Benchmark_MgqMultiSetAndGet-4         2000        546617 ns/op (1829 per/s)Benchmark_MgqMultiSetAndGet-4         2000        583259 ns/op (1714 per/s)Benchmark_MgqMultiSetAndGet-4         2000        723603 ns/op (1381 per/s)Benchmark_MgqMultiSetAndGet-4         2000        754741 ns/op (1324 per/s)