Common open Source messaging systems

The role of messaging systems: asynchronous processing, cutting spikes, and reducing coupling between components.

Select the messaging system to consider the following aspects of your business needs:

1. is persisted
2. Throughput capacity
3. Highly Available
4. Distributed scaling capabilities
5. Compatible with existing protocols
6. Easy to maintain
7. Other, such as message loss and duplicate processing
8. Avoid single point of failure
9. Load Balancing

Common Message system protocols:

1. STOMP
2. Amqp
3. Protocol similar to MEMCACHE
4. HTTP
5. Custom Format

1, 2 is a good optional open source components: 1. Kafka/metaq: Widely used within Linkedin (similar to the Java version of the domestic Metaq)

Due to the priority of throughput, it is more suitable for message collection and processing of big data, such as log analysis, real-time report of user behavior information, gathering and analysis of cluster state information.

1. Prioritize persistent design and rely on page cache to manage memory
2. High throughput 112mb/s 11K msgs/s (more than BEANSTALKD >70x throughput capability)
3. Supports asynchronous replication
4. High-availability, Zookeeper-based cluster design, support for consumer-fail re-load balancing
5. Kafka provides PHP class library
6. Support for Ganglia JMX monitoring
7. The need for policies to avoid duplicate messages, the way Consumers Update Zookeeper offset (Metaq has provided several ways to avoid message duplication)
8. Metaq provides HTTP interface

http://www.mail-archive.com/[email protected]/msg02082.html
Https://github.com/neophenix/StateOfTheMQ/blob/master/state_of_the_mq.pdf?raw=true
Http://s.urge.omniti.net/i/content/slides/Surge2012-ErikOnen_Kafka_Messaging-Paradigms.pdf
Http://research.microsoft.com/en-us/um/people/srikanth/netdb11/netdb11papers/netdb11-final12.pdf
Http://dirlt.com/kafka.html
Http://dirlt.com/index.html
Http://wiki.secondlife.com/wiki/Message_Queue_Evaluation_Notes

2. Nsq–golang

No center design, automatic node registration and discovery. Can be considered as the basis for an internal communications framework.

Https://github.com/bitly/nsq
* Pursuit of simple deployment
* The pursuit of high availability, avoid single point of failure, no center design
* Ensure message delivery
* Producer consumers automatically discover, consumers connect to all producers, push to consumer mode
* HTTP Interface available

Https://speakerdeck.com/snakes/nsq-nyc-golang-meetup
https://github.com/davegardnerisme/nsqphp

http://www.davegardner.me.uk/blog/tag/nsq/

3. BEANSTALKD

1. Supports persistent binlog design, no loss of restart message
2. So so
3. No high-availability design
4. The same distributed extension approach as memcached
5. Various class libraries
6. Have Web management tools
7. Supports synchronous calls and waits for return
8. Only similar to Memcache TCP ASCII protocol, single file deployment
9. Support Message priority
10. 9K jobs/s into queue 5K jobs/s out queue
11. Single point of failure
12. No master-slave synchronous replication mechanism
13. Best standalone Multi-instance deployment

Https://github.com/kr/beanstalkd/wiki/Tools
Https://github.com/pda/pheanstalk

4. Redis

Need to package pub/sub yourself

• Redis-based replication high availability

Other common open source messaging systems: ZeroMQ: Lightweight Base Message Library

Suitable for scenarios that do not need to be persisted and that require their own encapsulation

1. does not support persistence, only provides message distribution, the best performance
2. No Broker design, no center fault

RabbitMQ

• 2500 job/s into queue 1300 job/s out queue
• Suitable for small messages
• Distributed no single point design
• The bottom layer is the Erlang implementation
  There are comments: RabbitMQ could not enqueue/dequeue fast enough.

Https://blogs.vmware.com/vfabric/2013/04/how-fast-is-a-rabbit-basic-rabbitmq-performance-benchmarks.html

Restmq

http://restmq.com/

Memcacheq

http://memcachedb.org/memcacheq/

Httpsqs

https://code.google.com/p/httpsqs/

Gearman

Http://gearman.org/presentations
https://code.google.com/p/shard-query/

Kestrel

http://robey.github.io/kestrel/
Http://robey.github.io/kestrel/docs/guide.html

HornetQ

Poor performance not considered [3]

Resque

3800 jobs/s into queue jobs/s out queue
Https://github.com/blog/542-introducing-resque
Redis-based Message Queuing

Starling

Https://github.com/starling/starling

Squirrelmq

https://code.google.com/p/squirrel-message-queue/

Sparrow–ruby

https://code.google.com/p/sparrow/

Apache ActiveMQ

ActiveMQ crashed constantly under load.

STOMP HTTP Protocol

Http://stomp.github.io/stomp-specification-1.2.html

Reference:

Http://hiramchirino.com/stomp-benchmark/ec2-c1.xlarge/index.html
https://blog.serverdensity.com/queueing-mongodb-using-mongodb/
[3] http://x-aeon.com/wp/2013/04/10/a-quick-message-queue-benchmark-activemq-rabbitmq-hornetq-qpid-apollo/
https://news.ycombinator.com/item?id=5531192
http://www.javaplex.com/blog/high-performance-message-queues-get-reviewed/
Http://adam.heroku.com/past/2010/4/24/beanstalkasimpleand fastqueueingbackend/

Related articles

Common open Source messaging systems