The role of messaging systems: asynchronous processing, cutting spikes, and reducing coupling between components.
Select the messaging system to consider the following aspects of your business needs:
- is persisted
- Throughput capacity
- Highly Available
- Distributed scaling capabilities
- Compatible with existing protocols
- Easy to maintain
- Other, such as message loss and duplicate processing
- Avoid single point of failure
- Load Balancing
Common Message system protocols:
- STOMP
- Amqp
- Protocol similar to MEMCACHE
- HTTP
- Custom Format
1, 2 is a good optional open source components: 1. Kafka/metaq: Widely used within Linkedin (similar to the Java version of the domestic Metaq)
Due to the priority of throughput, it is more suitable for message collection and processing of big data, such as log analysis, real-time report of user behavior information, gathering and analysis of cluster state information.
- Prioritize persistent design and rely on page cache to manage memory
- High throughput 112mb/s 11K msgs/s (more than BEANSTALKD >70x throughput capability)
- Supports asynchronous replication
- High-availability, Zookeeper-based cluster design, support for consumer-fail re-load balancing
- Kafka provides PHP class library
- Support for Ganglia JMX monitoring
- The need for policies to avoid duplicate messages, the way Consumers Update Zookeeper offset (Metaq has provided several ways to avoid message duplication)
- Metaq provides HTTP interface
http://www.mail-archive.com/[email protected]/msg02082.html
Https://github.com/neophenix/StateOfTheMQ/blob/master/state_of_the_mq.pdf?raw=true
Http://s.urge.omniti.net/i/content/slides/Surge2012-ErikOnen_Kafka_Messaging-Paradigms.pdf
Http://research.microsoft.com/en-us/um/people/srikanth/netdb11/netdb11papers/netdb11-final12.pdf
Http://dirlt.com/kafka.html
Http://dirlt.com/index.html
Http://wiki.secondlife.com/wiki/Message_Queue_Evaluation_Notes
2. Nsq–golang
No center design, automatic node registration and discovery. Can be considered as the basis for an internal communications framework.
Https://github.com/bitly/nsq
* Pursuit of simple deployment
* The pursuit of high availability, avoid single point of failure, no center design
* Ensure message delivery
* Producer consumers automatically discover, consumers connect to all producers, push to consumer mode
* HTTP Interface available
Https://speakerdeck.com/snakes/nsq-nyc-golang-meetup
https://github.com/davegardnerisme/nsqphp
http://www.davegardner.me.uk/blog/tag/nsq/
3. BEANSTALKD
- Supports persistent binlog design, no loss of restart message
- So so
- No high-availability design
- The same distributed extension approach as memcached
- Various class libraries
- Have Web management tools
- Supports synchronous calls and waits for return
- Only similar to Memcache TCP ASCII protocol, single file deployment
- Support Message priority
- 9K jobs/s into queue 5K jobs/s out queue
- Single point of failure
- No master-slave synchronous replication mechanism
- Best standalone Multi-instance deployment
Https://github.com/kr/beanstalkd/wiki/Tools
Https://github.com/pda/pheanstalk
4. Redis
Need to package pub/sub yourself
- Redis-based replication high availability
Other common open source messaging systems: ZeroMQ: Lightweight Base Message Library
Suitable for scenarios that do not need to be persisted and that require their own encapsulation
- does not support persistence, only provides message distribution, the best performance
- No Broker design, no center fault
RabbitMQ
- 2500 job/s into queue 1300 job/s out queue
- Suitable for small messages
- Distributed no single point design
- The bottom layer is the Erlang implementation
There are comments: RabbitMQ could not enqueue/dequeue fast enough.
Https://blogs.vmware.com/vfabric/2013/04/how-fast-is-a-rabbit-basic-rabbitmq-performance-benchmarks.html
Restmq
http://restmq.com/
Memcacheq
http://memcachedb.org/memcacheq/
Httpsqs
https://code.google.com/p/httpsqs/
Gearman
Http://gearman.org/presentations
https://code.google.com/p/shard-query/
Kestrel
http://robey.github.io/kestrel/
Http://robey.github.io/kestrel/docs/guide.html
HornetQ
Poor performance not considered [3]
Resque
3800 jobs/s into queue jobs/s out queue
Https://github.com/blog/542-introducing-resque
Redis-based Message Queuing
Starling
Https://github.com/starling/starling
Squirrelmq
https://code.google.com/p/squirrel-message-queue/
Sparrow–ruby
https://code.google.com/p/sparrow/
Apache ActiveMQ
ActiveMQ crashed constantly under load.
STOMP HTTP Protocol
Http://stomp.github.io/stomp-specification-1.2.html
Reference:
Http://hiramchirino.com/stomp-benchmark/ec2-c1.xlarge/index.html
https://blog.serverdensity.com/queueing-mongodb-using-mongodb/
[3] http://x-aeon.com/wp/2013/04/10/a-quick-message-queue-benchmark-activemq-rabbitmq-hornetq-qpid-apollo/
https://news.ycombinator.com/item?id=5531192
http://www.javaplex.com/blog/high-performance-message-queues-get-reviewed/
Http://adam.heroku.com/past/2010/4/24/beanstalkasimpleand fastqueueingbackend/
Related articles
Common open Source messaging systems