NSQ: A distributed real-time messaging platform

This is a creation in Article, where the information may have evolved or changed.

NSQ is a distributed real-time messaging platform based on the Go language, which is based on the MIT Open Source protocol, which is hosted on GitHub and is currently the latest version of the 0.3.1 version. NSQ can be used for real-time messaging services in large-scale systems and can handle hundreds of millions of levels of messages per day, designed to provide a robust infrastructure for decentralized services running in a distributed environment. The NSQ has a distributed, decentralized topology that features no single point of failure, fault tolerance, high availability, and the ability to guarantee reliable message delivery. The NSQ is very easy to configure and deploy, with maximum flexibility and support for many message protocols. In addition, the official go-to and Python libraries are available for unboxing. If readers are interested in building their own client, you can also refer to the official protocol specification.

NSQ is comprised of four important components:

• NSQD: A daemon responsible for receiving, queuing, and forwarding messages to the client
• NSQLOOKUPD: Daemon that manages topology information and provides eventual consistency of discovery services
• Nsqadmin: A Web user interface to view cluster statistics and perform a variety of management tasks in real time
• Utilities: Common basic functions, data flow processing tools such as Nsq_stat, Nsq_tail, Nsq_to_file, Nsq_to_http, NSQ_TO_NSQ, TO_NSQ

The main features of NSQ are as follows:

• Distributed and no single point of failure topologies support horizontal scaling to seamlessly add cluster nodes without disruption
• Low-latency message push, see the official performance documentation
• Message routing with combined load-balancing and multicast forms
• Good at dealing with streaming (high throughput) workloads and working with job-oriented (low-throughput) workloads
• Message data can be stored in memory or stored on disk
• Producer, consumer auto-discovery and consumer auto-connect producers, see NSQLOOKUPD
• Supports Secure Transport Layer Protocol (TLS), which ensures message delivery security
• Message structure independent of data format, support JSON, Protocol buffers, Msgpacek and other message formats
• Very easy to deploy (almost no dependencies) and configuration (all parameters can be configured from the command line)
• Client function libraries with a simple TCP protocol and multiple languages
• HTTP interface For information statistics, Administrator operations, and implementation of producers, etc.
• Integrated statistical Data Collector for real-time detection STATSD
• With a powerful cluster management interface, see Nsqadmin

To achieve a highly efficient distributed messaging service, NSQ implements a reasonable and intelligent tradeoff that makes it perfectly suited to production environments, as follows:

• Supports the size setting of the message memory queue, which is fully persisted by default (value 0), and the message can be persisted to disk or saved in memory
• Ensure that messages are delivered at least once to ensure that messages can eventually be sent successfully
• Messages received are unordered and loosely ordered
• Discovery Service NSQLOOKUPD has eventual consistency, and messages can eventually find all topic producers

The official and third parties also developed numerous client function libraries for NSQ, such as the official HTTP-based NSQD, go client go-nsq, Python client pynsq, node. JS-based JavaScript client NSQJS, asynchronous C client LIBNSQ, Java client Nsq-java and numerous third-party client function libraries based on a variety of languages. For more client function libraries, please click here to view.

According to NSQ's design document, a single NSQD is designed to handle multiple streams of data at a time, and the data flow model in NSQ is composed of stream and consumer. Topic is a unique stream,channel that is a logical grouping of consumer that subscribes to a given topic. The NSQ data flow model structure is as follows:

As you can see, a single NSQD can have multiple topic, and each topic can have multiple channel. The channel is able to receive a copy of all messages from the topic, enabling the multicast distribution of messages, and each message on the channel is distributed to its subscribers for load balancing, all of which form a powerful framework that can represent a variety of simple and complex topologies.

NSQ was originally developed for applications that provide short-link services bitly. In addition, there are many well-known applications using NSQ, such as the social news website Digg, the private social application path, the famous open-source application container engine Docker, the payment company stripe, the news aggregator website BuzzFeed, the mobile app Life360 to see where the family is located, Network Tools company SimpleReach and so on.

Interested readers can also contribute code to NSQ, feedback questions, or experience the performance gains from NSQ for high-throughput network services through an official QuickStart tutorial. For NSQ installation, client development and other related information, please log on to their official website to view.

Thank Guo for the review of this article.