BEANSTALKD a high performance distributed memory queuing system

The popular queue framework is roughly: Memcacheq,fqueue, RabbitMQ, BEANSTALKD, and LinkedIn's Kafka. RABBITMQ use more widely, BEANSTALKD is a rising star. BEANSTALKD to RABBITMQ, like Nginx in apache,varnish to squid. Later in the process of using BEANSTALKD in the project, it is found that its simple, lightweight, high performance, easy to use and other characteristics, as well as priority, multi-queue, persistent, distributed fault-tolerant, time-out control and other characteristics. Here is a brief introduction to BEANSTALKD.

Design ideas

High performance can not be separated from the asynchronous, asynchronous inseparable from the queue, and its interior is the principle of Producer-comsumer mode.

Figure 1 Producer-comsumer Mode

Application

BEANSTALKD, a high-performance, lightweight distributed memory queuing system, was originally designed to perform time-consuming tasks asynchronously in the background to reduce page access latency for high-capacity Web applications and to support Facebook causes apps with 9.5 million users. Later open source, now has postrank large-scale deployment and use, daily processing millions tasks. BEANSTALKD is a typical class memcached design, protocols and usage are the same style, so users who have used memcached will feel beanstalkd familiar.

Core Concepts

The core concept of BEANSTALKD design:

Job

A task that requires asynchronous processing is the basic unit in BEANSTALKD, which needs to be placed in a tube.

Tube

A well-known task queue, used to store a uniform type of job, is an object of producer and consumer operations.

Producer

Job producer, put a job into a tube with the put command.

Consumer

The job's consumer, through the Reserve/release/bury/delete command to obtain the job or change the status of the job.

BEANSTALKD is shown in 2. A job has ready, RESERVED, DELAYED, buried four states. When producer puts a job directly, the job is in the ready state, waits for consumer to process it, and if the delay Put,job is selected first to delayed state, the wait time is migrated to ready state. After consumer acquires the current ready job, the job's status is migrated to reserved so that the other consumer can no longer manipulate the job. When consumer completes the job, it can choose Delete, release or Bury operation; After delete, the job is extinct from the system and can no longer be acquired; the release operation re-migrates the job state back to Ready ( You can also defer the state migration operation) so that other consumer can continue to acquire and execute the job; it is interesting that the Bury operation can hibernate the job, wait until it is needed, and then kick the dormant job back to the ready state, or delete Buried the status of the job. It is this interesting operation and state that can be based on this to make a lot of meaning of the application, such as the implementation of a circular queue, you can reserved the status of the job to hibernate, and so on without the ready state of the job, and then the buried state of the job once kick back to the ready state.

Figure 2 Life cycle of the job in BEANSTALKD

Characteristics

BEANSTALKD based on the source code installation and use is very simple, this skip. Here are some of the nice features that are highlighted here.

Priority level

Support 0 to 2**32 priority, the lower the value, the higher the priority, the default priority is 1024.

Persistence of

The job and its status can be logged to the file via Binlog, and the previous job and status can be restored by reading Binlog the next time BEANSTALKD is started.

Distributed fault tolerance

Distributed design is similar to memcached, BEANSTALKD each server does not know each other's existence, is through the client to implement the distributed and according to the tube name to go to the specific server to obtain the job.

Timeout control

To prevent a consumer from taking up a task for a long time but cannot handle it, BEANSTALKD sets a timeout time for the reserve operation, and if the consumer cannot complete job,job within a specified time, it will be migrated back to the ready state. For other consumer execution.

Insufficient

Found in the use of a BEANSTALKD has not yet provided the removal of a tube operation, can only be tube job in turn delete, and let beanstalkd from delete empty tube. There is also the BEANSTALKD does not support the client authentication mechanism (the developer locates the application scene in the local area network).

BEANSTALKD a high performance distributed memory queuing system