A lightweight asynchronous task executor based on Redis and Disque implementations

Brief introduction

Horae is an redis disque asynchronous task executor based on and implemented, 轻量级 高性能 its core is to disque provide the task queue, and the queue has 先进先出 the timing relationship, gu name: horae .

Horae : The goddess of timing, Greek mythology, the three goddess of seasonal time and human order, and the "Dutch".

Horae's focus is not on the implementation of the queue service itself (there are already many implementations of the queue service), but rather the redis disque implementation of an asynchronous task executor with the help of a high-performance queue mechanism with pure memory provided. It is free to configure which queue service The task comes from, which does not focus on the final state of the task execution (where it is written) or on which system it interacts with, it gives you an executor and a simple way to write the task execution logic.

Depending on the requirements, this actuator requires a single-node Redis server to operate when the requirement is not high.

If you are willing to sacrifice a bit of performance in exchange for higher queue reliability Assurance (in which case I strongly recommend that you use the AMQP protocol and its open source queue implementation: RabbitMQ ). If you want to do this, then the actuator is also available, but you need to implement the details of interacting with rabbitmq yourself. You can use it to connect a variety of other queues to consume messages and perform tasks that have full extensibility and freedom. But I still recommend you to use disque .

Applicable scenario snapping/seconds kill

Snapping up business is a typical short-term, high-concurrency scenario, similar to what 学生选课 can be attributed to traditional industries.

Social relationship Processing

A pure memory Computing/counter scenario, such as moving friends and relationships from a social system to in-memory processing.

Time-consuming Web requests

Common time-consuming Web requests, such as,,, 生成PDF 网页抓取 , and 数据备份 邮件/短信发送 so on.

Distributed system front-end buffer queue

After it is placed on the application server, the core service is used as the buffer queue for the request.

To sum up is,, 服务器峰值扛压 异步处理 , of course, 纯内存计算 you can use it as a normal queue is also possible.

Performance

Currently support disque with both redis queue services (main push disque , redis the queue temporarily with list the data interface lpush & brpop implementation, but it is not highly reliable, and there is no ACK mechanism). These two pure-memory queues first guarantee the performance of the consuming task. The execution performance of specific tasks, depending on the usage scenario, analyzes two scenarios here:

Pure Memory & Single thread & No lock

If the message consumed by the task processor is completely stored in memory, then it is necessary to isolate the data accessed by the isomorphic tasks (the means of isolation is to partition the key namespace), and if there is no way to isolate it, single-queue single-threaded, no-lock processing is possible.

General & Multi-threaded & Multi-queue

If it is a common application scenario, such as accessing the database, because the database has a mature data consistency guarantee. So you can divide tasks into different queues and execute concurrently with multiple threads to speed up the processing of tasks.

Of course, the most recommended way to use is: redis as a configuration, coordination, control center, with the disque queue service, the task needs to access the data as far as possible to store redis in.

High Availability one master multi-slave

The actuator executes at run time: Master single node runs, multiple slave do standby mechanism to guarantee the service availability. In fact, from the master downline to one of the slave, a successful campaign for master requires several heartbeat cycles of time. Because the consumer of the executor as a queue is fully decoupled from the queue, a short pause in consumption does not have much impact on the availability of the entire system.

Heartbeat mechanism

Master and slave through redis-Pubsub to maintain the heartbeat. The current design is master one publish -way Heartbeat, Slave subscribe Master's heartbeat. The reason for this design is simple, and given that each slave is a stateless actuator and does not involve state maintenance and synchronization issues, master does not need to care about slave's survival.

Competition Master

Once the master is offline (for example, because of a failure), it is necessary to quickly elect a new master from multiple slave, and the algorithm is very complex.

Usually the way to elect master is done by a separate Manager role-taking node, and if there is no such node, it is usually implemented based on a distributed election algorithm ( Zookeeper which is better at this). It is easy to preempt master with a similar implementation of competing distributed locks.

How can I tell if Master is offline? This is a very critical issue, because if a miscalculation occurs, it will result in a gap period for the overall system service, which is a significant time overhead. The method of judging is double detection :

• Slave subscribe to Master's heartbeat channel to determine if the heartbeat is timed out
• Slave go to Master's data structure to get Master's own refreshed heartbeat timestamp, and compare with the current time to determine whether to timeout

Specific implementation: Each service will have a heartbeat thread, Master's heartbeat thread do two things:

• Refresh your heartbeat time stamp
• Publish his heartbeat to the heartbeat channel.

Slave heartbeat thread To do the above double detection , slave will wait for a few heart cycles, if during this time, both detection is that master lost heartbeat, then Judge Master offline.

Master offline, it involves a number of slave competition master problem, here we do not use the blocking wait in the competition lock, but the use of a relatively small risk way: tryLock :

    Private Boolean Trylockmaster() {LongCurrentTime = Redisconfigutil.redistime (configcontext); String val = string.valueof (currenttime + constants.default_master_lock_timeout +1); Jedis Jedis =NULL;Try{Jedis = Redisconfigutil.getjedis (configcontext). get ();BooleanLocked = JEDIS.SETNX (Constants.key_lock_for_master, val) = =1;if(locked) {Jedis.expire (Constants.key_lock_for_master, Constants.default_master_lock_tim Eout);return true; }        }finally{if(Jedis! =NULL) Redisconfigutil.returnresource (Jedis); }return false; }

Only after judging the master offline, tryLockMaster It is only to try to obtain a lock, if successful, will give the lock set a very short expiration time, which is the same as the heartbeat expiration time. If the acquisition fails, the heartbeat continues to be detected. The slave that acquires the lock becomes master immediately and refreshes its heartbeat quickly, so that the other slave detects that the master offline fails and will not be called again tryLockMaster . Avoid the usual situation, always blocking, competition lock this road.

Extensibility Extension Features

Thanks to Redis PubSub , we can implement many similar 指令下发->执行 feature, such as getting the execution progress of the task in real time, and letting the servers report their status. Because of the time relationship, this block is just left with an expansion port:

• Uplink channel: The actuator has a upstream channel for uploading the native information of each node.
• Downlink Channel: The system has a downstream channel for passively accepting information/instructions from upstream.

Here the semantics of upstream and downstream are: all service nodes are downstream, the redis Configuration center should be a central node, upstream you can customize a control console for the management redis of the configuration center and the downstream service node issued instructions.

Extended Queue Service

If you want to extend it, you want it to support another queue service (for convenience, suppose you want to support RABBITMQ). Then you need to do the following steps:

• com.github.horae.exchangernew class under Package: Packages RabbitConnManager to manage client-to-RABBITMQ connections
• Also com.github.horae.exchanger under package: New class: The RabbitExchanger queue and queue logic for implementing the message, the class needs to implementTaskExchanger
• TaskExchangerManagerAdd a createTaskExchanger new branch judgment within the method.
• The partition.properties new partition can be configured under, specifying RABBITMQ in the matrix

Note that: TaskExchanger the dequeue interface method, the default behavior is block the form. If your extended queue does not support block-style consumption, you may need to implement it yourself, in a way that can be implemented java.util.concurrent.BlockingQueue .

Multiple reliability levels

The reliability of the queue is related to the reliability of the whole distributed system, which is an unavoidable problem. If you say that redis the implementation of the queue can be done to both maintain 高性能 and can both 高可靠 , the answer is 不能 . Or it's not a professional queuing service (otherwise the Redis author has no need to start another disque project). If, from a reliability standpoint, I rank several major queue servers (or can provide a queue service): RabbitMQ > Kafka > Disque > Redis . Although this actuator is built to support disque and redis be implemented as a queue, it does not have a very tight coupling with the queue service you choose, you can choose other queue services, usually you only need to implement a few functions,,, 入队消息 出队消息 ack消息 管理连接 .

Partition

The concept for me 分区 comes from Kafka, but the zoning here is not the same as the Kafka nature. First, let's see why there is such a demand?

As a stateless service, it can run for a long time (in a way, it's a bit like storm) without having to go offline. In order to fully extract the value of the CPU. We may want to let it run multiple heterogeneous services during the lifetime of a service (so-called heterogeneous tasks, which are tasks of different nature). Therefore, it is necessary to distinguish between multiple heterogeneous tasks, and this means 分区 . The reason it differs from Kafka is that it is more of a logical division than Kafka physically storing messages by partition. Let's see what a partition isolates:

Partition. Root=p0,p1p0. Matrix=redisp0. Host=127.0. 0. 1P0. Port=6379P0. Class=com. GitHub. Horae. Task. RedistaskP1. Matrix=disquep1. Host=127.0. 0. 1P1. Port=7711P1. Class=com. GitHub. Horae. Task. Disquetask

• Matrix: What kind of queue implementation service, currently supported disque /redis
• Host: Host of the queue Server
• Port: Port of the queue Server
• Class: Fully qualified name of the implementation class that handles the queue task

From the above isolation approach, the partition here can also be done to the task queue physical isolation. There are two partitions configured above, and two partitions correspond to two queue services respectively. There is no limit to the correspondence between partitions and the queue service, and even multiple partitions for a single queue server are possible, because there is also a mapping relationship for the partition to the queue name:

Such as:

Summary: Partitions isolate the queues for heterogeneous tasks, and the queue service in which queues are stored, where they are stored, and the processing logic of the tasks depend entirely on the configuration.

The above parsing clearly clarifies the correspondence between partitions and task-processing classes. For ease of administration, a partition also has its own line pool to isolate the threads of heterogeneous tasks.

Writing Task handlers

Before you write a task processor, you should be aware that the task processor you are writing is acting as a 队列的消费者 . The next thing you need to know is that the task processor you write will run in a pool of threads, and the management of thread pools requires you to be concerned, but you need to know: 一个任务队列将会对应一个线程 . That's all you need to know, write a task handler below:

• First you need to create a new MAVEN project
• In the library directory of the Horae release package ( ./horae/libs ) horae , locate the jar file that starts with it and add it to your maven dependency, just a local dependency:

        <dependency>            <groupId>Com.github.horae</groupId>            <artifactid>Horae</artifactid>            <version>0.1.0</version>            <scope>System</Scope>            <systempath>/usr/local/horae/libs/horae-0.1.0.jar</systempath>        </Dependency>

• You need to create a new class, Inherit TaskTemplate , and implement run the method, and here is a template:

    publicvoidrun() {        try {            signal.await();            //implement task process business logic        catch (InterruptedException e) {        }    }

• To write a construction method:

    StringMap<String, Object> configContext,                         Partition{        super(signal, queueName, configContext, partition, taskExchanger);    }

In the first sentence of the Run method, you need to invoke an CountDownLatch instance of the await method to block it. Explain, why do you need to do this?

In fact, each service starts by reading the queue that is configured in Redis and initializes the thread pool to the execution readiness State. This step, all the services, both master,slave are the same. But the difference is different in this sentence:

signal.await();

When the master node is started, the signal immediately releases the signal (through signal.countDown() ) and all the task handlers start executing immediately.

While the slave node is started, it will always block the above code until the master is offline, and after the node competes to master, the unblocking signal is immediately released and subsequent code executes immediately.

So doing so makes it possible for all slave to enter the task execution state at the fastest speed after the master is offline, although for some slave nodes, this is a waste of system resources.

• Compile the project and package the jar, taking care not to include the above Maven dependency, which already exists in the horae executable class library.

• Place the generated jar ./horae/libs/ below and it will be automatically added to the classpath

• Edit the configuration file ./horae/conf/partition.properties , new/Modify a partition p{x}.class , and the value is the fully qualified name of the task implementation class you just wrote.

Installation deployment

The following installation steps are verified by the Mac OS X system (similar to the Linux system, but there are some differences). Mac users need pre-installedHomebrew

• Installing JSVC

install jsvc

• Installing Redis

install redis

• Installing Disque

Because Disque currently does not have a stable version, so temporarily by the homebrew Head-only Warehouse, the installation command slightly different:

brewinstall--HEADhomebrew/head-only/disque

• horaeSource code compilation, packaging

assembly:assembly

• Copy the package file to the destination folder and unzip

${project_baseDir}/target/horae.zip /usr/localunzip /usr/local/horae.zip

• Configure executables, primarily commands and paths

sudo vi /usr/local/horae/bin/horae.sh

• Configure the configuration file under Conf

sudo vi /usr/local/horae/conf/${service/redisConf/partition}.properties

• Execute command

sudo${start/stop/restart}

Precautions

• Only one of the configuration items in the service.properties under Conf is master set to true in all nodes. If it is offline, it will not start again as master.
• Because jsvc need to write the process number (PID), so as far as possible to execute as a system administrator, will be configured in horae.sh user root , and to sudo execute

About Disque

Currently Disque is still in alpha version and commands are still being adjusted. Although it is supported, both the Disque server and its Java client: jedisque There are bugs, so it is not recommended for the time being, at least wait until you release the stable version.

A self-implementing jedisque connection pool. Currently, the Jedisque client has not provided a connection pooling mechanism, which is written by the same developer as the mainstream Java client of Redis jedis . Considering that the jedis internal use of the implementation of the apache commons-pool connection pooling mechanism, in the implementation jedisque of the same scenario is used, and so on, jedisque after the official provision of the pool, the official connection pool will be used.

disque, the command and command parameters may be adjusted during the development process, and horae this will be followed up. Although disque the stable version has not yet been released, the level and reputation of the Redis author is obvious, so you have reason to believe it will surprise you.

Open source address for this project: Https://github.com/yanghua/horae
For more information, please visit: http://vinoyang.com

Copyright NOTICE: This article for Bo Master original article, without Bo Master permission not reproduced.

A lightweight asynchronous task executor based on Redis and Disque implementations