Implementing distributed Message Queuing based on Redis (2)

1. What are the features that Message Queuing needs to provide?

In functional design, I advocate the law of the Ames Razor.
For Message Queuing, only two methods are required: production and consumption.
The specific business scenario is the task queue, and the code is designed as follows:

publicabstractclass TaskQueue{    privatefinal String name ;    publicgetName(){returnthis.name;}    publicabstractvoidaddTask(Serializable taskId);    publicabstractpopTask();}

Multiple queues are supported at the same time, and each queue should have a name. Final ensures that Taskqueue is thread-safe. The Taskqueue implementation class should also ensure thread safety.

AddTask adds a task to the queue. Only the ID of the task is saved in the queue, and the business data for the task is not stored.

Poptask takes a task out of the queue to execute.
This design is not particularly friendly, because she needs the caller to ensure that the task executes successfully, and if the execution fails, make sure to re-put the task back into the queue. In any case, such a mechanism can work. Think about the law of the Ames Razor, and we'll take a look at the design.
If the caller has the business data in the database, the business data contains the status column, identifies whether the task is executed, the caller needs to manage the state itself, and controls the transaction.

2. Subsequent functions that may be provided 2.1. Introduction of the task life cycle concept

Different application scenarios and different requirements.
In a rigorous application scenario, you need to ensure that each task executes "successfully".
For the above mentioned Poptask "mode", this is another "run mode", the two modes can exist in parallel.

In this new mode, there are 3 task states: Newly created (new, just called AddTask added to the queue), executing (in-process, calling Poptask, before finish), finished (done, executed OK, after the call Finishtask).
The adjusted code is as follows:

 Public Abstract  class taskqueue{    Private FinalString name; PublicStringGetName(){return  This. Name; Public Abstract int GetMode(); Public Abstract void AddTask(Serializable taskId); Public AbstractSerializablePoptask(); Public Abstract void Finishtask(Serializable taskId);}

2.2, increase the function of the batch removal task

Poptask () Take out one task at a time, it's so abrasive.
Like we want to buy 5 bottles of water, drive to the supermarket to buy, every go to buy 1 bottles, a little something.
We need a way to take more than one task at a time.

publicabstractclass TaskQueue{    ... ...    publicabstractpopTasks(long cnt);}

2.3. Increase the blocking wait mechanism

Imagine a scene:
Xiao Ming classmate, take out a task, found that can not dry, put back to the queue, and then go to fetch, take out to find or do not, and put back. Repeatedly.
Is Xiao Ming's children's shoes swollen? Maybe he needs a network to work, and the network is broken. Maybe he needs to write the disk and the disk is full.

If Xiao Ming is as good as a neighbor's child, he should calm down and rest for a while when he finds something wrong.

But what if he's not? Only we can help him.

If there are 10,000 todo tasks in the queue.
This is when Xiaoming comes. Should we stop him after he failed 100 times? should not, unless he volunteered (configured in system parameters). After 5,000 times? Nor should he, unless he asks for it voluntarily. Our principle is: all the things we do, for the caller, are predictable.

We can ask the caller to set a threshold of n in the system parameter, if not set, the default is 100. After successive failures n times, let the caller sleep for a while, sleep for how long, let the caller configure.

If our underlying implementation contains a backlog of sub-queues, redo sub-queues, and complete sub-queues (this design is so complicated!) Pop when the first pop redo, or the first pop to do, complex dead! I wish I didn't need this.)
There are 10,000 tasks in the TODO sub-queue.

After Xiaoming failed 10,000 times, all the tasks were in the sub-queue again. Should we stop him at this time?
Redo the sub-queue to set the size, beyond, let the next visitor wait.
And so on will be involved in timeouts, after the timeout, the task can not be discarded.
It's too complicated! Setting a limit on the number of consecutive failures is enough!

2.4. Consider adding a task class

The data that does not save the task is the basic principle, absolutely unwavering.
Adding a task class can manage the life cycle, and, more usefully, you can design the task itself as listener, as the code might be:

 Public Abstract  class Task{     PublicSerializablegetId(); Public int getState(); PubicvoidDotask (); Public void whenadded(FinalTaskqueue TQ); Public void whenpoped(FinalTaskqueue TQ);//public void Whenfaild (final taskqueue TQ);     Public void whenfinished(FinalTaskqueue TQ);}

Through the task interface, we can perform more robust management of the calling process (for example, transaction control), exert more control over the caller, and the user can gain more interaction opportunities and better interaction with taskqueue (such as persistence in whenfinished).

But are these really necessary? Is it too intrusive? Is the annotation way better?
Consider it again.

2.5. Adding system parameters

Seemingly need a config class, uncomfortable!
Would like to do a very small and very delicate small things, if you have to add it.
If you do, you need to support properties, annotation settings, API mode settings, spring-injected settings, and annoying.

Time-back: the Redis mechanism and the taskqueue fit.

Implementing distributed Message Queuing based on Redis (2)