Introduction to the Enode framework: a design idea for the message retry mechanism

In the last article, the paper briefly introduces the design idea of message queue in Enode frame, and introduces the design idea of the retry mechanism of the relation message in the Enode frame.

For an EDA architecture based framework, the core is message driven and then based on the principle of final consistency. So, the very important point is, if the message does not succeed at once, what should be done? The response I can think of is a retry of the message. I find this article more difficult to write, because it is not easy to feel the need to express the complex things clearly. When it comes to trying again, what is the message retry? How can we try again? The retry I mentioned here refers to a message that is processed after it has been fetched from the message queue, but the processing fails, and then tries again to process the message; This problem is more complex, can not be explained in a simple one or two words.

As mentioned above, if the message processing failure to try again, in fact, is a relatively rough answer. For example, a message in the processing of a total of 5 steps, if the first 2 steps are successful, but the 3rd step failed, then try again, the first 2 steps need to be implemented? My idea is, if you can do it, do not do the first 2 steps, but directly from the 3rd step to try again. Therefore, this practice is equivalent to "where to fall, where to continue";

So how do you try it again?

After analysis, we found that the whole Enode framework to retry the point is very much, such as the command generated by the event to send to the queue, if the failure of the need to try again, such as event persistence failed, also need to try again, and so on. So, obviously, we should design a reusable retry service that provides support for certain retry scenarios.

Let's think about what kind of retry function we want to have. Take "event persistence failure" as an example, if this step fails, we would like to retry the step several times immediately, such as 3 times, if the success of 3 times, then success, continue to do the following logic; If you fail? Did we just give up? In fact, we can't give up because generally if the event persistence failure is most likely due to a network problem or a eventstore problem, and if we give up, it is likely that the entire business logic process will be interrupted so that the final consistency of the data cannot be achieved. Therefore, because of this temporary IO problem caused by the failure, we can not casually give up the retry, should be tried several retries still fail to take the necessary means, can be in the IO recovery, can automatically process the message again, but we can not use the current thread unlimited retry down, Because this leads to no way to deal with other news; so it's natural to think that we should put the message into a dedicated retry queue when the message is retried several times, and then another separate thread will periodically remove the message from the queue to retry, and then try the messages again, so that when IO recovers, These messages can be successfully processed soon;

Another question, does this special retry queue need to support message persistence? No, we just need a memory queue, okay? It is not removed from the message store until a news has been completely successfully processed; So, even if the machine reboots, the message can be processed after the machine restarts, and when the machine is not restarted, The specially-retried memory queue retries The message periodically with a separate thread;

So how many do you need for this dedicated retry queue? Theoretically we can design a retry queue for each point that needs to be retried, but this is too complicated and the system's performance can be affected by a lot of threads, so we need to weigh the design of a retry queue for all the things that are going on in the same phase. Any one of these steps in this phase fails and is placed in the corresponding retry queue at that stage;

There is also a problem, if a message succeeds on one retry, but we want to continue to follow up on the message after success, how do we implement it? It's a bit of a hassle to think about at first, because we may not have some context for the message. Most importantly, how do we know what to do next when the message retries successfully? And even knowing what to do next, but what if we do this next step, if we fail again? Do you want to try it again? So, we find this is critical.

After some of my thoughts, I found that if a message is handled at a certain stage with multiple steps, and some steps are conditional, such as only if the outcome of the 2nd step is successful, then we have to do the following 3 steps; Normally, if everything goes well, it's a step down from the top down. But because there may be problems with any one step, and we want to continue with the next step after any failure and then retry the success. So, based on these features, I think we can design a mechanism that resembles a callback function, when a logical execution succeeds and executes the callback function, we can store the next logic in the callback function; Obviously, I think we need some sort of recursive data structure; To support the need for such a callback function above, I designed one of the following data structures:

<summary> a data structure that encapsulates a section of logic to be executed and related contextual information///</summary> public class Actioninfo {//
        /<summary> represents the name of an action///</summary> public string name {get; private set;} <summary> represents an action that encapsulates a logical///</summary> public func<object, bool> action {get; p Rivate set; ///<summary> Indicates the data information required for action execution///</summary> public object data {get; private se T  ///<summary> represents the next action message to execute after the action is executed successfully, which reflects the recursive///</summary> public actioninfo
    
        Next {get; private set;} Public Actioninfo (string name, Func<object, bool> action, object data, Actioninfo next) {if (AC
            tion = = null) {throw new ArgumentNullException ("action");
            } name = name;
            action = action;
            data = data;
        Next = next;
  }  }