Design ideas and architecture of the microkernel Process Engine (published in the seventh silver bullet magazine)

Two months ago, at the invitation of the editor of the silver bullet magazine, he wrote an article on "process" and "microkernel, the title is "microkernel process engine design ideas and architecture ". Now the seventh issue of panyuan magazine has been published, and the full text is now transferred to my blog.
The original address of the silver bullet: http://gocom.primeton.com/modules/gSpace/modules/techresource/article1803.htm

After reading this article, I will read "Demystifying jbpm Process Engine kernel design ideas and architecture", which may be easier to understand some of the theme of this article. We also recommend that you refer to the "Workflow Model Analysis", "workflow authorization control model", "Workflow Organization model", "workflow engine scheduling algorithm and petrinet", and "microkernel" I wrote a few years ago. workflow Engine architecture and other documents. These documents can be downloaded on my personal homepage (http://www.javafox.org.

 

Design idea and architecture of microkernel Process EngineAuthor: Hu Great Wall (yinhu 999), tibco China R & D center, has been increasingly sought after and recognized by customers for its application systems based on process engine technologies over the past two years, the ability to support "customizable processes, Changeable Processes, and operational" has gradually become one of the main criteria for customers to measure an application system. For example, SOA (Service-Oriented Architecture), which is widely mentioned, provides new guiding ideology and methods for customers to solve the "business agility problem. However, the overall architecture of SOA must rely on the support of three technologies: technologies and standards for interconnection, such as the well-known message bus technology, jbi, and SCA; solve the technology and standards of process management, such as BPM and workflow, and solve the technology and standards for building business models, such as the well-known MDA (model-driven architecture. The importance of technologies related to process applications can be seen. Recently, many enterprise friends complained to me that when they implemented workflow projects for customers, it was always very tricky to use third-party workflow products or to expand and develop open-source workflow engines, many difficult problems are encountered, and these problems usually occur in the later stages of the project. This is because the customer cannot know what functions the process system should have during the demand investigation process. It can be said that many precautions can be elaborated from the perspective of workflow project implementation, but this article helps you to look at the process from another perspective. This is entirely from a "Source" perspective-if you understand the Implementation ideas and architecture of a process engine, I think you will not be surprised by the "weird customer needs". On the contrary, you can easily deal with it. Yes. The purpose of this article is to explain the design ideas and architecture of the microkernel process engine ". Before entering the text, I need to add two more points: (1) in the previous article, Mr. Yang Hongbo has explained the similarities and differences between workflows and business process management (BPM. To reduce misleading terms, this article uses the process concept to avoid the possible conceptual differences between workflow and business process.Of course, the fundamental problems solved by workflows and BPM are process problems. (2) The implementation technology of the Process Engine has exceeded the pure technical language, technical model, and architecture.After reading this article (or before), if you are interested in the implementation of the workflow engine, you can view the basic concepts, models, modeling methods, and system reference models of workflows as much as possible, which may help you read this article. When we try to implement a workflow project or develop a process engine, we will face a lot of problems that need to be solved: the process has branches, aggregation, and customers have to sign and roll back; organization models must be adapted, and permissions must be controlled to Data. As shown in the figure above, it may be easier to solve a single problem, but so many things need to be considered are merged into a very complicated problem. However, complex problems can also be solved through gradual decomposition, divestiture, step-by-step, hierarchical construction. The purpose of this article is to provide a set of design process ideas and engine architecture to help you solve this problem.Limited by the length of the article, this article can only be "click to stop" in many places. If you are interested in building a process engine, after reading this article, it takes more time to consolidate relevant knowledge. In the "workflow management: models, methods, and systems" book by the workflow master Aalst, a reference framwork is provided to understand the workflow ). The reference framework consists of three parts: basic concepts, process modeling and analysis, and the function and architecture of the workflow management system. When trying to design a process engine, we need to focus on these three aspects: Understanding the basic concepts to really understand what the process is; understanding process modeling to know how to describe the process; A process engine can be designed only when the workflow system structure is understood.  Process Design Principles (IDEAS) 1: Process ModelingSome people may say that "describing a process" is not a problem at all. Isn't the xpdl defined by the workflow management Alliance (WfMC) exactly the "XML-based process description language. Indeed, WfMC uses an xpdl language to tell people how to describe a process, or in an official language, xpdl is the implementation of interface 1 in the workflow reference model. But when you are in the complex schema of javasxpdl, have you noticed the workflow definition meta model described by WfMC? Many may not have noticed this. When WfMC published the "Workflow Reference Model" document in 1995, it already elaborated on the process definition meta-model. However, the process definition meta-model described by WfMC is not unique. That is to say, many process products do not use this descriptive language. This difference is caused by the difference in "process modeling methods. At present, there are mainly the following modeling methods in the process field: Petri net model, finite state machine (FSM) model, activity diagram model, and event process drive chain (EPC) model. Of course, we know that the process definition meta-model estimation of WfMC can also be considered. For example, the Open Source engine JBoss jbpm we are familiar with is the activity graph model. The topic and length issues are limited. I will not describe the knowledge of process modeling methods here. Any process is composed of two basic elements: "Node" and "directed connection ". There is almost no ambiguity about "directed connections". In all process modeling descriptions, "directed connections" have the "from" and "to" features. However, for "nodes", there are many different understandings due to different perspectives and functions. For example, the process definition meta-model of WfMC states that the "Node" is "activity", but there are many implementation types of this activity: manual and automatic, there are sub-procedures, and so on. For example, for the activity diagram on which the jbpm engine depends, the "Node" is described in two ways: one is to represent a node in the real business sense, called the state; the other is a node that represents a logical connection. For example, in the EPC modeling method, three types of nodes are described as follows: one is a node indicating a certain state, which is called an event; the other is the fuction, which indicates the real business function, and the third is the node that represents the logical connection. However, the meanings and types of the logical connection nodes of EPC and jbpm are different. The above example shows that we can use many methods to "describe the process ". When you try to develop a process system, finding the most appropriate method to describe the process is the first. It is very difficult to go beyond this hurdle, because many technical staff do not have much understanding of the basic concepts and modeling methods of workflows. Fortunately, the xpdl of WfMC is still very complete, providing us with a very detailed reference implementation. In addition, the process definition of many well-known workflow products at home and abroad follows the xpdl specifications. Process Design Principles (IDEAS) 2: Relationship between process definitions and InstancesIf the process definition model is determined at this time, the next most important thing is to consider the relationship between the process definition and the process instance. This is not a simple relationship between "class" and "object instance. In the WfMC specification, a simple relationship between process definition and process instance is described, as shown in: this relationship is a very simple expression, but the application in reality is far more complex than this. In the same process instance, does an activity only have one activity instance? In real-world applications, there are complicated multi-instance scenarios. For example, for some reason, "rollback" may occur before returning, the same active node may be executed twice or even multiple times. For example, the asynchronous execution of multiple handlers on a node, the execution of subsequent nodes may also be repeated. This relationship with the instance must be considered before the process engine is designed. Otherwise, the customer may be caught off guard in the future. The above only describes two basic design ideas. When designing any process engine or expanding the development of existing engines, we need to clarify these two ideas and design features. Of course, there are many other aspects to consider for the overall engine design, such as process scheduling, the dependent status, and events. This article is not described here. Design Concept 3: microkernelThe concept of micro-kernel was first proposed by Richard Rashid, although this was initially designed to build a micro-kernel Operating System Based on the message transfer mechanism, not to serve the software architecture. However, in recent years, with the development of the "componentized" and "layered" software system, microkernel technology and construction ideas have been gradually introduced into the software design architecture, it is used to "decouple the relationship between components as much as possible ". In contrast to the microkernel design philosophy, the macro kernel (monolithic kernel) is also a concept derived from the operating system level. For the macro kernel, the entire operating system is a whole, including process management, memory management, and file system. For the micro kernel, most of the operating systems are outside the kernel, communicate with each other through messages. The reason why we talk about the relationship between the macro kernel and the micro kernel is that the relationship between "process scheduling", "Process Engine", and "application processing" determines the design of the Process Engine, it will inevitably involve the consideration of "micro kernel" and "macro kernel. Many designers generally use the "microkernel" design as the guiding ideology when designing the process engine, as shown in the figure above. However, in the later stages, as the application complexity increases and the customer's business needs become more specific, the final process engine may be shown in the right figure, thus ultimately deviating from the microkernel's purpose. The main reason for this is that there is no "consistent standard" for the component design and Interface Description of the Process Engine, that is, an engine is not isolated, it must also be composed of many components, service interfaces, rules, and other aspects. These components need to communicate with each other, but there is no unified specification that can constrain this communication mechanism and data. Therefore, they ultimately depend on the Implementation and abstraction capabilities of each vendor. This is not like the well-known JBoss app server. It relies on the JMX standard to build a perfect "microkernel" server. All services and components are mounted to the microkernel architecture by implementing JMX interfaces. calls between components and Data Interaction headers depend on unified standards. Although it is difficult for us to construct a process engine to implement a pure "micro-kernel" at the present stage, through the guidance of the design philosophy of "micro-kernel, and constantly abstract and refine the components inside the engine, and strip and reconstruct services to gradually reduce the coupling between business applications and process scheduling. Microkernel process engine architectureI wrote a document named "microkernel workflow engine architecture" two years ago, which describes several layers of the microkernel workflow (process) engine and corresponding solutions. This document can be downloaded free of charge on my homepage www.javafox.org. The principle of the micro-kernel engine is to remove the "services required by the engine" from the "scheduling computing of the engine kernel" whenever possible. Try to cut the engine kernel part as much as possible, and try to place some processing operations on peripheral extensions. The figure above may show similar content in many product introductions or workflow articles, but it rarely mentions the "Process Engine" structure. The microkernel process engine structure is generally built around three layers: (1) Engine kernel processing:This is the real kernel of the engine. The most important problem solved at the layer is: Process Scheduling and execution. (2) Engine execution service layer: Provides some public services and resource processing for the engine. (3) Extended Implementation Layer:This layer is the most cumbersome part of the microkernel, and there are various extension points for peripheral extension. To sum up, there are mainly three types: a) supported process running: These extension points must be implemented, otherwise the engine cannot run. For example, provide storage adaptation for the engine. B) auxiliary process execution: These extension points allow the engine to perform more processing, such as some function interfaces and conditional processing interfaces. C) Enhanced process running: using these extension points, the engine can be enhanced and improved, such as some event listening and processing interfaces and custom policy implementation. "Micro" KernelIn general, the microkernel has two meanings: (1) The supported functions are very simple: only the core process scheduling functions are processed.That is to say, when a node is completed, the kernel of the process engine can be accurately defined based on the process and according to the scheduling algorithm and mechanism constructed in advance, to compute the next node to be executed. (2) The implementation structure is very simple: there are not too complex implementations, but there are several basic object bodies.In terms of implementation structure, there are no standard rules, basically every engine will be different. You can compare the jbpm, osworkflow, and workflow engines with different structures. I will not analyze the structure of the three open-source engines in detail here. You can refer to the core Scheduling Algorithm of the workflow engine I have written. some of the content involves structure comparison. Although the structure may be quite different, the basic implementation idea is more or less similar to the basic key Object Body. (1) Each engine Kernel provides an executecontext object to take charge of the environment information of each instance. (2) Each engine Kernel provides a mechanism for calculating the flow of processes. This mechanism varies greatly with different algorithms and ideas. (3) Each engine Kernel provides an execution mechanism to execute the "nodes allowed to be executed" (4) each engine Kernel provides a set of kernel instance objects to indicate the persistence information of the Process instance. EndIt is difficult to explain the design ideas and architecture of the micro-kernel Process Engine in just five thousand words. A lot of content can only be skipped. The design of the process engine must be integrated with "theory" and "technology ". Not only do you need to have a deep understanding of software development, but you also need to have the basic concepts, concepts, process modeling methods, system structures, and other theoretical foundations of workflow.

Statement: For reprint, please specify the address and author information. Thank you.
==========================================
Author information:
Hu Great Wall (silver fox 999)
Tibco China R & D center, infrastructure team
Home: http://www.javafox.org
Blog: http://blog.csdn.net/james999