How to use the open-source workflow engine arc of jwfd to implement simple automatic data processing (1)

How to Use the Open Source Workflow Engine arc of jwfd to implement simple automatic data processing

Note: arc is a process automatic operation control algorithm that I implement in Java language in the Open Source Workflow Engine of jwfd (the prototype of this algorithm is a transformed image breadth-first traversal algorithm, after several modifications, it has long been quite different from the classical graph breadth-first Traversal Algorithm in terms of structure. In jwfdv0.94, we can find the initial form of this algorithm ),

I am an amateur workflow enthusiast. I am very enthusiastic about the opinions and suggestions of my friends, classmates, and experts from various professional fields. I hope you will give me more advice.

In a narrow sense, an automatic data processing system is a program dedicated to providing data processing engineering tasks for geographic information systems. However, in a broad sense, automatic data processing covers a wide range of fields, all tasks that can be programmed to perform various numerical and non-numerical calculations can be called automatic data processing. Since I have never worked in this field, I also understood the meaning of these things from a few words from netizens and implemented them on the basis of the existing code, so it is far different from the design level of the real professional automatic data processing system, please forgive the errors.

The Graphic Structure of a workflow is non-linear in most cases, with both serial and parallel structures, the integration of the algorithm modules embedded in each node of the flow chart with such non-linear topology structure leads to a very complex computing process. Although the algorithms in each node have a single function, however, the computing capability of the entire process will become very powerful. This may be the reason we want to use a workflow system to process data. Of course, this is just my guess.

 

The core San Algorithm Model of arc is a circular-judgment-recursive structure. It is only a preliminary attempt to realize the automatic operation of the process under the control of this structure, using this automatic operation control structure to realize an automatic data processing is just a kind of exploration ....

I think the biggest problem with the use of Arc Technology for automatic data processing is that due to the lack of flexibility in the structure of arc and the lack of dynamic flexibility in changing the computing topology path caused by parameter changes, not to mention handling more flexible changes in the computing structure caused by changes in the Process topology. If we can only process a previously designed computing flow, in this way, compared with the traditional programming technology, there is no major breakthrough .......

However, if you want to take a bite, you have to take a step ..... Although it seems that the current arc model is not so flexible to implement automatic data processing, we can first add a feedback Processing Model Based on the loop-judgment-recursive structure, make this fixed computing model a little more flexible. Then let's look for a new solution to solve more flexible problems ........

This is how technology develops... Every small step forward has to be paid, but long-term persistence will always produce some positive results.

Because the design and R & D of the workflow engine are difficult and valuable, there are also many domestic colleagues engaged in this work. However, apart from the papers published by academic and junior colleges, it is rare for me to see articles and materials published by Chinese manufacturers on workflow technology and theory research. It's not just spring. We just have a few open-source projects there, it is difficult to generate a climate, so I try my best to make my code and design public, with only one intent, so that everyone can have more public research topics and thoughts, in fact, I am doing this by myself. Sometimes I feel that it is not easy to think of a new method to break through the original architecture. However, if everyone blocks it, everyone is conservative, then technological advances in the entire industry and field will become more difficult .....

Bytes ----------------------------------------------------------------------------------------------------------------------

Design principle:

At the early technical level, we need to develop various data processing algorithms and subsystems in advance to implement a data processing system, which is solidified in the program, this design is like when we design a analog circuit, we need to first draw the circuit board, and then weld each component, and then debug and process it... Different applications have different circuit boards, so the flexibility is poor!

If our data processing methods and models change, our entire program needs to be re-developed, including the underlying Algorithm Module and the entire system structure, this makes the system lacks a reusable framework and increases the development cost and project application difficulty ......

Now, with the development of workflow technology, we find that using workflow technology or workflow ideas to re-build the data processing architecture will be helpful to the problems mentioned above?

Overall Design Concept

A new architecture for automatic data processing is formed by secondary development of the Designer module and operation control module of the original workflow engine, the main difficulty of transformation lies in the branch data transfer and interrupt process processing in the automatic workflow operation controller.

In the data transmission process of branches and aggregation nodes, what measures should be used to accurately transmit these parameters and computing results? We need to consider this issue from the top-level design. If the objective of jwfd is only to implement an automatic data processing system, I do not need to consider these issues, however, in the future, if nlp fd needs to be at the level of neural network computing and ai systems

These problems must be taken into account. The transmission of data and parameters is not only for automatic data processing systems, but also for the design of neural network computing and ai systems in the future ....

..........
Full-text download

I am very grateful to the csdn and iteye websites for their support for the Open Source Workflow of jwfd. I will write more and better articles and code to give back to you .....