Drools Rule Engine memory overflow due to memory leaks

before entering this question, let's look at Drools:
in many industry applications such as banking, insurance, business rules are often very complex and rules are constantly evolving, and many of the existing system practices are essentially binding business rules to program code.

the main problems are as follows:

1) When the business rules change, the corresponding code will have to change, every time even small changes need to go through the development, test validation on-line process, change costs are relatively large.
2) long-time systems become increasingly difficult to maintain.
3) The development team is generally composed of a business-familiar BA (Business Analyst) and a number of familiar with the technology of the developer, the developer of business rules is far less than BA, but in fact, has assumed the business rules to achieve the task of accuracy.
4) The system is rigid and new requirements are difficult to insert.
5) New demand on-line cycle longer.
Is it possible to make our business systems more flexible?
Idea: To extract the business rules from the technology implementation, to realize the separation of technology and business, developer processing technology, business analysts define business rules, each do their own good things.
Scenario: There is already a relatively mature open source product support, it is drools, we define business rules in database or brms (Management System), through the management of DB or brms to achieve dynamic change of business logic.
When should you use the rule engine?
Although the rule engine solves many of our problems, we also need to think about whether the rule engine is appropriate for our project itself. Some of the points that need attention are:
1) How complex is my application?
For applications that simply pass data out of the database and do not do more, it is best not to use the rules engine. However, the use of drools can be considered when there is a certain amount of business logic processing in Java. This is because many applications are becoming more complex over time, and drools can make it easier for you to handle all of this.
2) How long is my app's life cycle?
If the life cycle of our application is very short and there is no need to use drools, the use of the rule engine will benefit in the medium to long term.
3) Does my application need to be changed?
This answer is generally yes, "the only constant in the world is change", and so is our need, whether in the development process or after development, drools can benefit from the frequently changing needs.
The rule engine is the core part of the rule-based expert system, consisting mainly of three parts: rule base (Knowledge Base) +working Memory (Fact base) + Inference machine (rule engine), The rule engine performs inference logic according to established facts and knowledge base according to certain algorithms to get the correct result.
Drools is a rete algorithm based on Charles Forgy's, easy to access enterprise policy, easy to adjust and easy to manage open source business rules engine, in line with industry standards, fast and efficient.
It is easy for business analysts or auditors to view business rules to verify that the code rules are executing the required business rules.



Drools is a Java-based rule engine, open source, can be complex and changeable rules from hard-coded to free out, in the form of regular scripts stored in the file, so that the change of rules do not need to modify the code to restart the machine can immediately online environment effective.



Drools Basic Work process: Usually we use an interface to do things, first of all to pass in the parameters, and its secondary to the implementation of the implementation of the interface is finished, and Drools is the same, we need to pass in the data, for the rules of the check, call the external interface, You may also need to get the results after the rule has been executed. In Drools, the object that passes the data in, the term is called the fact object. The fact object is an ordinary Java bean, which can read and write to the current object, invoke the method provided by the object, and when a Java bean is inserted into workingmemory, the rule uses the reference to the original object, and the rule passes the read and write of the fact object , to read and write to the application data, and for the attributes in it, it is necessary to provide getter setter accessors, which can be dynamically inserted into the current workingmemory to delete the new fact object.
A rule file can use a. drl file, or it can be an XML file.
Rule syntax:
Package: For a rule file, the whole package must be defined and must be placed in the first line of the rule file. In particular, the package's name is arbitrary, I think it can be directly understood as the namespace namespace on the line, do not have to correspond to the physical path, and the Java package concept is different, here is just a logical distinction. The function and query defined under the same package can be used directly.
Example: Package Com.drools.zken.test
Import: Importing the rule file requires the use of external variables, the use of this method is the same as Java, but unlike Java, where import imports can not only be a class, it can also be an accessible static method in this class.
Like what:
Import Com.drools.zken.test.User;
Import Com.drools.zken.test.User.getUserById;
Rule: Define a rule. Rule "RuleName". A rule can consist of three parts:
Properties section: Defines some of the properties of the current rule execution, such as whether it can be repeated, expiration time, effective time, and so on.
The conditional section, LHS (left Hand Side), defines the conditions of the current rule, such as when User (); Determines whether a user object exists in the current workingmemory.
The result section, RHS (right Hand Side), can be written in plain Java code, which is the action that the current rule condition satisfies, and you can invoke the method of the fact object directly to manipulate the application.
Rule instance:
Rule "Name"
No-loop true
When
$user: User (id = = 6666)
Then
System.out.println ("OK");
$user. Setsalary (10000);
Update ($USER);
End
In the properties above:
No-loop: Defines whether the current rule does not allow multiple loops to execute, the default is false, that is, the current rule can be executed indefinitely as long as the condition is met. Under what circumstances will a rule be executed once and repeated repeatedly? Drools provides APIs that allow you to modify or increase the number of fact objects in the current incoming workingmemory, such as the Update method above, to update the property of the user type's fact object in the current workingmemory. This action triggers the rule's re-match execution, which can be understood as a fact object update, so the rules need to be re-matched again, so the question is whether the data for the properties of the fact objects that were previously executed and modified by the rule will not be reset. The result is no, it has been modified and will not be reset, and the previous changes will take effect after the update. Of course, the modification of the fact object data does not necessarily need to call update to take effect, simple use of Set method settings can be done, here is similar to the Java reference call, so when to use update is a need to carefully consider the problem, once inadvertently, It is very likely to cause a dead loop of rules. The above No-loop true, which sets the current rule, executes only once, and does not execute the current rule again if the RHS part of itself has an operation that triggers the rule to be re-executed, such as update.
But other rules will be re-executed, wouldn't it be possible to cause repeated executions, data disturbances or even a cycle of death. The answer is to use other label restrictions, which can also be controlled: lock-on-active true
Lock-on-active true: With this tag, you can control that the current rule will only be executed once, because the recurrence of a rule is not necessarily triggered by itself, or it may be triggered by other rules, so this is a no-loop version of the. Of course, the regular usage of the label will have other label mates, follow-up mentioned.
Date-expires: Set the expiration time of the rule, the default time format: "Day-month-year", the Chinese and English format is the same, but the wording to use their corresponding language, such as Chinese: "29-7 months-2010", but still recommend the use of more precise and customary format, This requires manually setting the current system's time format in Java code, followed by references. Examples of attribute usage: date-expires "2011-01-31 23:59:59", where we used a more customary time format
Date-effective: Sets the effective time of the rule, in the same time format.
Duration: Rule timed, duration 3000 3 seconds after rule execution
Salience: Priority, the larger the number the first execution, this can control the order of execution of the rules.
Other properties can be viewed in the relevant API documentation for specific usage.
The conditional part of the rule, that is, the LHS part:
When: the rule condition begins. Conditions can be single or multiple, with multiple conditions in order, such as
When
Eval (True)
$customer: Customer ()
$user: User (id==6666)
The above list of three conditions, the current rules only when these three conditions are matched to the execution of the RHS part, the first of the three conditions
Eval (true): is a default api,true unconditional execution, similar to while (true)
$user: User (id==6666) means: The current workingmemory has a fact object with the user type and the value of the id attribute 6666. This object is usually inserted through external Java code or by itself in the RHS section of the rule that has been executed earlier.
The preceding $user represents the reference variable for the current condition, and in the subsequent conditional and RHS sections, you can use the current variable to refer to the qualifying Fact object, modify the property or invoke the method, and so on. Optional, you may not write if you do not need to use it.
Conditions can be combined, such as:
User (name== ' Zhang San ' | | (Id > + && age = = 27))
The RHS must use the getter and setter methods for the private property of the fact object, while the RHS must be used directly in the. method, such as
$order: Order (name== "shopping_001")
$user: User (id==6666 && orders contains $order && $order. name== "shopping_001")
In particular, if the conditions are all && relationships, you can use "," instead, but they can't be mixed.
If there is a list in the fact object now, you need to judge the condition, how to judge it.
See an example:
User {
int id;
List<string> names;
}
$user: User (id==6666 && names contains "Zhang San" && names.size >= 1)
In the above conditions, the ID must be 6666, and the names list contains "Zhang San" and the list length is greater than or equal to 1
Contains: Compare whether an action is included, the contained target of the operation can be a complex object or it can be a simple value.
Drools provides 12 types of comparison operators:
> >= < <= = = = Contains/not Contains/memberof/not memberof/matches/not matches
Not contains: Contrary to contains.
MemberOf: Judging whether a fact is in a collection, unlike contains, the object to which he is compared is a collection, and contains is compared to a single value or object.
Not memberOf: just the opposite.
Matches: Regular expression matching, unlike Java, does not consider the '/' escape problem
Not matches: just the opposite.
The result part of the rule
When the rule condition is satisfied, it goes into the rule result part execution, the result part can be pure Java code, for example:
Then
System.out.println ("OK"); Will print out OK on the console
End
Of course, you can also invoke the fact method, such as $user. shopping (); Operations database and so on all operations.
The result section also has the method that Drools provides:
Insert: Inserting a new fact object into the current workingmemory will trigger the execution of the rule unless no-loop is used;
Update: Updates
Modify: Modified, unlike update syntax, results are update operations
Retract: Delete
The RHS section can also invoke methods defined in the rules file, in addition to calling the methods of the API and the fact object provided by Drools, and the definition of the method uses the Function keyword
function void Console {
System.out.println ();
Dbhelper.getconnection ();//Call an external static method, DBHelper must be imported using Import, getconnection () must be a static method
}
Drools also has a keyword that can define a class:
Declare can define a class in a rule file that is similar to a normal Java object, and you can use the getter and setter methods to manipulate its properties in the RHS section.
DECLARE Address
@author (Iamzken)//meta data, only for descriptive information
@createTime (2015-10-13)
City:string @maxLengh (100)
Id:int
End
What is the above ' @ '? is a metadata definition, data that describes the data, nothing to do with the meaning
You can use the address address = new address () method in the RHS section to define an object.

Drools is easy to use, but if used badly, there is a high likelihood of an oom problem .

The above general introduction of Drools, the following into the topic:

directly on the code:


This is the main code snippet:

Workingmemory w = init ("SERVICE.DRL");//Initialize Workingmemory object
User user = new user ();
facthandle fact = w.insert (user);
W.fireallrules ();
W.dispose ();
W.retract (fact);

Rule file (SERVICE.DRL):


Package Com.iamzken.test.service
Import com.iamzken.test.model.User
rule "Rule-test"
Activation-group "test001"
salience
lock-on-active true
dialect "Mvel" when
$user: User (Salary >, gender== "1") then
$user. Setsalary (+);
Update ($user);
End

At first, I have been unable to understand is: the test environment on the amount of data is about 3.7 million, the amount of data on the production environment is 3.5 million, from the data, the production environment does not have a test environment of data volume, why the test environment did not appear oom and production environment on the Oom. 

with this in doubt, we first increased the memory of the production environment machine and tuned the JVM memory-related parameters to find or Oom

Finally, after a variety of troubleshooting, including the construction of data, using the Jprofiler tool, analyze the dump file, and ultimately to the Drools rule engine is the cause of the problem.

Reason:

Modify the previous code:
W.insert (user);
W.fireallrules ();
W.dispose ();

after the modified code:

facthandle fact =  w.insert (user);
W.fireallrules ();
W.dispose ();
W.retract (fact);
or:

W.insert (user);
W.fireallrules ();
W.dispose ();
W.retract (w.getfacthandle (user));

As shown in the code above, W.insert (user) will return a Facthandle object after executing this code, need workingmemory call retract method to remove it from memory, otherwise, even if the rule becomes a garbage object, It cannot be reclaimed by the garbage collector, because Workingmemory also refers to the inserted user object.

Why is there no oom in a test environment with large data volumes? The reason is simple: because the data volume of the test environment is large, but the amount of data that can match the rules is small, that is, the number of user objects inserted into the workingmemory is less, and the production environment is the opposite, which is why the data volume is small but also the cause of oom.
conclusion: This is due to memory overflow caused by drools memory leaks.

The online information about drools memory leaks is still very small, and is hereby recorded, hoping to help friends who meet the same problem.