An IBM ODM Memory performance optimization case

A customer used a number of years of ODM, the system deployed a large number of business rules, when the month-end business volume is set, the performance of the rules server will be under pressure, the application server JVM often occurs full GC, or even lead to outofmemory, serious impact on business operations.

Let's take this as an example to see how to handle such performance problems in a structured way.

Step 1-Confirm the problem

First we look at the customer's GC log to confirm the problem:

The following behavior can be observed from the GC log:

• Full GC occurs every few seconds
• 4-5 seconds per full GC duration
• Older generations and persistent generations of memory use more than 90%
• After full GC, the memory footprint in older generations is still high

It can be speculated that there are a large number of referenced old generation objects in the JVM that cannot be recycled by GC.

Step 2-code review

And then we review the customer's rules. Application architecture and rules design mainly cover rules project, object model, rule writing, Rule Liu Design, service call integration and so on, no obvious design problems that can lead to memory leak are found.

Customers use EJBS to implement remote rule service invocation, although this is not a recommended best practice, but generally does not cause memory problems.

Step 3-Memory analysis

Now that the review code cannot find the obvious problem, we have to drill down into the Java process's Heapdump file to see if there is a memory leak and what objects are consuming a lot of memory.

The following is a Heapdump analysis diagram reproduced on the test server:

You can see that the Ilrtranslationdebugsupport object that contains 30,016,942bytes in the RuleSet object with an in-memory size of 55,401,558 bytes should also be used by the debug as a name. It is important to note that the object is not small in the test environment because only 6 rule services are used in the test. In a production environment, dozens of rule sets are loaded into the rule engine, resulting in significant memory consumption. The Heapdump analysis of the production environment server shows that the modified object consumes more than 500M of memory, which leads directly to the full GC.

Step 4-Solutions

Log in to res and select the corresponding rule set to see if the Ruleset.bom.enabled property is set to true, and set it to false to prevent the rule engine from generating ilrtranslationdebugsupport objects.

This property is primarily used for monitoring purposes and is explained as follows:

After modifying all relevant ruleset property settings, testing and observing Heapdump, the Ilrtranslationdebugsupport object should no longer appear, and Ilrruleset object memory consumption is significantly reduced (approximately 40%).

Conclusion

The above solution can effectively reduce the memory consumption of the rule server, and there are additional recommendations to help customers improve performance:

1. Integration of rule services using POJO or other lightweight remote call technology

2. Decision engine features using the ODM8

These two recommendations also apply to most ODM design scenarios, which I will explain in more detail in other articles.

Note: This article is published in http://decisionrule.com/zh/2015/02/memory-performance-optimization/, reproduced please indicate the source.

An IBM ODM Memory performance optimization case