"End-to-end" performance diagnostics for Applications with Oracle RUEI+EM12C

Overview

We know that the impact of a B/s application performance factors, roughly speaking, has the following several major links:

1. Client link

2. Network links (may include WAN and LAN)

3. Application and middle-tier links

4. Database Layer Links

The "end-to-end" diagnosis can be regarded as the "through" diagnosis of the problems in each link.

There are many tools available for this type of diagnosis, which we'll look at later, just to show you examples of application performance diagnostics from the front-end to the final end using Oracle's tool software.

The following are some of the Oracle software products involved:

1. Ruei (real Customer experience insight)
2. EM12C Basic Framework
3. WebLogic Monitoring Module
4. JVMD (Java Virtual Machine Diagnostics)
5. Oracle database monitoring related modules

The products listed above are actually components of Oracle Enterprise Manager products, so let's start with a brief introduction to software that is not so popular:

Ruei Introduction

Ruei (Real User Experience Insight) is a quantitative measurement of an application's performance tools from a customer experience perspective. Using traditional methods, we can only measure the performance of an application from the backend, from the point of view of resource consumption, or from the front-end, from the perspective of the user's perceptual evaluation. For example, we can look at the CPU consumption of the host from the backend, look at the database waiting events, and so on, or use the survey questionnaire to investigate the user's actual use of feelings, but often get, such as "Too Slow", "easy to crash", and so very emotional evaluation, end users will never accurately tell you quantitative evaluation, For example: How long does it take to load the page to complete?

And Ruei can tell this information, Ruei use the network sniffing technology, crawl the B/s application between client and server side of the packet, analysis of the time stamp, through the corresponding algorithm, the application user access to the application of quantitative performance data.

The basic diagram of Ruei is this:

JVMD Introduction

JVMD was originally called Ad4j,application Diagnostic for Java, is a separate product, then slowly by Oracle integration into the EM12C, it is installed on the Java EE Application Server diagnostic tools, The performance information of the application module is analyzed by using the sampling algorithm (Oracle claims that it is not a BCI method-byte-code injection). JVMD can perform live thread parsing while restoring the application's SQL calls, which can be helpful for diagnosing from the middle tier across to the database tier.

Because there are many products involved, if the specific configuration method, the article will be very long, so today only see the case, the specific method, if everyone is interested, follow-up can continue to introduce in other articles, you can also refer to the experts written by Haitian expert "Oracle Cloud Management Platform: Enterprise Manager 12c Practical Guide", There are detailed configuration methods in the book. (also small to make an advertisement, haha)

Case

A province of the new application, there are some performance problems, I hope we help to diagnose and optimize. Customer application is b/s architecture, the structure is relatively simple, the client Access Apache HTTP server, the Apache is responsible for processing static pages, dynamic page forwarding to the backend WebLogic server cluster, background database is Oracle.

After deploying the Ruei and em12c (WebLogic monitoring, jvmd,oracle database monitoring), the tool is used for diagnostics:

(in order to make it easier for everyone to understand, I only use as little information as possible to illustrate the problem, the actual situation may be much more complex.) ）

First use Ruei to see the overall situation:

Green on behalf of the customer satisfied with the amount of traffic (page return within 2 seconds), yellow for normal traffic (2-4 seconds page return), red represents the user "angry" traffic, more than 4 seconds to return. (The time of the page satisfaction can be set by the customer)

Overall, the situation seems to be acceptable.

Look at the performance information from the Ruei point of view, client, network, server three links:

In addition to the individual pages will have the browser busy time (such as the browser to perform JS such action time-consuming) has a very small amount of time, a large number of page browser time is very little, less than the degree that the software thought can be ignored (displayed as 0 milliseconds)

From this diagram, you can also tell if the network is slow (per-click transfer time) or if the server is slow (server time per click).

Obviously, server time is the "Big Head".

Further analysis below, in the end is which page requests consume server time longer:

Find time-consuming pages, if ECID (run context ID) is enabled on the HTTP or WebLogic side, we can drill down to JVMD directly from the Ruei and proceed to the next diagnosis. In our case, the customer does not have ecid enabled, so we can only perform a manual "Drill down".

The most time-consuming page from Ruei: xxxxxframe.do, using this request in JVMD, the most conditional, as the JVM Thread information filtering condition:

Get specific information about the thread that executes or this request:

Click on the green display of the wait, get a single thread specific analysis information, which will have thread waiting for the SQL information:

In the further click on the SQL ID, go to the SQL Information page of the database:

In fact, once the SQL statement is found, the problem is simple for the DBA. With limited space, it is no longer mapped.

This case is not the result, but the process, and if we do not have such a tool, we may face several problems:

1. Non-quantifiable Customer experience diagnostics
2. Can not distinguish between the time-consuming (client, network, server)
3. Diagnostic efficiency is poor on the server side
4. The need for manual page requests and SQL Association, inefficient, and sometimes, without the help of developers, is impossible to complete the task

Today, these tools from Oracle can dramatically increase the accuracy and efficiency of diagnostics, while also giving customers the ability to see quantifiable customer experience data.

"End-to-end" performance diagnostics for Applications with Oracle RUEI+EM12C