[Compilation] Getting started with streaminsight application optimization (4) -- parsing the memory consumption of streaminsight applications

Refer to blog:Http://blogs.msdn.com/ B /domain_connect4

This article will introduce three main aspects: first, we will introduce a scenario where the streaminsight application consumes memory. Next, we will introduce some terms related to the memory consumption of the streaminsight application, this article will end with an actual case study.

Application scenarios

Assume that the streaminsight ApplicationProgramIs the only process running on the machine, so all the memory available for it will depend on "The physical hardware and operating system of the processCommit limit)". The upper limit of delegation refers to the sum of physical memory and page file size. In actual situations, not all physical memory is counted as the maximum entrusted memory, because the operating system itself reserves some physical memory for your own use. We usually call the Total virtual memory used by all active processes as"Current commit charge)And the value does not exceed the system delegate limit. OnceCurrently approved usageApproachingDelegate limitVirtual Memory Allocation will fail.

ObservationCurrently approved usageAndDelegate limitThe simplest way is to use the "System Information" window in the process explorer tool, as shown in:

Figure 1: use process explorer to view memory usage

The two marked in the red rectangle indicate the current approved usage, the delegate upper limit, and the percentage of the former in the latter. If the value of "current" in the figure is very close to "Limit", the application will face the problem of "out of memory (memory depletion.

Currently approved usageIt contains the physical memory and page file size, but sometimes it is difficult to accurately calculate the page file size used by all processes. Therefore, when troubleshooting the streaminsight application, you can consider adding the performance counter "Process <youapplicationname> \ private bytes" to check whether it has arrived.Delegate limit.

Streaminsight memory consumption terminology

In the streaminsight event debugger diagnostic view introduced in the previous article, you may see many memory parameters, such as "events memory used (bytes )", "index memory used (bytes)" and so on. To diagnose streaminsight memory consumption, You need to observe the changes of these parameters. Monitoring streaminsight servers and queries on msdnArticleEach parameter in the diagnostic view has a basic introduction. You can read the following information first. This article will focus on the parameters related to memory consumption.

• Memory usage for all active events in the system [CEP:/Server/eventmanager]
  

All events memory

Memory usage (in bytes) for all active events in the system ).

During query and processing, the streaminsight engine allocates and allocates memory for input events, intermediate events, and output events on the native heap.

• Index memory

Operatorindexmemory

The total memory (in bytes) occupied by the index structure in the operator. It mainly stores the operator status. For example, it stores the operator for incremental computing or output correction events.

• Stream memory

Streammemoryincludingevents

The amount of memory used by the stream and all its events (in bytes ).

This memory is mainly allocated for internal stream queues and events in the queue. it slightly overlaps with the [Event Memory] of the event part. In most cases, the total number of events in the stream queue is relatively small.

• Managed heap

Not only does the streaminsight query manager apply for memory in the managed heap, but also the input/output adapters. You can use the performance counter [. Net CLR memory \ # bytes in all heaps] to view the total heap memory used by the process.

Case study: What should I do if the streaminsight application reports an OOM exception?

I believe that the reader has learned a lot from the above introduction to view the memory usage of a process. Next we will analyze it in depth from a case. This case is not uncommon, that is, when the streaminsight application is running at a certain time, it will report "out of memory (Memory leakage, OOM for short )".

To make the following instructions clear and easy to understand, we assume that only the streaminsight application is running on the machine and only one streaminsight query is running.

When troubleshooting the memory problems of the streaminsight application, it is better to check the overall memory usage and the memory usage of each part in sequence. The following describes in detail:

View overall memory usage

The total memory used by the process can be observed through the performance counter [Process \ private bytes]. This value shows the amount of memory that the process has not shared with other processes. These memories contain allCodeManaged Data and unmanaged data.

View memory usage

You can view the following specific entries to know where the memory used by the process is used:

• Native memory usage-obtained by viewing the streaminsight diagnostic View
  • A =Operatorindexmemory
    • Obtain the value of the following address in the query diagnosis view: ["CEP:/Server/application/<appname>/query/<queryname>"]
  • B =Alleventsmemory
    • Obtain the value of the following address in the query diagnosis view: ["CEP:/Server/eventmanager"]
  • C =Streammemoryincludingevents
    • Obtain the value of the following address in the query diagnosis view: [CEP:/Server/application/<appname>/query/<queryname> "]
• Managed memory usage
  • D =All managed memory consumption
    • View the value of the performance counter [. Net CLR memory \ # bytes in all heaps] to obtain the managed memory usage. Use the Windows performance monitor perfmon to add counters for the corresponding instance.

Generally, the value displayed in the performance counter [Process \ private bytes] is slightly higher than the sum used in the above native memory and managed memory, this is because the application may have additional overhead or some code segments that are not shared by other processes.

In addition, the total memory usage E = A + B + C + D, then the following figure shows the usage of each parameter in the peak position: 23 GB () + 13 Gb (B) + 23 MB (c) + 27 GB (d )~ = 66 GB (e ). We can see that the peak value in [Process \ private bytes] is about 67 GB, slightly higher than the above synthesis.

Figure 2: memory usage of operator indexes during query, with a peak value of 23 GB ()

Figure 3: Total memory usage of events during query, with a peak value of 13 Gb (B)

Figure 4: Total heap memory usage during query, with a peak value of 27 GB (c)

Figure 5: total memory usage of processes during query, with a peak value of 67 GB (E)

Among the preceding five memory usage values, (a), (B), (d), and (e) are the objects to be focused on. (C) is often less important when tracking the memory fault of streaminsight applications, because its memory usage is much smaller than the other four factors, therefore, it is ignored in most cases.

 

At this point, the streaminsight application optimization getting started series has ended. Thank you for reading this article and wish you a smooth journey!