[Original] How to prepare an exquisite performance test report?

How to prepare a refined performance test report?

The above are my questions raised by zhihu. Thanks to Wu Tong for your answers. The words in the quote are my descriptions.
During the system process, you may need to perform tests on the entire system, including some benchmark tests, capacity tests, and stress tests. How should we prepare a refined performance report and what quantitative indicators should be considered during the testing?

There is a strange phenomenon. I have tried to figure out various performance testing concepts in this industry for six years, but there seems to be no unified standard so far, therefore, a performance testing method that everyone understands may be totally different from what I understand. Some people can define three methods, seven methods, or even more for performance testing, but no matter how many methods, there is always a need for them, performance requirements are the requirements of stakeholders. Such requirements are not easy to classify. Here are some of my answers to my understanding:

"How to make an exquisite performance test report ?"

The subject has already listed benchmark, capacity, and stress tests. I believe you have some knowledge about these tests. However, if you learn more about these tests, you will find that, in fact, not all stakeholders will be concerned about these tests.

A: Here we talk about the different dimensions of tests that different people care about in the project.

For example, as an end user of the system, it is impossible to care about the performance benchmark of the system. The end user is often concerned with the actual performance experience that the system can bring to him. It is like flash sales, you only care about how fast the payment page will respond, regardless of the level of the system performance benchmark, how the system performance capacity is, and how to scale the performance through device expansion.

However, these problems are precisely the concern of system O & M personnel. O & M personnel are also concerned with "configuration testing". By adjusting the software and hardware environment of the tested system, measure the test taker's knowledge about the impact of different environments/parameters on performance testing, and find the optimal allocation principles of various resources, as well as reliability testing and failure recovery testing, that is, whether the system can still provide normal response services after a device fails. for developers, they are concerned about the locks, memory leaks, and other design and implementation issues.

• User: actual experience performance
• O & M: optimized, available, and restored
• Development: Bug, performance

What I understand and refine the performance test report: it is a testing report that shows the user's actual performance experience, a performance testing report that informs the system O & M personnel of the system's actual performance risks, or a test report that can inform developers of the performance defect reports in system design and implementation, these problems often do not appear in the same report. Therefore, please first find out who the report is facing and what the relationship is.

Which quantitative indicators need to be considered during the testing process?

From the user perspective, the average response time can well measure the overall performance experience of the user, but the premise is that the overall distribution of the same request response time must be normal, otherwise, the average response time does not mean anything except misleading you.

Therefore, a 90 percentile of the response time is a more useful indicator that may cause unnecessary troubles for testers and developers, that is, the 90% request response time is in the range. This is a demanding indicator for the system, but it is very reliable.

Corresponding time interval as a speed indicator

In addition to the 90 percentile of the average response time and response time, the "standard deviation" of the response time is also an indicator to determine whether the overall user can obtain stable performance experience, in a "relatively stable" performance experience, the standard deviation (experience) is often half of the average.

Corresponding time standard deviation as a stability indicator

The throughput per second during access to application services is a kind of network resource overhead, but the system's usage of bandwidth actually affects the user's performance experience.

From the O & M and development perspective, there must also be their own performance indicator threshold standards, or the performance risks they are concerned about, for example, CPU usage, memory, hard disk performance counters, JVM memory, GC frequency, and pause time.

O & M and development use hardware consumption rate as an indicator

Summary

For project/product managers, the report focuses on the actual performance experience that users can gain under different business scenarios and different load levels in different scenarios; for O & M personnel, the report should focus on the risks of system resource overhead, the expansion of device resources, and the impact of configuration changes. for developers, the report focuses on design defects, locks, memory leaks, resource overhead, and release.