Comparison of performance testing, load testing, and stress testing in software testing

When interviewing testers, This is a good question: How do you define performance/load/stress testing? In many cases, people use them as the same terminologies that can be replaced by each other. However, the differences between them are quite large. This post is based on some of my own experiences. I wrote a simple comment on these three concepts. Of course, I also referred to some definitions in the test documents, for example:
"Testing Computer Software" by Kaner et al
"Software Testing Techniques" by Loveland et al
"Testing applications on the Web" by Nguyen et al

Performance Testing

The purpose of performance testing is not to find the bugs, but to eliminate system bottlenecks and establish a benchmark for future regression testing. Performance testing is actually a very careful and controlled measurement and analysis process. Under ideal circumstances, the tested software is stable enough at this time, so this process can proceed smoothly.

A set of clearly defined expected values is the basic element for a meaningful performance test. If you don't even know what the system performance is to be tested, it does not have any guiding significance for the methods you want to test *. For example, to test the performance of a web application, you need to know at least two things:
The expected load value for different concurrent users or HTTP connections *
Acceptable Response Time

When you know your goal, you can start to use the method of increasing the load on the system to observe the bottleneck of the system. Taking the web application system as an example, these bottlenecks can be stored in multiple layers. You can use multiple tools to find out where they are:
At the application layer, developers can use profilers to find low efficiencyCodeFor example, poor searchAlgorithm
At the database layer, developers and Database Administrators (DBAs) can use the specific database profilers and the query Optimizers)
At the operating system layer, System Engineers can use some tools such as top, vmstat, and iostat in UNIX operating systems to monitor CPU, inner, swap, and perfmon in Windows systems, disk I/O and other hardware resources; dedicated kernel monitoring software can also be used at this layer.
At the network layer, network engineers can use packet detectors (such as tcpdump), network protocol analyzers (such as ethereal), and other tools (such as netstat, MRTG, ntop, MII-tool)

From the test point of view, all the activities described above are a white box method, which reviews and monitors the system from the inside out to the outside. After the measurement data is obtained and analyzed, the adjustment to the system is taken for granted the next step.

However, in addition to the above method, the tester is running a load test on the tested system * (here we do not mix it with the load test-load testing concept we understand, translation for load testing), also adopted the black box method. For Web applications, testers can use tools to simulate concurrent users or HTTP connections and measure response time. The lightweight open-source load testing tools I have used before include AB, siege, and httperf. A more important tool is opensta, but I have never used it. I have never used the grinder tool, but it ranks top in what I will do.

When the results of the load test * show that the system performance has not reached its expected goal, it is time to adjust the corresponding and database. At the same time, you must ensure that your code runs as efficiently as possible, and that the database is optimized under the given operating system and hardware configuration. Practitioners of test-driven development (TDD) will find this context framework very useful *, for example, you can enhance the junitperf * of Mike Clark of the existing unit test code through the load test * and the function of the time test *. After a specific function or method is parsed * and debugged, the developer can go to junitperf, put its unit test * to ensure that it can meet the performance requirements of load and time. Mike Clark called this "continuous performance testing ". By the way, I have also made a preliminary study of Python-based junitperf, which is called pyunitperf.

If the application has been debuggedProgramAnd the database, the system still does not meet the expected performance goals, in this case, there are still some other debugging procedures * can be used for the above several layers. The following is an example of how to improve the performance of the Web application system beyond the application code:
Use Web Cache, such as the device provided by squid
Static pages with high access volumes to avoid calling databases with high access volumes.
Scale the structure of the web server horizontally using the load balancing method *
After the database group is horizontally scaled and divided into read/write servers and read-only servers, load balancing is also required for the read-only server group. *
By adding more hardware resources (CPU, memory, disk, etc.), vertically scale the web and database server group
Increase network bandwidth

Because the current web application systems are very complex systems, performance debugging sometimes needs to be artistic. You must be very careful when modifying a variable and re-measuring it. Otherwise, there will be many uncertain factors that are difficult to determine and repeat in the change *.

In a standard test environment, such as a test experiment, it does not often reproduce the server configuration environment in actual applications. In this case, the segmentation test environment, that is, a subset of the actual production environment, can be used. However, the expected performance of the system also needs to be lowered.

The "Run load test *-> measure performance-> debug System" cycle must be repeatedly executed until the tested system meets the expected performance standards. At this time, the tester can understand how the system works under normal conditions. At the same time, these can be used as a standard for evaluating the performance of the new version of the software in regression testing.

Another goal of performance testing is to establish a set of benchmark data for the tested system. Such industry-standard benchmark data is available in many industries, such as those published by TPC. Many software and hardware manufacturers have carefully debugged their machines to rank top in the TCP rankings. Therefore, you should be very careful to explain that during your testing, you did not perform all tests in a wide variety of software and hardware products *.

Load Testing
We have all experienced load testing during performance testing and debugging. In that environment, it means that automatic tools are used to continuously increase the load on the system. For Web applications, the load is the number of concurrent users or HTTP connections.
The term "Server Load balancer" is generally defined in the test documentation as the process of adding the maximum number of tasks that can be performed on the system under test. Load Testing is also called"Capacity Test", Or"Durability test/durability test"*

Capacity Test example:
Test the Text Processing Software by editing a huge file
Test the printer by sending a huge job
Test the mail server through thousands of user mailboxes
There is a special capacity test called "Zero Capacity Test", which adds a blank task to the system for testing.
Durability test/durability test example:
The client is continuously running in a loop for more than one extended period *.
Purpose:
Find some buckets that are not found in the rough test conducted in the previous stage of the test process, such as memory management bugs, memory leakage, buffer overflow, and so on.
Ensure that the application reaches the performance baseline determined in the performance test. This can be achieved by loading a specific maximum load when running a regression test.

Although performance testing and load testing seem very similar, their purposes are still different. On the one hand, performance testing uses load testing technologies and tools, as well as different load levels to measure and baseline systems. On the other hand, the load test is based on some defined load levels. Generally, after adding the maximum load to the system, the system should still be able to provide all the functions. Here, we need to make it clear that load testing is not to overload the system so that the system cannot work, but to make the system sound like a machine full of oil *.

In load testing, I think it is very important to have enough data for testing. I learned from my experience that if we don't need Big Data * for testing, there will be a lot of serious bugs. For example, there are thousands of users in the ldap/NIS/active directory database, thousands of mailboxes in the mail server, G-to-G tables in the database, and deep file or directory layers in the file system, and so on. Obviously, testers need to use automated tools to generate these huge datasets. Fortunately, any good scripting language is competent for these tasks.

Stress Testing

Stress testing refers to the system crash by loading excessive resources to the system or the system has no resources that should make the system operate normally (in some cases, it can also be called negative testing ). The main purpose of this crazy behavior is to ensure that the system fails and can be properly restored. What is the Recovery feature called recoverability.

When Performance Testing requires a controllable environment and continuous measurement, stress testing is a joy that can lead to confusion and unpredictability: from this point we can see that the author is a very good tester ). Take the web application system as an example. Below are some feasible stress testing methods for the system:
Twice the number of baseline concurrent users or HTTP connections
Randomly shut down and re-opened the port of the hub/router on the network connected to the server (for example, it can be implemented using the SNMP command)
Disconnect the database and restart it.
When the system is still running, create a raid array.
Processes that consume resources (such as CPU, memory, disk, and network) on Web and database servers

I am sure some testers who often use unconventional methods to destroy the system can further enrich this list. Stress testing is not simply about destroying the system for the pleasure of destruction. In fact, it allows test engineers to observe the system's response to a fault. Does the system save the status when it fails? Is it suddenly crashed? Is it just hanging there? Does it have some reaction when it fails *? After the restart, can it restore to the previous normal running status? Does it display some useful error messages to users, or just some very hard-to-understand hexadecimal code? Is the security of the system reduced due to unexpected faults? You can keep asking these questions.

Conclusion
I understand that I only mentioned some basic problems, tools and methods that should be worth noting in performance/load/stress testing. I personally think performance testing and debugging are a very interesting topic. I will post more relevant content in the future.