Performance: top priority of software testing

Performance testing plays an important role in ensuring the quality of software. It includes a wide range of testing content. The China Software Evaluation Center has summarized the performance test into three aspects: test the client performance of applications, test the network performance of applications, and test the server performance of applications. Under normal circumstances, the combination of the three aspects is effective and reasonable to achieve comprehensive analysis of system performance and Forecast of bottlenecks.

　　Test the performance of applications on the client

The purpose of the application performance test on the client is to check the performance of the client application. The test entry is the client. It mainly includes concurrency performance testing, fatigue strength testing, large data volume testing, and speed testing, among which concurrency performance testing is the focus.

◆ Concurrent performance tests are important

Concurrent performance testing is a process of load testing and stress testing, that is, gradually increasing the load until the System Bottleneck or unacceptable performance points, the process of determining the system's concurrent performance through comprehensive analysis of transaction execution indicators and resource monitoring indicators. Load Testing is used to determine the system performance under various workloads. It aims to test the corresponding output items of the system components when the load increases gradually, for example, the system performance is determined by excessive traffic, response time, CPU load, and memory usage. Load Testing is a process of analyzing software applications and supporting architectures and simulating the use of real environments to determine the performance that can be received. Stress testing is to obtain the maximum service level test that the system can provide by determining the bottleneck of a system or the performance points that cannot be received.

The purpose of the concurrent performance test is mainly reflected in three aspects: based on the actual business, select representative and key business operation design test cases to evaluate the current performance of the system; when the extended application function or new application will be deployed, the load test will help determine whether the system can still handle the expected user load to predict the future performance of the system; by simulating hundreds of users and executing and running tests repeatedly, you can identify performance bottlenecks and optimize and adjust applications to find bottlenecks.

When an enterprise organizes its own strength or entrusts a software company to develop an application system on behalf of it, especially when it is used in the production environment in the future, users will often have doubts, can this system withstand the simultaneous access of a large number of concurrent users? This type of problem is most common in database applications, Web browsing and on-demand video streaming systems using online transaction processing (OLTP. To solve this problem, we must use scientific software testing methods and advanced testing tools.

◆ Example: China Telecom Billing Software

As we all know, around the 20 th of every month is the peak of the municipal telephone fee delivery, thousands of charging outlets in the city started at the same time. The billing process is generally divided into two steps. First, you need to query the charges generated for the current month based on the phone number you have proposed, and then collect the cash and change the user to the paid status. A user looks simple in two steps, but when hundreds of terminals execute such operations at the same time, the situation is very different, so many transactions occur at the same time, the application itself, the operating system, the central database server, the middleware server, and the network device are all severely tested. It is impossible for decision makers to consider the system endurance after a problem occurs, and predict the concurrency endurance of the software. This is a problem that should be solved in the software testing phase.

At present, most companies need to support hundreds of thousands of users, various application environments, and complex products assembled by components provided by different vendors, with unpredictable user loads and increasingly complex applications, this causes the company to worry about problems such as poor serving performance, slow user response, and system failure. The result is the loss of company profits.

How can we simulate the actual situation? Find several computers and the same number of operators to perform operations at the same time, and then use the stopwatch to record the response time? Such a hands-on workshop-based testing method is impractical and cannot capture internal changes in the program. This requires assistance from stress testing tools.

The basic strategy for testing is automatic load testing. By simulating hundreds or thousands of virtual users on one or more PCs to execute services at the same time, the application is tested, records the processing time of each transaction, the middleware server peak data, and the database status. Repeated and real tests can thoroughly measure the scalability and performance of applications, identify problems, and optimize system performance. Knowing the system's endurance in advance provides a powerful basis for end users to plan the configuration of the entire operating environment.

◆ Preparations before concurrent Performance Testing

Test environment: Configuring a test environment is an important stage of test implementation. The suitability of the test environment seriously affects the authenticity and correctness of the test results. The test environment includes the hardware environment and software environment. The hardware environment refers to the environment where necessary servers, clients, network connection devices, Printers/scanners, and other auxiliary hardware devices are tested; software environment refers to the environment consisting of the operating system, database, and other application software when the tested software is running.

A fully prepared test environment has three advantages: a stable and repeatable test environment, which ensures that the test results are correct and that the technical requirements for test execution are met; ensure that correct, repeatable, and easy-to-understand test results are obtained.

Testing Tool: the concurrent performance test is a black box test executed on the client. Generally, it is not manually executed, but automated. At present, there are many mature concurrent performance testing tools. The selection is based on testing requirements and performance-to-price ratio. Famous concurrency performance testing tools include qaload, LoadRunner, benchmark factory, and webstress. These test tools are automated load testing tools. Through repeated and real tests, they can thoroughly measure the scalability and performance of applications, you can perform test tasks automatically across multiple platforms throughout the development lifecycle. You can simulate hundreds of users to run critical services concurrently to test applications.

Test data: in the initial test environment, you need to enter some appropriate test data to identify the data status and verify the test cases used for testing, debug the test case before the formal test starts, and minimize errors at the beginning of the formal test. It is necessary to back up the data status during the testing process. Creating initial data means storing the appropriate data and restoring it as needed. The initial data provides a baseline to evaluate the test execution results.

During the formal execution of the test, you also need to prepare the business test data. For example, to test the concurrent query service, the corresponding database and table must have a considerable amount of data and the data types should cover all the services.

Simulate real environment testing. Some software, especially the commercial software for the public, often need to evaluate the performance in the real environment during testing. For example, when testing the scanning speed of anti-virus software, the ratio of different types of files on the hard disk should be as close as possible to the real environment, so that the tested data has practical significance.

◆ Types and indicators of concurrent Performance Testing

The types of concurrent performance tests depend on the objects monitored by the concurrent performance testing tool. The following uses the qaload automated load testing tool as an example. The software provides DB2, DCOM, ODBC, Oracle, netload, CORBA, qarun, SAP, sqlserver, Sybase, telnet, tuxedo, uniface, WinSock, WWW, and Java Script for various test targets. different monitoring objects, supports windows and Unix testing environments.

The most important thing is the flexible application of monitoring objects during the testing process. For example, the current three-layer architecture is widely used, and the concurrency Performance Test of middleware is mentioned on the agenda, many systems use domestic middleware. You can select a Java Script monitoring object and manually write a script to test it.

Concurrent performance tests executed using automated load testing tools follow the following testing processes: test requirements and test content, test case preparation, test environment preparation, test script recording, writing, and debugging, script allocation, playback configuration and loading policy, test execution tracking, result analysis and problem locating, test report and test evaluation.

The concurrent performance test monitors different objects, and the main testing indicators are also different. The main testing indicators include transaction processing performance indicators and UNIX Resource Monitoring. The transaction processing performance indicators include the transaction result, number of transactions per minute, and transaction response time (Min: Minimum Server Response Time; mean: Average server response time; max: Maximum server response time; stddev: Deviation of the server response of the transaction processing server. The larger the value, the larger the deviation. Median: median response time; 90%: 90% server response time of the transaction processing) and the number of concurrent virtual users.

Application Example: Performance Test of "Xinhua News Agency multimedia database V1.0"

China Software Evaluation Center (CSTC), according to the "Requirements for performance testing of multimedia databases (phase I)" and GB/T 17544 "quality requirements and testing of software packages" proposed by the Technical Bureau of Xinhua News Agency, the performance of Xinhua News Agency's multimedia database V1.0 was tested using industry-standard load testing tools.

The purpose of performance testing is to simulate multi-user concurrent access to the multimedia database of Xinhua News Agency, perform key search services, and analyze system performance.

The focus of performance testing is to conduct concurrent testing and fatigue testing for the main retrieval services with high system concurrency load. The system adopts the B/S operating mode. In the concurrency test, we designed a number of concurrent test cases for single-checkpoint, multi-checkpoint, variant search, and hybrid search services in the Chinese Library, English library, and image library within a specific period of time. The fatigue test case shows that the number of concurrent users in the library is 200, and the test cycle is about 8 hours for single-checkword search. During concurrency and fatigue testing, the monitoring metrics include transaction processing performance and Unix (Linux), Oracle, and Apache resources.

Test conclusion: In the test environment and Intranet test environment of Xinhua machine room, the system is able to withstand the load pressure of 200 concurrent users in the case of mbps bandwidth, the maximum number of transactions per minute reaches 78.73, and the operation is basically stable. However, as the load increases, the system performance degrades.

The system is able to withstand the fatigue pressure of 200 concurrent users for a sustained period of about 8 hours, basically stable operation.

By monitoring the system's Unix (Linux), Oracle, and Apache resources, the system resources can meet the preceding concurrency and fatigue performance requirements, and there is still room for good use of the system's hardware resources.

When the number of concurrent users exceeds 200, HTTP 500, connect, and timeout errors are monitored, and the Web Server Reports memory overflow errors. The system should further improve the performance to support a larger number of concurrent users.

We recommend that you further optimize the software system and make full use of hardware resources to shorten the transaction response time.

◆ Fatigue strength and large data volume test

Fatigue Testing is the maximum number of concurrent users that can be supported in stable system operation. It can continue to run services for a period of time, by comprehensively analyzing transaction execution indicators and resource monitoring indicators, the system determines the process of processing the maximum workload and performance.

The fatigue strength test can be automated by tools, or you can manually write a program for testing. The latter accounts for a large proportion.

In general, the maximum number of concurrent users that the server can respond to requests normally and stably perform fatigue tests for a certain period of time to obtain the transaction execution indicator data and system resource monitoring data. If an error occurs and the test fails to be executed, adjust the test indicators in time, such as reducing the number of users and shortening the test cycle. In another case, the fatigue test is to evaluate the current system performance. The fatigue test is carried out for a certain period of time based on the number of concurrent users in normal business conditions of the system.

The big data test can be divided into two types: for some systems storage, transmission, statistics, query and other businesses to test the large data volume of independent data volume; A comprehensive data size test solution that combines Stress Performance Testing, load performance testing, and fatigue performance testing. The key to testing large data volumes is the preparation of test data. You can rely on tools to prepare test data.

Speed Testing is mainly for manual testing of key businesses with speed requirements. You can calculate the average value based on multiple tests and compare and analyze the results with the response time measured by the tool.

　　Test the network performance of applications

The focus of application performance testing on the network is to monitor network application performance, analyze network application performance, and forecast network by using mature and advanced automation technologies.

◆ Network Application Performance Analysis

The purpose of network application performance analysis is to accurately display how changes in network bandwidth, latency, load, and TCP ports affect user response time. Network Application Performance analysis tools, such as application expert, can be used to discover application bottlenecks. We can see the application behavior that occurs during each stage of application running on the network, and analyze application problems at the application line level. Multiple problems can be solved: Does the client run unnecessary requests on the database server? When the server receives a query from the client, does the application server spend unacceptable time contacting the database server? Forecast application response time before production; use application expert to adjust application performance on the Wide Area Network; Application Expert allows you to quickly and easily simulate application performance, based on the response time of end users in different network configuration environments, users can decide the network environment for application production according to their own conditions.

◆ Network application performance monitoring

After the system trial run, you need to know exactly what is happening on the network, how the application is running, and how many PCs are accessing the LAN or WAN; which applications cause system bottlenecks or resource competition? Network Application Performance Monitoring and network resource management are critical to the normal and stable operation of the system. By using the network application performance monitoring tool, you can get twice the result with half the effort. In this regard, we can provide network vantage. In layman's terms, it is mainly used to analyze the performance of key applications and locate whether the root cause of the problem is the client, server, application, or network. In most cases, users are concerned about which applications occupy a large amount of bandwidth and which users generate the largest network traffic. This tool can also meet the requirements.

◆ Network Prediction

Considering the future expansion of the system, it is very important to predict the impact of changes in network traffic and network structure on the user system. Make predictions based on the planning data and provide network performance forecast data in a timely manner. We can use the network predictive analysis capacity planning tool predictor: set the service level, complete the daily network capacity planning, offline test network, network failure and Capacity Limit Analysis, complete routine fault diagnosis, predict the impact of network device migration and network device upgrades on the entire network.

Obtain the network topology from the network management software, and obtain the traffic information from the existing traffic monitoring software (if there is no such software, you can manually generate traffic data). In this way, you can obtain the basic structure of the existing network. Based on the basic structure, you can generate reports and charts based on the changes in the network structure and network traffic to explain how these changes affect network performance. Predictor provides the following information: it helps users upgrade networks in a timely manner based on the predicted results to avoid system performance degradation due to critical devices exceeding the threshold value. Which network device needs to be upgraded, this can reduce network latency and avoid network bottlenecks; Avoid unnecessary network upgrades based on the predicted results.

　　Test the performance of applications on servers

You can use tools to monitor the performance of applications on servers, or use monitoring commands of the system. For example, you can use top commands in Tuxedo to monitor resource usage. The purpose of the test is to comprehensively monitor the performance of servers, operating systems, database systems, and applications on servers. The test principle is as follows.

(Omitted temporarily)

Figure: Performance Testing Principle of applications on servers

UNIX Resource Monitoring metrics and descriptions

Metric description

If the average load system is normal

Average Number

Number of conflicts detected on the Ethernet at the conflict rate per second

Process/thread exchange rate: number of exchanges between processes and threads per second

CPU usage (%)

Disk exchange rate disk Exchange Rate

Receive packet error rate: number of errors per second when an Ethernet packet is received

Packet input rate the number of Ethernet packets entered per second

Interrupt rate: Number of interrupts processed by the CPU per second

Error Rate of the output packet when an Ethernet packet is sent per second

Number of Ethernet packets output per second at packet Input Rate

Read Memory Page rate Number of pages read per second in physical memory

Write Memory Page rate: Number of pages written from physical memory to page files per second

Or the number of pages deleted from the physical memory.

Memory Page switching rate: Number of pages written to memory pages per second and read from physical memory

Process entry/exchange rate The number of processes entered in the exchange zone

Process exchange rate The number of processes output in the exchange zone

System CPU usage (%)

CPU usage in user mode (%)

Number of bytes blocked by the disk per second