Stability testing of performance tests (reliability testing)

The last two days in the system review performance testing knowledge, combined with previous performance testing experience has some summary, hope to write to share with you, hope to mention valuable advice, common progress ~

Concept

First of all, performance testing : Performance is a non-functional feature of the software, he is concerned about whether the software has completed a specific function, but the software in the completion of a specific function is the timeliness of display. Timeliness represents different indicators from different perspectives:

1. User: Response Time
2. System administrator: Resource utilization, scalability, system stability , system capacity
3. Developers: System architecture, database design, design and code implementation

It can be seen that system stability is of great significance to system administrators, and the stability is also directly affecting the "response time" that the end user cares about, so it is very important to test the performance of the stability test.

Stability Testing (also known as reliability testing) allows the system to run continuously for a period of time (typically 7x24 hours) by loading a certain amount of business pressure on the system, and detecting whether the system can run stably.

How to Implement

• Identify and confirm the main business of the software (whether stability testing is required)
  • The focus of the stability test in the software is the most valuable place, such as a ticket system, it is the most valuable is when a certain number of users at the same time to buy tickets is the system of the corresponding times, resource utilization and other can be normal and stable, rather than how users add new contacts, modify personal information, etc.

• List the main user scenarios and corresponding load levels
  • User scenarios can be set according to the main business of the software
  • A clear definition of the load on the main scene (or a load test to verify the load on the user scenario, which will be used as a standard load in the stability test)

• Develop a stability indicator model (Modeling)
  • Create a suitable and reasonable stability indicator model based on user scenario modeling (there will be an example later)

• Test environment Preparation (configuration of the hardware and software environment: the source of the configuration can be the customer environment simulation, the configuration required by the requirements document, or the best configuration test results)
• Identify key performance indicators (KPIs) for stability
  • The system metrics used to describe the stability test, such as response time, CPU, memory usage, etc., need to be defined according to the specific business

• Test execution and data collection
Analyze test results according to the corresponding stability indicator model (Modeling)
• Apply test results to the stability test model to see if the stability requirements are met
• Continuous improvement (if necessary)

Stability Indicator Model (example) below we take a bi (business intelligence) example to conduct our stability test modeling

1. Software main business: extracting (ETL) customer-care data from a large amount of metadata and ultimately generating a report (this article takes Microsoft platform bi for example: SSIS,SSAS,SSRS)
2. User scenario: Use the SSIS package for ETL operations to import metadata calculations into the data cube (cube).
3. Typical load: 3,000 users per hour, 100,000 data, 7x24 hours
4. Test environment: Configuration specified in the requirements document
5. Main Performance Index:
   1. ETL Time: 9 minutes, Difference: 1 minutes, Variance: <0.1
   2. System Related: Cpu,memory,private mbytes/sec, etc.
6. Stability Indicator Model:
   1. Calculation formula

　　

2. Stability model

3. It can be seen from the chart that:

• • ETL limit is 12 minutes (that is, if more than 12 minutes to prove bottlenecks, need to investigate)
  • ETL average of 9 minutes
  • The upper and lower sides of the control line are AVG plus minus 3 times times the variance
  • The actual usage time is distributed around the average value (the standard for the same direction can not be 7 consecutive points: such as 7 consecutive actual detection values are above the average, then need to investigate)

4. System-related indicators can also be analysed in this way. (Of course, systems with low system resource requirements can analyze stability directly by using the footprint curve.)

7. Perform the test to obtain the results, and apply the indicator model for analysis

Summarize

In general, the stability test is used to verify whether the product can run stably for a long time under a certain load, its main purpose is to verify the ability, and to find the factors of the system instability during the verification of the ability and to analyze and solve it.

Stability testing of performance tests (reliability testing)