How should the test strategy of Internet products be designed-------clock-in 11th day

Before starting today's topic, please think about why I will take the test strategy of Internet products alone to discuss, the test strategy of Internet products and traditional software products testing strategy in the end what is different?

The difference between the research and development process determines the different testing strategies
If the direct answer to the Internet products and traditional software products testing strategy is different, you will be somewhat confused, then according to I have been emphasizing the principle of knowing it, you can first summarize the two types of product development itself the biggest difference is what?

That is, the "fast" of Internet products.

In the previous article, it has been mentioned that the online cycle of Internet products is usually based on "days" or even "hours", while the traditional software products have a period of "month", even "year" as the unit.

The huge difference in the release cycle determines that the testing strategy of traditional software products is not necessarily applicable to the testing of Internet products, and the test strategy of the two must be significantly different in test execution time and test execution environment.

For example, for functional automated test cases, it takes 12 hours to perform a full regression test, which is not a problem for traditional software because the release cycle is long and the time for testing is plentiful.

Don't say full regression test execution takes 12 hours, even if it takes a few days and nights without any problems, like I used to do traditional software testing in Cisco, a complete regression test of the number of GUI test cases nearly 3,000, the number of API test cases is nearly 25,000, It takes nearly 60 hours to run through all the use cases.

But for Internet products, there are usually one to two releases in 24 hours, and the release process typically includes the entire process of code static scanning, unit testing, compiling, packaging, uploading, downloading, deploying, and testing. Obviously leave the test execution time is very limited, the traditional software frequently more than 10 hours of test execution time, in the Internet product testing, simply does not work.

Generally, Internet products require that full regression testing be performed for no more than 4 hours.

So, how to ensure the test quality and test coverage, the effective reduction of test execution time?

First, you can introduce a concurrent execution mechanism for testing, and execute the test cases concurrently with a test execution cluster that contains a large number of test execution nodes.
The test execution cluster, which you can simply understand, is a batch of machines dedicated to executing test cases concurrently. A common test execution cluster consists of a master node (master) and several child nodes (node). Where the master node distributes the test case to each child node, and each child node is used to execute the test case.
At present, many Internet enterprises have set up their own test execution cluster.

Secondly, you have to find a breakthrough in the test strategy, which is the topic I share with you today.
Next, I will briefly introduce you to the traditional software product testing strategy design, and then share the Internet product testing strategy, so you can through the traditional software product testing strategy review, to deepen your understanding of the Internet product testing strategy.

Design of testing strategy for traditional software products
Testing strategies for traditional software products, usually using the pyramid model shown in 1. The pyramid model is Mike. Mike Cohn, for a long time, is considered a best practice for testing strategy design.

Figure 1 Pyramid testing strategy for traditional software products
First, Unit Testing

The bottom of the pyramid is unit testing, which belongs to the scope of white box testing, usually by the development engineer himself, because the earlier the discovery of defects, the lower the cost of repair, so the traditional software product testing strategy advocates for unit testing high input, unit testing this layer will usually do more "thick".

In addition, traditional software products, the life cycle is relatively long, there will often be multiple versions of the continuous release, in order to be able to detect and quickly locate the problem in the later version upgrade process, the unit tests are repeated repeatedly each build process, which also reflects the importance of unit testing from another perspective.

Second, API testing

The middle part of the pyramid is API testing, mainly for the interface exposed by each module, usually using gray box test method. Gray box test method is a test technique between white box test and black box test, and its core idea is to use the code coverage of test execution to guide the design of test cases.

As an example of API interface testing, first design how to invoke test cases of the API in black-box mode, while also statistical code coverage during test execution, and then supplement more and more targeted test cases based on code coverage.

Overall, the number of API test cases is less than the unit test, but more than the upper-level GUI tests.

Third, GUI testing

The top of the pyramid is the GUI test, also known as the end-to-end (e2e,end-to-end) test, which is the closest test type to the real user behavior of the software. This is usually a simulation of the behavior of the real user using the software, that is, simulating the user's various actions on the software interface, and verifying that the results are correct.

The advantage of GUI testing is that it can simulate real user's behavior, directly verify the business value of software, the disadvantage is that the cost of execution is relatively large, even if the use of GUI Automation test technology, the maintenance and execution cost of use case is still very high. Therefore, you should avoid excessive reliance on GUI testing as much as possible.

In addition, the stability problem of GUI test is an important reason to hinder the development of GUI test for a long time. Even if you adopt many methods such as retry mechanism and abnormal scene recovery mechanism, the random failure rate of GUI test is still high.

Design of testing strategy for Internet products
For Internet products, Mike's Pyramid model is no longer applicable, I will through the GUI test, API testing, unit testing three aspects, to talk to you about the Internet product testing strategy changes, how should design.

First, the GUI test

The online cycle of Internet products determines that GUI testing cannot be carried out on a large scale.

The iterative cycle of Internet products determines that the time left for developing GUI automation test cases is very limited;

The frequent changes in the client interface of Internet products make it very inefficient to implement GUI automation testing, which is also the worst.
Because of the quick feedback in agile mode, in the next iteration it may be necessary to modify and tweak the client interface based on feedback, so the GUI automation test cases that have just been developed or even haven't been developed will have to be modified together.
This frequent modification is very detrimental to developing GUI automation tests. Because, the newly developed automation use cases only ran once, even once did not have time to run the need to update, resulting in GUI automation testing is not as efficient as manual testing.

As a result, the GUI test of Internet products usually adopts the test strategy of "manual-oriented, automation supplemented", and manual testing often uses exploratory testing idea to test the new development or newly modified interface function, while the focus of automatic testing is mainly on the relative stability and the basic ability verification of core business. Therefore, the GUI Automation test often only covers the core and directly affects the main business process e2e scene.

In addition, from the number of GUI test cases, traditional software GUI testing is heavyweight, there are thousands of use cases, because the traditional software testing cycle is very long, test cases can be queued slowly to execute, long time point also does not matter.

While Internet products require GUI testing to be lightweight, have you ever seen or heard of any Internet product designed with thousands of GUI test cases? The online cycle of Internet products directly determines that you are not allowed to execute a large number of use cases.

Second, API testing

You may now ask, since the Internet product is not suitable for heavy-weight GUI testing, then how to ensure its quality?

In fact, for Internet products, the focus of testing on the API testing, is the most sensible choice. Why is it? I have summed up the following five reasons for you.

API test case development and debugging efficiency than GUI testing is much higher, and test case code implementation of the comparison specification, usually is to prepare test data, initiate request, verify response these standard steps.

The execution stability of API test cases is much higher than that of GUI testing. The stability of GUI test execution is always a challenge, even if you use a lot of technical means (these specific technical means, I will explain the GUI test and then detailed deployment), it can not be 100% stable.
API testing is inherently non-performing, because the test execution process does not depend on any interface operation, but instead calls the backend API directly, and calls the procedure to compare standards.

The execution time of a single API test case is often much shorter than the GUI test. API testing can be performed in a concurrent manner when there are a large number of API tests to be performed, so the execution of high-volume API test cases can be completed in a short period of time.

Now many Internet products adopt micro-service architecture, and the testing of microservices is essentially testing different WEB service, that is, API testing.
In a microservices architecture, client application implementations are based on calls to backend microservices, and if you test every backend service, you will have full confidence in the overall quality of your application. Therefore, API testing of Internet products is very important.

API interface changes are generally relatively small, even if there are changes, in most cases also need to ensure the back compatibility (backward compatibility). The most basic requirement of the so-called back compatibility is to ensure that the original API calls remain unchanged.
Obviously, if the calling method does not change, then the original API test cases will not need to make big changes, so that the use case reusability is very high, which can guarantee a higher input-output ratio

It is obvious that these characteristics of Internet products determine that API testing can achieve good input-output ratio, so should be the focus of testing Internet products. This is why the test strategy for Internet products is more like a diamond structure.

2 shows this diamond-like test strategy, followed by the "Heavyweight API test, lightweight GUI test, Lightweight unit test" principle.

Figure 2 Diamond testing strategy for Internet products
Third, Unit testing

With the "Heavyweight API test" and "lightweight GUI test", I'll tell you why it's a "lightweight unit test."

In theory, whether it is a traditional software product or an Internet product, unit testing is an important means to ensure the quality of software from the source, so it is very important. But the reality is that Internet products really can fully carry out unit testing, and strict control of code coverage of enterprises is still rare.

Any existence will have its rationality, I think the main reason is that the Internet products "fast", quickly realize the function, quickly seek user feedback, quick trial and error, fast iteration update.

In this mode, the Internet product is the fastest implementation of the function and online, basically will not give you time to do comprehensive unit testing. Even if you reserve the time for unit tests, frequent iterations can also leave the unit test in a state of constant rewriting. Therefore, the original value of unit testing, it is difficult to be reflected in the actual operational level.

So, can internet products really do not have to do unit testing? The answer is no, just not the unit test strategy here to adopt the idea of "divide and conquer".

Internet products are often divided into application tiers and back-end services, and backend services can be further subdivided into application services and basic services.

Back-end basic services and some public application services are relatively stable, and for the system as a whole is "reaching", so the back-end service is necessary to carry out comprehensive unit testing, and for very volatile client applications and non-public back-end application services, it is seldom to do unit testing.

In addition, for some core algorithms and key applications, such as Bank Gateway Interface, third-party payment integration interface, but also to do more comprehensive unit testing.

In conclusion, the comprehensive unit testing of Internet products will only be applied to the relatively stable and core modules and services, while application-level or upper-layer business services rarely carry out unit tests on a large scale.

Summarize
Traditional software usually adopts the test strategy of pyramid model, and nowadays Internet products often adopt diamond model. The diamond model has the following four key points:

1. Use the API test in the middle tier to focus on comprehensive testing.
2. Lightweight GUI testing covers only the E2E scenarios that directly affect the core business process.
3. The top-level GUI tests often use exploratory test thinking to discover as many potential problems as possible in a manual test.
4. Unit testing uses the idea of "divide and conquer" and only conducts comprehensive unit tests on relatively stable and core services and modules, while the application layer or top level business does only a small amount of unit testing.

How should the test strategy of Internet products be designed-------clock-in 11th day