Discussion on an effective test strategy

In a recent large project, we set a goal early on: not using a large number of QA staff in software to focus on manual testing. Manual testing to find bugs is extremely time-consuming and costly, prompting the team to try to embed quality within the product as much as possible. But that doesn't mean manual testing is worthless because people can always give you some special surprises on how to use the software.

This is a 18 month or so long term project, and the follow-up will continue to be updated. In the early stages of the project, the team realized that the key to the success of the project was a good test strategy, especially for our team: 1 to continuously improve the team's productivity as the project time progresses. 2 regardless of whether the change is big or small can have enough confidence.

It took us a long time to determine an effective strategy. This is largely because we have to learn how to make our programs measurable on all levels. Although all project team members have experience with TDD (test-driven development), this is not enough to establish effective testing strategies.

Test tiering

It's annoying to classify different tests. Functional testing, integration testing, unit testing, acceptance testing, slow tests,fast tests,ui testing ... Wait, wait. Then we found that the tests that belonged to us had three main types:

System test

Subcutaneous test

Unit Test

The difference between them is mainly the scope of the content being tested. System testing refers to the application of the external interface to operate, whether the object is a browser, file Drop-down menu, queue, window form application or something else.

The subcutaneous test runs under the external user interface. If the test is a Web application, the subcutaneous test in our understanding means that in the real class
And the form objects that are sent through the command processor when the environment is deployed. Bypassing the UI layer logic to reach the domain layer directly. Sends the form object to throw the execution result of success/failure.

For the bottom tier, we have unit tests. Unit tests are used to test a class, and can be one of the fast test or slow test. Fast test is a generic TDD test for incremental class design. Test doubles was once considered necessary, but rigorous, interactive testing is not so valuable unless the system interaction is very interesting. We also have slow unit tests, which can also be categorized as interactive tests. Of course they can also be categorized as repository tests, persistence tests and so on.

Unit: Subcutaneous: System Testing in our testing work of their respective proportion is almost 10:2:1. To complete the project we did about 5,000 unit tests, 1000 subcutaneous tests, 500 system tests using WAITN and Gallio driver browsers. The 6,000 unit/subcutaneous test takes approximately 10 minutes to execute, while the remaining 500 UI tests take approximately 50 minutes to complete.

Unit Test Strategy

Unit tests are developed in a rigorous TDD mode. We write unit tests before the feature is implemented and use these tests to drive code design. These tests can help identify design problems, encapsulation problems, code odors, and so on.

We strive to avoid writing code that is purely designed to provide testing. They often mean that we have design problems, misplaced responsibilities, or that encapsulation has been compromised.

As we get deeper into the project pipeline, we are less focused on interactive testing. If you're really interested in interaction, then the interactive test through the mock is just fun. But more often than not, we are more interested in the additional role, and the interaction is just an implementation detail. Instead, we often do mock (mock) code that is slow or unpredictable, such as the repository, the appearance of external services, configuration classes, and so on. Otherwise, our finite simulations merely simulate the observations that we are interested in.

At some point in a large project, in order to extract the concept of faster function delivery, design often requires large refactoring. In our last project, we discovered the following ideas:

AutoMapper

Processing a form as a separate command message

Input Builders

With these, unit testing is the guarantee of refactoring. But only when we rely on these tests to capture all the interesting behavior in the application can we have a guaranteed effect. To allow for efficient medium and large refactoring, we need to add additional test levels.

The strategy of subcutaneous testing

Under the skin test, as the name suggests, all tests are performed under the user interface. In an MVC application, the subcutaneous test is all that follows the test controller. For Web service, all tests are done under the terminal. The idea of subcutaneous testing is that the top of the application does not perform any actual business logic, but only the connection between the external interface and the underlying service.

This column more highlights: http://www.bianceng.cn/Programming/project/

The importance of subcutaneous testing is reflected in our desire to test business logic while running the entire system while throwing out external connections such as user interfaces and external services. Relative to the unit test focus on the design of small modules, subcutaneous testing does not concern the design, but the whole system to test the basic input and output.

To establish effective subcutaneous testing, we can try to build uniform pinch points through common logical processes. For example, we can create a command message processing system, or a common query interface. On the most recent processing batch file project, each row in the batch file is converted to a message. Then we create a message, send it to the system, and then verify all the exceptions that are handling the message.

Because subcutaneous testing is based not on design but on advanced (business) behavior, they are ideal scenario based test strategies, such as BDD or TestCase Class per fixture mode. If we are going to do large refactoring, we need these high-level tests to establish a comprehensive security for business practices. Since the subcutaneous test is more focused on end-to-end logic, it is also an important target for the completion of the Logo function point.

While subcutaneous testing allows us to perform large refactoring safely, it still does not guarantee that we can safely upgrade the system to a production environment.

System-wide test strategy

At first, we called system-wide testing "UI testing" until our projects were increasingly involved in integration strategies. This is no longer a browser in our system, but instead a message from the REST endpoint, FTP, or batch file. UI tests are only a subset of system-wide tests. The idea behind system-wide testing is that we want to test them in terms of how the software is used in the production environment. For an MVC application, it's a browser-based test. For batch files, we use the actual files. Use the actual HTTP request for REST. For messages, the actual queues and messages are used.

If we want to know whether an application can work as expected before it is deployed to a production environment, an effective and efficient way is to create an automated test to test the entire system. If I can have the UI test log in to the application, create an order, and then I can verify that an order request has been generated, then I will feel good.

A common misconception about system-wide testing is that it is a black-box test. However, the whole system test is characterized by the fact that you have to know what's going on inside the system. In fact, system-wide testing can even use a domain model to generate data, rather than a system-back door built purely for testing purposes. A big hole that many teams fall into easily is that they are not tested in the same code path as the production environment, which causes the system to be in an odd, invalid, and difficult state to handle.

In our project, the system-wide test code is the last code we write before we claim a feature/story is done. The cost of manual testing is too high and unreliable for a "completed" feature that describes a feature. But if I can test it in a production environment and do it through the exact same external interface, then the manual test will be successful.

Overall consideration

In an untested application, as a means of increasing coverage, we found that the most valuable test strategy was actually starting with system-wide testing and then moving down until unit testing. Let's start with the most relaxed and simple assertions, then slowly move down until the cell level logic. In newly developed applications, we tend to focus not only on a particular area, because all tests are important for a system that requires long-term maintenance.

This test strategy does require a certain amount of investment. We find that the overall consideration is particularly effective when we know that the application has a decisive effect on the customer's business. If an application has a decisive effect on the business, it will have to face changes. When the change comes, we'd better be able to implement the change safely without affecting the customer's business.