[Tool] four principles for designing test cases

The most basic requirement of the test case design: cover the function to be tested. This is not a basic requirement, but it is just a simple sentence, to achieve full coverage, A comprehensive understanding of the features of the product to be tested, a clear scope of testing (especially what is not required for testing), and basic testing techniques (such as division of equivalence classes) are required. If the above requirements are met, is the designed test case a good test case? Answer: theoretically, it is far from true engineering. The reason why there is such a difference between theory and practice is that we should not consider it theoretically.

The most basic requirement of the test case design: cover the function to be tested. This is not a basic requirement, but it is just a simple sentence, to achieve full coverage, A comprehensive understanding of the features of the product to be tested, a clear scope of testing (especially what is not required for testing), and basic testing techniques (such as division of equivalence classes) are required. If the above requirements are met, is the designed test case a good test case? Answer: theoretically, it is far from true engineering. The reason for this difference between theory and practice is that, in theory, it is not necessary to consider things, but in actual engineering-cost. The costs include: test plan cost, test execution cost, automated test cases, test automation cost, and test analysis cost, and additional costs introduced by technical limitations, bugs in the test environment, human factors, and unpredictable random factors.

Due to the intervention of cost factors, the principle of test cases designed in the project is not only "covering the functions to be tested by the domicile, the following are the other four principles I have summarized based on my work experience. I hope you can make a brick and correct yourself. These principles are especially applicable to test cases that require automation and are frequently executed.

1. Single case coverage minimization principle.

This principle is the "boss" of all these four principles. It is also the most easily forgotten and ignored in the project. It affects other principles more or less. The following is an example. If you want to test function a, it has three sub-functions: A1, A2, and A3. You can use the following two methods to design a test case:

Method 1: Use a test case to cover three sub-functions-test_a1_a2_a3,

Method 2: Use three separate use cases to cover the three sub-functions-test_a1, test_a2, test_a3

Method 1 is suitable for small projects, but for projects with a slight scale and quality requirements, method 2 is a better choice because it has the following advantages:

Define the coverage boundaries of test cases more clearly

The test results are more oriented to product problems.

Test cases have the lowest coupling degree, and the interference between them is lower.

These advantages directly bring about the lowest debugging, analysis, and maintenance costs for test cases. Every test case should be as simple as possible. It should only verify the content you want to verify. Do not "put your eyes on the rabbit" or bring everything in, this will only increase the burden and risks of the test execution phase. David astels described this in his book test driven development: A Practical Guide. The best test case has only one assert statement. In addition, it is easy to combine and generate new tests by covering simple and clear test cases. The ordered test concept is introduced in Visual Studio.

2. functional principles of replacing product documentation with test cases.

Generally, at the initial stage of development (the first two days of each scrum Sprint), we use Word documents or OneNote to record the product requirements, function descriptions, and any details that can be determined at present, outlines the features to be implemented to facilitate team communication and refinement, and reach consensus on product functions within the team. Assume that after reaching a consensus at this time, the function described is A. As the product development goes deeper, the team will have a new understanding of the product features, and the product features will be further refined, the final function implemented at the end of an iteration or sprint is probably a +. In this way, after constantly listening to and absorbing user feedback and modifying product functions, the function originally described as a may eventually become Z after multiple iterations. This is the time to go back to the Word documents and OneNote pages, but still record. This is because few people constantly update those documents to accurately reflect the current status of product functions. It's not hard to do it! It should be noted that the early word or OneNote documents are still necessary. It can at least ensure that the team has a certain and accurate understanding of the functions to be implemented in the early stage of iteration.

Is there anything that can accurately describe the features of the product? Answer: Of course, there are product code and test cases. The product code implements product functions, which must accurately describe the current features of the product. However, due to various programming technologies such as object-oriented, abstract, design patterns, and resource files, this makes it difficult for product code to be easily understood. It is often necessary to read the code on the premise of knowing the product function, rather than reading the code to understand the function. A good code has detailed comments, but the comments here are the explanations and comments of the implementation code, not the descriptions of product functions. Here is a very old Blog reading code is hard, which describes how to make the code more readable.
Only test cases are available, and the test should faithfully reflect the product functions. Otherwise, the test cases will fail to be executed. In the past, we used test cases as test cases. In fact, the understanding of test cases should be upgraded to another level. It should be able to play the role of the product description document. This requires that the prepared test cases should be detailed enough, and the organization of test cases should be subject to adjustment, priority, and priority. A general tool such as Word, Excel, or OneNote alone is far from complete, more dedicated test case management tools are needed. For example, Microsoft test Manager is introduced in Visual Studio 2010.

In addition, for automated test cases (whether at the API or UI level), code writing should also have a different product code writing style. readability and description should be the key considerations. In the test code, of course, we can introduce good design ideas such as object-oriented and design patterns, but we must use them appropriately. process-oriented coding methods are more conducive to organization, reading, and description.

3. single input cost and multi-input cost principle.

This is not so much a principle of judging test cases as a thinking angle and principle of thinking about problems. Cost is always the primary factor to consider when making decisions for any project. The same is true for tests in the project, cost considerations should also be objective and comprehensive in the entire phase of test design, implementation and maintenance. When you encounter some difficult problems during the test and need to make decisions, you can try to analyze them from the cost point of view, which may be helpful for your decision-making.

The costs in the test can be divided into one-time investment cost and multiple-time investment cost based on the time span. For example, writing test cases can be considered as a single investment cost, because the writing of test cases is generally carried out in the testing plan phase (the start phase of each scrum Sprint, although there will be minor changes in the later stage, most of them are basically formed in the initial design phase; the same is true for automated test cases, which also belong to one-time investment; test cases (including: manual and automated test cases) execution is a multi-time investment cost, because all test cases (or BVT tests only execute high-priority test cases) are required for each new build) analyze the test results, debug failed test cases, and determine the cause of the failure of the test cases (product defects, test case defects, test framework defects, or random problems lead to the failure of the test cases ), to verify whether the overall quality of this version has reached the specified standard.

All the thinking that we need to introduce the single and multiple costs is to distinguish the impact of different activities in the test on the test cost, in this way, we can make reasonable investment in different stages and make correct decisions, so as to ensure that the test work can be carried out to the maximum extent while the test cost is limited. For example, when we realize that the design and automation of test cases are a one-time investment, while the execution of test cases is a repeated investment, we should actively think about how to improve the efficiency of testing execution that requires repeated investment, and give priority to the efficiency of multiple input activities when one input and multiple activities need to be balanced, in fact, there is a lot of work to do here.

For example, the first principle-the minimum overwrite principle for a single use case-is a good example to test the three functional points A1, A2, and A3 of function, on the surface, the use case test_a1_a2_a3 is the simplest in design and automation, but it brings many problems in the repeated execution phase:

First, the failure analysis of such cases is relatively complex. You need to confirm which function points have caused the test failure;

Second, the debugging of automated use cases is more complex. If there is a problem with A3 functions, you still need to constantly pass through A1 and A2 before reaching A3, which increases the debugging time and complexity;

Third, manual test cases with multiple steps increase the uncertainty of manual execution, while those with multiple steps increase the possibility of automatic execution failures, especially those with UI automation technology;

Fourth, (last but not least) coupling irrelevant functional points to reduce the possibility of early detection of product regression defects. For example, if test_a1_a2_a3 is an automatic test case and is executed in the order of A1-> A2-> A3, when A1 has a bug, the entire test case fails, a2 and A3 were not tested and executed. If it takes a long time to fix the A1 bug for some reason, test_a1_a2_a3 is always considered to fail because of the A1 bug, while A2 and A3 are never overwritten, there are potential risks and vulnerabilities. When you finally fix the A1 bug before the product is released and take it for granted that test_a1_a2_a3 should pass, the A2 and A3 problems will pop up at this moment, you have to work overtime to fix the A2 and A3 problems. It's not an alarmist. When the A2/A3 code is related to A1 bug fixing, when you have many test cases that are so designed, the problem may be worse... ..., Really! :(

To sum up, design such as test_a1_a2_a3 reduces the workload and risks of one-time design and automation, but increases the workload and risks of testing execution that requires multiple inputs, therefore, when you need to make a decision (in fact, this decision often happens, especially when designing test cases), choose test_a1_a2_a3, test_a1, test_a2, and test_a3. Please consider the cost.

4. Simplify the principle of Test Result Analysis and debugging.

This principle is actually the previous one-single-input cost and multiple-input cost principle-expansion and continuation of automated test cases. When writing automated test code, you should focus on how to make the test result analysis and test debugging easier, including the use case log and debugging auxiliary information output. Because the execution of test cases is a lot of investment, testers often need to analyze test results and debug test cases. The investment in this part of activities is considerable. Sometimes, some auxiliary APIs for testing the framework to provide functions can help to achieve this principle well. For example, coded UI test provides similar APIs. For details, see-Vs 2010 test function learning (18)-coded UI test, to help automated test cases based on the coded UI framework have a better debugging experience.

The testing theory points out a major direction for the testing work. In actual engineering, we also need to constantly "Activate" these theories so that the theory and practice can better fit together. In my opinion, software engineering projects, whether successful or not, are invaluable to each of our participants. As a caring person, we have learned many things that have never been mentioned in books. As long as you constantly observe, understand, and summarize, you will have more understandings, understandings, and discoveries of yourself. There are many people who write books and praise the code and test. In fact, the project is also very beautiful, just to see if you can look at it more objectively.

[Tool] four principles for designing test cases