On the effective renewal of test cases and the pesticide paradox

On the effective renewal of test cases and the pesticide paradox

In 2014, our team tried to push one thing-the reverse side of the problem of the backend (customers, customer service, manufacturing, etc.) to the test case, to the test case library, to avoid the subsequent duplication of problems--earlier in the entrepreneurial program, Liu asked a contestant, as the Boss, You have to deal with something first thing every day. The contestants say a lot according to their priorities and importance. "You should prioritize recurring problems," Liu said.

The theory of re-disk is the housekeeping skills of Lenovo, which only borrows this meaning.

Try to do this for some time, the already formed reverse supplementary test cases, extended to the relevant test case library, and then in the actual implementation and inspection, a period of time roughly the following several phenomena:

One, the vast majority, does not carry out at all.

Second, a small part, selective execution.

Three, a small part, rewritten, incorporated into the original test case to execute.

There are many reasons for the first phenomenon, aboveboard and not so aboveboard-and I would prefer to think of the reasons mentioned below.

As for the second to third phenomenon, the question I was asked was: if we do not follow the written format, pull out the individual pulls out and have the result, then it is impossible to use the manual or the tool to count whether the new use cases have been executed? The data can not be taken, thus it is not possible to determine whether the testing of the optimization and progress.

Put down the complex execution and inspection of the specific problem, just from the test itself- a problem arises, whether to take the problem of the steps, defects of the scene, similar to the possible logic, are written in the test case, in subsequent projects, repeated execution?

The answer is not necessarily--the test design is an area of the master to do things, rather than the mere yiren of the rigid description. Or, to put it another way, test cases are at the heart of the test and are creative things, rather than having an absolutely correct methodology that can be done once and for all.

Cite a number of different examples to illustrate the complex relationship between appearances and nature:

First, the cause of the problem, what is its frequency?

ex1--If the problem is because the developer mistakenly hand over a piece of code comments, or because of a variety of clerical errors caused by defects, found after the revision code recompile, problem resolution.

Then the probability of this problem is one-time. Once this flaw is repaired, the probability of a recurrence is very small-unless the code is left behind, and then developed and boldly modified with some old code. Then own leader has no code review, directly submitted. Then it is possible for the problem to be uncovered. Normally in this case, it is not necessary to write a few cases, subsequent projects are executed every time.

ex2--has a resource, a number of modules will be called, and these modules business logic coupling more tightly, and the joint tune has been doing a bad job, even because of the shortcomings have happened many times in the end is your my his such a broken thing.

Then the problem should be the probability of occurrence. After this flaw repair, not only this flaw produces the operation follow up to add, even these several modules calls the resources the method, previously did not have too much attention, the follow also should strengthen the test design appropriately.

Ii. What are the components, sub-branches and versions of the problem?

ex3--in embedded devices, the word "Yanzhou" cannot be displayed as "mouth". The problem is that when the embedded device memory is small, it is possible that the font uses a primary font, then all of the two-level font text will show an exception.

2.1, with uniqueness:

If the entire company is using a unified font library. Then only update this font, all the embedded device two-level font problem will be resolved, this defect once repaired, do not need to be included in the test case.

2.2, there are multiple branches:

There are a lot of outsourcing projects, to display different fonts, different countries of the language, in short, there are a lot of branch font exists.

2.A, if there is a good comprehensive defect analysis and the spread of notification methods, we each repair, and do not need to write to test cases, because it is a one-time behavior.

2.B, if there is a certain degree of defect awareness, different sub-tributaries can be perceived, but the timeliness is poor, then this matter will be cured in the test case, the implementation of a period of time.

2.C, if there is not a certain degree of defects in the way, we are based on a flow, the subsequent development of maintenance, then it must be written in the test case, or even to organize small special tests to focus on the issue of different versions.

Third, is there a strong sequential dependency relationship?

ex4--If a problem is strongly related to the order of business logic, it is necessary to pass the necessary 1, 2, 3, 4, 5 steps to lead to a certain bug. From the test staff's work, can find such a bug (commonly known as God-level bug), it is their own knowledge of the business of the best performance, and even as their own anecdotes continue to boast.

However, this bug, after regression test, really do not have to form a test case, so that each subsequent project, to repeat the execution-the strong business order relationship repair, the follow-up nature will not appear. As to whether there are other implicit logic, it is another matter whether you need to perform other branch state tests.

Four, the verification conditions do not have?

ex5--A variety of complex external vendors to pass the problem.

This type of bug, mostly in the field, through packet analysis, stream analysis, and then constantly replace the temporary version to repair. If the agreement is standardized, can be strengthened in the testing process, if the manufacturers of the rapid development of non-standard protocol, no way, can only solve the scene.

So, you can write a, a device, you need to access a manufacturer of XXX products/b manufacturers of yyy products, the ZZZ function test. However, these test cases are not enforceable.

For this kind of interconnection problem, the best solution is to find a device model a lot of customers, maintain good customer relations, the release of new products, their own equipment in the past, the joint adjustment is done.

This example requires testing strategies and scenarios for such problems, rather than stiff additions to test cases that cannot be performed.

ex6--after a long period of operation caused problems, such as XX equipment after three years of operation, the device aging, or the version, file loss without reason.

This involves the reliability and the flash repeatedly read, fragments and black swan events and so on respectively.

To test this type of problem, in a short period of time to simulate the effect of three years, only through the amount of test equipment, and through the formula to deduce the approximate stability. Written in the test cases, in the daily work, it is clearly impossible to achieve, it is better to honestly do special testing, concentration of manpower, equipment and so on. Take this kind of problem once and for all.

The above is the first concept of reverse refining test cases from defects-learn from the problems, avoid repeating the same problems in the future, thinking and logic are right. But definitely does not mean than gourd painting scoop, some problems may be one-time, some problems may have a bigger problem, some problems you know but can only see the probability and input-output ratio, or try to solve by other methods.

The second concept is related to the team and people, and the real workings of a team are often only insiders know. In the same way, the real cause of the problem is often hidden within a team, so whether you can write accurate test cases, and only the team within the group can make a definite. This means that if the test case update is not a proactive trigger within your own department, but is driven by a third-party department (Quality department, Process department), then you are destined to get only some of the goods.

The real cause of the problem will make your case completely different.

for example: the software client decodes no sound.

But if you add a test case: "After the software Installation/update is successful, check that the codec status is normal, the expected result: image, sound normal." "The person who gets this use case will think that the writer's show is funny, so basic things have been measured, but also serious new, the final result is either not executed, or no brain hook through."

But this most basic problem, will appear again and again, behind nature has deep-seated things exist.

EX7: Compatibility issue, an audio format has been flipped, not considered for compatibility.

Earlier versions of the audio stream sent over, decoding failed, this no sound is the standard compatibility problem-so the supplementary test case, it will be written, and each product version of the compatibility test to see whether the audio is normal.

Does it seem to catch a real problem? But this use case is also a theoretical comprehensive use case, it is impossible to actually be executed (reference VI, test case enforceability)--there may be more than 20 historical products, the historical version may also have more than 20. You move the lip interconnection, and not that is not the test environment there are so many devices, is in all the smooth situation, the version replacement and test round, also need a few days. In the context of testing resources and time-intensive projects, this case will be carried out in a strange way.

In combination with the above view, see whether the version of the codec component has changed, and then decide whether to perform the compatibility test between different versions of the codec, then through the equivalence class, select the product and version, so that the test execution in half an hour to one hours can be executed, is the positive solution.

Ex8:dll is covered by the problem. The system first installed the product's codec plug-in, and then installed other players (Storm audio, Chin listened to the problem has occurred), the same name codec plug-in is overwritten, decoding failed.

This kind of problem, the troubleshooting process may be more tangled, but after the clear, whether to write a test case, each time will be included in the implementation: "First install our products, and then install the Storm audio and video, to encode and decode, see if normal." Expected Result: visual audio is normal. "Is this use case executable?"

Look at the solution to the problem:

8.1, if the solution is a sales evasion-the server is installed independently, the installed software is a standard version, does not allow the installation of other software, then this problem does not need to be resolved at all. Just uninstall the non-permitted software and reinstall it once.

8.2 If the solution is to unify the path of the DLL by the system directory, modify the problem to the specified directory to circumvent the DLL being overwritten. Then this case will need to execute a period of time, and to explicitly check, after Setup, check the xx path xx file, whether to update the success of this check.

8.3 If the solution is not uniformly specified, each software team is their own designated directory, and the characteristics of the DLL is different, there are multiple versions at the same time, then there will be a number of their own software calls DLL conflict phenomenon, or, to say the word, some people even DLL search path " The current directory->system directory->windows directory-environment variable directory specified by path is not considered. Then this thing if you want to expose, you need to find a few people set up a special test, or even periodic inspection-but once the bad to this situation, is the software products do not have uniform rules, we closed the door to their own ideas set, and simply think that customers will only install a product. If it is me, certainly strike--the system department sits down to make a rule, everybody changes together, can once and for all, minute matter. The result is to dump the problem to the back-end team, find a few people maki time-consuming, long-term to toss. This is not to take others as a person to look at, also did not consider the overall cost of the project, or simply did not fulfill the responsibility, why let the tester to back the pot?

a phenomenon that looks the same, because of the different causes, the actions that may be taken are very different. If you are not in the project group, do not know the reasons inside. Just to urge a certain person, this is not your problem? This is easy to stimulate the rebellious mood, the overall promotion of products, will produce very large negative energy.

Vi. the enforceability of test cases.

An example has been given above.            It is irresponsible to write test cases of poor rate test at random.

A: I wrote to traverse all the interfaces, all the formats, clearly, how did you not measure it?

B: Have you counted how much time you actually want to test in this line? You write a sentence, I want to toss a week.

Out of the question, you say you think, is the executive lazy, but such a tense test time, it is impossible to give a week to test such a basic function. Test prioritization, Test equivalence class partitioning, and even the risk-based test decisions of the customer using probabilities, is not the test design thing to do?

Form a chart to illustrate the point.

650) this.width=650; "title=" Untitled. png "alt=" Wkiom1c5m5nwrrq-aafvnzjxdlq096.png "src=" http://s4.51cto.com/wyfs02/M01 /80/23/wkiom1c5m5nwrrq-aafvnzjxdlq096.png "/>

The purpose of this watch is not to memorize, but when you think "the problem arises, do we have to write a case, and then go on to execute it?" " at the time, according to the actual characteristics of their products, to make the right analysis and judgment can be."

So the question of a return to the beginning, if it is in accordance with the cold rules of the Convention, all the problems must be included in the management of defects. In the face of complex and changeable realities, the style of landing may be varied (no need, choice of execution, long-term coverage, short-term coverage, special focus solution).

The various inspection methods of the third party department may not apply to one of the existing use cases, and the instinct of avoiding disadvantages, explaining the clear reason and solution to the people who do not know the business is very troublesome. So often the actual product verification side, rather than trying to explain clearly 1234, rather than simply do the superficial, write a few seemingly easy to read test cases, we will be more comfortable.

According to the concept of driving force 3.0, carrot sticks will kill other people's enthusiasm and creativity, so through the intelligence positioning and write enough cow B test cases such a high behavior, through standardization, inspection, often will be turned to write hydrological coping behavior-This is not the focus of this article, it is slightly donuts.

Regression test case update, optimization itself.

In addition to the defects extracted from the test case to reverse the addition, the benchmark library of test cases, also need to periodically review changes to update.

This is the pesticide paradox in the test case (pesticide paradox)-the more testing the software, the more immune the software is to testing software testers. Or literally, if the field is a long-term only a pesticide, then the worm (bug) will be resistant to the effect, resulting in more and more poor, and finally kill the worm. Or a different dimension to describe: The test case is a data quantification, you want to assess what, in the end, will inevitably get this quantitative results, but the actual increase in things may not help much.

Some test cases may have been designed to address some of the weaknesses of the product at the time. But several project tests were completed, and even a few years later, the effectiveness of these test cases tended to be 0.

1, may be code logic repair, such problems will no longer appear;

2, may be hardware and software changes, the original test plan needs to be adjusted;

3, may be the function point priority changes caused by the test case priority adjustment and so on.

For example, once in a test case, after the version is requested to be placed on the publisher, it needs to be re-downloaded, and then an installation test is made to confirm the version number information for each module. This is an inevitable test step under the prevailing conditions. Reason one, there have been different decompression software and the breakpoint continued download tool, resulting in the size of the file bytes inconsistent problem; second, the original version is a folder, which has a variety of INI, exe, bin, setup files, it is easy to test version and release version of the phenomenon of inconsistency Therefore, it is very necessary to check the version after re-installation.

But after a few years, the decompression software more and more, compatibility is getting better. Overwrite the decompression software is more and more impossible, it is better to specify the decompression software and version number more realistic; The publish file itself is not a folder, but one or a few zip packages, but also to avoid manual copying and pasting of multiple files, the risk of human confusion; third, the data check also do much better than before, there is no need to use the method of

Therefore, this use case, no doubt can be deleted, after all, download, replace, see version number, how to say also two or three hours to make the set.

Years of constant test cases, from the point of view of cost-effective work, this is invalid work content. It was as if the waiter standing at the staircase was standing there only because of the tradition, and did not know that it was just to remind everyone that the stairs were painted dry.

In terms of testing responsibilities and risks, this is the role of shirking managers. No matter what, the perennial test cases mean that there is no improvement in the Test team for many years, which is beyond doubt.

This article is from the "Willow" blog, please be sure to keep this source http://eilfei2000.blog.51cto.com/2956473/1773817

On the effective renewal of test cases and the pesticide paradox