Questions and answers about software testing (conversations with xianshen)

 

　 WorkAs a member of many programmers, I am full of problems in software development. Today is about testing. As a materialism, I believe that all things have their own gods, so I found the god of testing.

　　Q: Why do we need a test?
The great God looked at me with pity and said: My poor child, the reason why we need to test, is God's fault. God created you, but because there is no test, so you are all imperfect and irrational. You will be influenced by emotions and circumstances, and you will make mistakes.
When the great God said this, he was full of pity. Of course, he was full of pity for everything. I think I should call him the God of mercy.
I said, oh, that's not perfect because he also needs to test it?
The God of mercy is indeed a fairy: Where have you eaten?

　　I asked: we need to test, but why do I think testers only need to press the keyboard every day?
The God of mercy said: You are typing on the keyboard to write code. The tester is typing on the keyboard to obtain information about the current system. The real work of both is mental activity, it is the mental activity that comes with these physical actions before hitting the keyboard. If you don't think about it when you press the keyboard, you are not developing and testing it. Moreover, if you are thinking but not typing the keyboard, you may still be developing and testing it.
I said: the Key is thinking.
Said the God of mercy: Yes. He apparently hesitated for the last half of his sentence, but he said, As soon as humans think, we will laugh.
I think it's not funny at all. Even North Korea is better than Brazil. I said, in fact, for software developers, we should also use our own products to test it.
The God of mercy nodded and said, "Eat your own dog food." If you are worried about and despise your product, Why should someone else buy it?
I said, however, unless we develop development tools, we cannot use them at all. People on earth know that people who do diet pills never drink diet pills, and those who buy ham sausage never eat ham sausage... everyone needs to be a chemist.
The pity said: A test sample composed of developers is unlikely to represent the entire user group.
I said to myself, so we need testers to think about it from another angle.

　　I asked: Despite testing, why are there still defects in the system? Are they not testing all possibilities?
The God of mercy turned my white eyes and sighed. As revenge, I will refer to him as the god of white eyes in the following description.
The white-eyed mind says that for any program, the testing data may be infinite. The test may be convincing to indicate that there is a defect, but it never shows that there is no defect.
I thought about it and said that our current system needs to import the customer's legacy data, and the light report can be as much as 0.2 million. If one piece of validation report content and style is displayed, it will surely die.
How did you test it?
I said, sample extraction test. I said, I understand that because we cannot test all possibilities, any test is actually a sample test to some extent, these samples represent a part or segment of the entire possibly tested set in some way.
The white-eyed god nodded and said, the test is just a sample!
I said that, just as we went to the hospital for a physical examination, all the tickets said, this result is only responsible for this sample.
In fact, sampling is also a psychological process and a perceptual process. Samples satisfying someone may make others feel unsatisfied at all.
I said that since it is impossible to perform a exhaustive test, we tend to be stuck between two goals: we hope that the test will cover all the conditions of interest and reduce the test set to manageable and tolerable levels. It's like eating a buffet. I want to eat everything, but I don't want to break my belly.
We need to select the most representative samples for testing as much as possible, which is the advantage of testers. In addition, problems detected by diversified samples may exceed those detected by large samples. Similarly, the test team may also find more problems.
I said, yes, blood is also drawn. Some people will require venous blood not to eat in the morning, and some people will need to take blood several times at intervals.

　　I asked: in development, we adopt TDD, and our unit function test coverage rate is also very good. In this way, I think we can reduce the number of testers? Reduce system testing?
God said, will you take a maiden flight because an airplane guarantees that all its components are tested before assembly?
I said, oh, Luo's launch failed.
However, good unit function tests can remove noise from system tests.
I said that none of the defects were self-built, but they were put forward by developers. Unit function testing can block the entry of defects in the development process to a certain extent, while blocking some simple defects. system testing cannot capture all defects. Compared with unit function testing, system Testing takes a longer time and cost, which can be partially saved through unit function testing.
Shen said that the developer's testing ensures that he understands and implements the same requirements. But does the developer's understanding of the requirements be consistent with the real requirements?
I said, so testers need immediate acceptance.
Shen said, In addition, from the perspective of TDD, it is difficult for a person to discover his or her own mistakes from a psychological point of view. Therefore, the practice of TDD and Pair programming will have a better effect.

　　I asked: why is the last integration test phase of many projects so long? This will always lead to our project extension, and almost all projects will be postponed.
What are you doing in the integration test phase?
I said, oh, let's pull out a release branch, freeze code submission, start testing, find out the cause of the defect, modify the defect according to the importance, and then retest it in this loop. Well, the problem is that there will always be a lot of defects at this time, and every time we fix some defects, we will always introduce new ones, so this takes a long time.
Shen said that fixing a defect introduces a new defect, which is called the fault feedback rate (FFR), which is the percentage of the situation where one repair causes another defect in the system. This value would be happy if it was less than 50%. In addition, stress and attempts to increase the speed of defect repair will increase FFR.
I said that it is normal to introduce new defects while fixing the defects. Well, my question is, why does the last test take so long?
God said, don't you think your manager misestimates the time that should be spent in the final integration test phase?
I said, oh!
In addition, the test is not equal to debugging.
I thought about it and said, Yes, it is about the final integration test phase, but it actually contains two actions: Test and debugging, and debugging takes more time than testing. Well, our current project features, because performance issues are not taken into account during development, lead to a lot of time for tuning before going online.
Shen said that the main problem with project extension is the delay in testing.
I said, yes. I am very impressed with this. If you perform big data testing during the development phase, it will save a lot of time. The later a defect is discovered, the higher the repair cost.
Shen said that testing should be performed first, and testing often started before the project starts development. testing should run throughout the development process and be carried out frequently, instead of being pushed to the final integration test phase. TDD and continuous integration are good practices, but they are only part of the test.

　　I said, the test ensures the quality.
Said Shen, isn't Sanlu tested?
I said ,....
The test only provides information. The definition, importance, and response of such information depend on people, and decisions made by people are emotional and interest-driven.
God said, if the products developed are inferior in quality, do you think it is a waste of time for testing? How can we test poor code? The more defects you discover over a period of time, it does not mean that the fewer the remaining defects, but it always means more defects.
I said, I understand that testing is just about collecting information, and nothing else can be done. As we go to the health check, the Health Check report can only reflect our physical status at that time. When Yu Jian is unhealthy, it depends on our daily habits. This is two separate tasks, health check does not guarantee my health.
Shen said that many large companies place great importance on the testing department because they cannot clearly understand their own development processes, so they can only obtain information from the tests, this information is only a part of software development. I have seen a project manager asking the tester if the project can be delivered. This is fundamentally shirking responsibility. Information and decisions are two different. What's more, the information collected by the test is only part of the information, in this case, it is better to let the tester become a manager.
I said, no wonder it's not normal for the lab doctor to check his health every time. The lab doctor is always impatient to ask the doctor.

　　Q: What test is a good test?
What do you think?
I said that the purpose of the test is to discover the defects, so the more defects found, the better the test.
What then?
I said, I will tell the tester, aha. The more defects you find, the more bonuses I will give you.
God said, it is an excellent example of poor management.
I said there is indeed a problem, so that for testers, a poor system has the highest value for them, and they will fall in love with bad developers and lead to office romance. This conflicts with the goal of the entire team-to deliver high-quality systems, and will cause conflicts and disputes between the Development Department and the test department. In addition, the detected defects increase, but the quality of information obtained is reduced.
God said, You obfuscated the purpose of the test. The purpose of the test is not to discover defects, but to obtain the information we need. In comparison, if all your items are normal, do you think the health check is of no value?
I said, Well, I will be very happy.
Shen said that testing is only a sampling, so a good test needs to be able to cover the most important part of the system according to the context, and cannot expand to a manageable level.
I said, remember, we often have different inspection packages during the health check, for example, for white-collar workers, we will focus on examination of the stomach and cervical spine; for people over 40 years old, increased cancer factor screening. That is to say, testing is not an isolated behavior. It needs to adopt different strategies based on different situations. In addition, we always hope that we don't have to spend too much money during the health check.
God said that "good" is not a property of a test. It can only be a property of the relationship between testing, implementation, cost, and application scenarios.
　　I said, unable to measure the test?
As a comparison, the use of the system after it is launched can also be seen as a continuation of the test.
I said, what you mean is that we can track the defects in the system's use, and then conduct statistics and analysis. For example, we can test which Typical defects are missing, in fact, there is not much need for a test, because the user does not use this function at all. In this way, we can improve the test in the future.

　　Q: So, do we need to track all defects?
God said, you have not tracked all defects?
I said that some defects are too small, such as spelling errors, and some defects can be easily fixed, so the tester told us directly without filling in the defect records.
What do you think?
I said, I think it is not bad. No developer is willing to make errors in their own programs. We often regard the programs as their own extensions, pointing out that the program has errors is pointing out that I have errors, therefore, for some very small errors caused by negligence, the tester directly told me not to record them, making me feel better.
God said that defects become errors only after the system is delivered.
I said, okay, but reporting my program defects will make me criticized.
Shen said that was another question about team management. If a person spends more time accusing others, the less likely the problem is to be solved.
I said, however, we should oppose the document. The document hinders communication, but it does not have much value.
God said that documents are of no value when they are not used. Some people do not oppose the document, but write the document by themselves. Information can be tracked and counted only after being documented, especially the defect information that is part of the project information. It reflects the project status and cannot be ignored, if defects are not recorded for the purpose of "saving time" and "face", the result is the loss of the project status, which will affect the subsequent judgment of the project, it will waste more time and money in the future.
I said, I think of it. Every time I get the results of my health check, I will see the comparison between the indicators of this health check and the previous results. This result reminds me that the cholesterol has increased, and the cervical spine is much better, so we should eat less fat and exercise more, while Xiao qing taught me to continue to exercise cervical spondylosis.

　　I asked: It seems that information collection is only the first step. What is the process of information processing?
God said, how did you do it?
I said, first, the tester will choose a sample to test the system. For example, if the system cannot upload images, the tester reports a defect and records the detailed information.
God said, this is the first step. Information is consumed. At this time, the information is meaningless before it is determined.
I said, then we will see this defect report and communicate with the tester. The tester expressed concern because uploading images is a very important feature for the customer. Does it mean that there are many problems in other parts of the system.
Shen said that the tester gave meaning to the information.
I said that we reproduced this defect with our testers. The system reported that the permission authentication failed. We think this is a permission issue, and the other functions of uploading images should not be affected, maybe someone has modified the permissions of the current test user, or maybe the last commit has damaged something. In short, the impact will not be great, because there are many unit function tests, if it is a commit failure, it should also be code on the page. This section of code is not much, and the error can be quickly located and repaired.
God said, you are giving meaning to the Information. Different people will give different meanings to the information.
Next, we will discuss the priority of this defect. Testers think that this defect is very important and should be repaired immediately; we think that although it is important, it is very easy to repair and another very important defect needs to be repaired at this time; the project manager asked us about this defect and said that we now have a particularly important defect about image search that needs to be repaired. I think this defect is the most important, because the customer's boss needs to be presented the day after tomorrow, the function of uploading images is also very important, but it is not within the scope of the demonstration, but since it is easy to locate, we can also quickly find out the cause of the problem, but do not fix it.
God said that the importance of information is being determined and the response is made. Different people determine different importance for the information.
I said that the complete process should be: uptake information-> determining meaning-> determining importance-> responding.
God said, yes.
I said, wait. I saw the problem. Because different people determine different meanings and different people determine different importance, they must have their own understandings and trade-offs, and make a final decision based on your own importance and other information. However, this is not the case in reality. A lot of information has been given the meaning and importance defined by the information provider before it is provided to the manager, because managers cannot obtain other information or simply do not think about it, they directly decide based on the meaning and importance defined by the information provider.
What did you think?
I said, loyal minister, traitor and faint. In fact, there are no loyal ministers and traitors, and only the fainting king. It seems that managers are high and powerful, but in fact their rights have long been divided by information providers.
God said, you are far away.

　I asked: Well, the next question is how to avoid testing difficulties?
God said, do you think it is a big system that is easy to test or a small system that is easy to test?
I said, of course, a small system is easy to test.
God said, so let the system be as small as possible.
I said that a common game now requires several GB.
God said, first, let the demand be controlled. This is a very important management task. If this task is not completed well, it is a management failure.
I said, this seems very difficult, because the customer is always asking us to add features, and some features are indeed very important.
Shen said that if a feature is required later in the project, it is actually telling us that there is a problem with the demand analysis and there is a demand omission. However, if a feature cannot be used for a variety of reasons, it is also a problem with the demand analysis, and the demand is fraudulent, which is different from the actual situation.
God said that to control the growth of demand, it is necessary for decision makers, demand analysts, and customers to distinguish between a specific thing and value-driven customers.
I said, what about second?
God said, second, do not try to handle all possible situations in the software.
I said, I have the impression that we need to import legacy reports to the new system. As the legacy data has exceeded 10 years, there are a small part of nonstandard web pages, if the program contains processing all the reports, it is quite difficult. Finally, we chose to manually process the data.
God said, third, code quality. The best implementation is always the most concise code, and the complexity of the Code should be minimized. Two programs of the same physical size may have a significant difference in internal complexity, and this will eventually be the main factor determining the difficulty of testing.
I said that we can use incremental build to control the complexity of code without doing anything at once.
In addition, attention should be paid to the separation of components, especially for projects with multi-team collaboration.
I said, this is the case. For large projects, it is often divided into modules for group development. Efforts should be made to minimize dependencies and unnecessary communication between groups and modules, therefore, code can be redundant. Each development team must be independent and have its own analysts, developers, and testers.
When it comes to incremental building, I have a new problem.

　　Q: Now we use iterative development. Can the demonstration of the customer after each iteration be regarded as a user acceptance test?
The demo is not a test.
I said that the customer touched the real system, intuitively understood the functions and usage of the system, obtained system information, and then gave feedback. This can be seen as part of the user acceptance test.
Shen said that for customers, if they do not actually use the system, they will not be tested. In addition, are you not prepared for the information the customer obtains?
I said, well, we will do a lot of preparation work before the demonstration.
God said that it is actually preparing information that the customer can obtain.
I said, should iterative development be continuous deployment?
Shen said: continuous deployment and continuous incremental use.

Converted from watercress: http://book.douban.com/review/4594239/

22:49:42 from: Haizi
Perfect software-comments on various fantasies about software testing