Is TDD beautiful?

A recent article in CoolShell, TDD is not as beautiful as it looks, has aroused high attention and encouragement debate in Agile communities. Today, InfoQ even held a special "virtual forum", "How beautiful is TDD?". Several domestic agile community celebrities have discussed this issue in depth. Regardless of the results, this spirit of discussion and reflection is very commendable. The incident can be simply summarized as "an influential developer challenges TDD, and a group of agile community celebrities explain and defend TDD ". Now, let me firmly stand on the CoolShell side, to contribute to the challenge and criticism of TDD!

 

First, let's take a look at the core concept of TDD. The first is Specification by Example, which uses test cases as a form of Requirement Specification. Traditional expressions of requirements include documents and Use cases, while TDD emphasizes expressing requirements through test cases. In addition, TDD test cases are Black Box Based on external interfaces. Therefore, they are actually designed for external interfaces. "Without regard test cases as tests, but from the perspective of requirements and design" is an important difference between TDD and traditional tests. The second important concept of TDD is Test First, which emphasizes the Driving Effect of testing on implementation. First, write Test cases before implementation and reconstruction. The essence of Test First is "First understanding the requirements, designing external interfaces, converting them into Test cases, and then implementing and restructuring ".

 

If "Use Case is a specification" also makes up for some shortcomings in the expression of requirements in documents and Use cases, it has some benefits. Test First has a big problem., Especially the "Do not write any line of code before the test case fails" is an extreme error.

 

If the test cases are requirements and designs, why cannot we write the test cases first? Isn't this the first requirement we are most familiar with before designing and coding? The answer is: The unexecutable Test cases (Test First) are very different from the ones that can be executed, If you write a test case, you will not see the actual running results.. Testing cases that cannot be executed do not have much better guiding significance than the documents written on paper! Unless in some simple cases, in actual software development, it is difficult for you to write test cases that truly meet the final requirements without executing test cases. For example, when you create a page, the page's performance requirements and design will usually be constantly adjusted after it can be run. before implementation, you can only have a rough outline and direction, the details in many aspects are either unclear or unexpected, and they cannot be achieved overnight. If we emphasize the Driving Effect of testing on implementation, the assumption that "requirements and design details can be clarified before implementation" is hidden. This is extremely agile and unrealistic!

 

Test First requires that you have a precise understanding of the software requirements when writing Test cases. However, the uncertainty of user requirements and external environment in the actual software development process will make the software requirements difficult to grasp and change frequently.

 

User demand uncertainty refers"The requirement cannot be clarified before the user can really see the effect".For example, if you want to develop a large game such as Wow, you can imagine that the effect of the game is that the designers have come up with an idea of precision to every detail at the beginning? For software such as games, the demand and design cannot be separated from the actual operation. Generally, game designers can only Use documents, sketches, Use cases, and other non-precise methods to provide a general requirement. First, they can make a prototype and then gradually refine and clarify the results, the increase and change of demand design will be accompanied by the entire software development process. In addition, there is another extreme case where there is no precise user requirement at all. For example, in automated translation software, can you fix the translation effect with test cases before implementation? Is there an absolutely correct translation method? Recently, when we talked to a large foreign company about a project requirement, the customer said, "At this stage, we cannot put forward very detailed requirements. Only when you come up with the first version, then we gradually adjust the details ". Our customers did not claim to be agile, but they were agile in their ways of thinking. They did not need to clarify their needs as soon as they came up, but they also needed to precisely write automated test cases. Some people say that we can still perform TDD based on our own understanding. This can be done: 1. Clarify the requirements based on test cases and customer communication; 2. Drive implementation. I have different opinions on this. The tested cases that can be executed are quite different from those that cannot be executed. The customer cannot get the real running experience from the test cases, you can imagine that Apple wrote the iPhone test case on a PPT and gave a speech to the user. Can the user give feedback on the iPhone design? Real-time software is required for real user feedback. Isn't it the agile idea of "Working software over document? In addition, since users cannot provide feedback before the actual experience, the demand analysis and design made by developers in the early stages of development are only an exploration, and adjustments or even overwrites can be made at any time, it is not worthwhile to invest in automated testing design before implementation.

 

The uncertainty of the external environment refers"When our system needs to be integrated with external systems, Assumptions about external system behavior cannot be fully determined before the actual integration operation". For example, to build a stock client to connect to the exchange system, because the exchange behavior will directly affect the development of the client, so we can develop high-quality clients only when understanding the exchange behavior. If the test driver is used, various test cases involving exchange behaviors are written, such as the situations where messages are sent, the message format, and interaction, however, whether these test cases are correct requires a big question mark! This is because the protocols provided by many exchanges are not clear enough or many are not clearly defined. On the other hand, even if there is no problem with the protocols, developers may also understand the Protocol incorrectly due to simple mistakes or lack of basic knowledge in the relevant fields. In fact, the most important way to really clarify the exchange behavior and clarify the client needs is to run the integration test in the test environment provided by the exchange. For Test First, the errors of Test cases are the most costly. They not only waste time and energy, but also combat the morale of developers. Who is willing to go back and forth? However, unfortunately, the Test case from Test First may be overturned after real integration at any time when the exchange behavior is not clearly defined at the beginning, and if it is a relatively high-level requirement analysis error, that would be disastrous for the overall architecture design. In actual development, our software needs to be integrated with other systems, however, it is expected that the behavior of the external system is unrealistic and not agile without actual integration.

 

So,Test First needs to have a precise understanding of the requirements and environment of the tested system. However, due to the two major problems of demand uncertainty and external environment uncertainty, Test First is unrealistic in many cases.. Actually, Test
The First and waterfall ideas are in the same line, both of which emphasize that the demand is prior to the implementation, while ignoring the feedback of the software demand generation will be implemented, and will be constantly adjusted, explored, and improved in actual operation. TDD is nothing more than expressing the results of Requirement Analysis with test cases, instead of traditional documents, but from a macro perspective, TDD and waterfall ratio are not truly agile. In addition to the simple situation, there is no need to break away from implementation. Can you implement a Linux system after specifying the requirements? Since you cannot implement a Linux system at all, what is the significance of such a requirement? Therefore, what needs can be raised cannot be separated from your implementation capabilities.The relationship between demand and implementation is not simple, but a relationship of mutual feedback., Which has nothing to do with how the requirement is expressed. Just as the waterfall model cannot make perfect Demand Analysis in the initial stage, TDD cannot make perfect test cases in the initial stage. In addition, the development and maintenance costs of automated test cases are far higher than those of documents. Therefore, in an agile environment, the requirements should be roughly expressed through documents, use cases, and other means in the initial stage of software development, so as to experience the results in actual operation and continuously optimize and explore and clarify the needs and external environment, when the requirement and the external environment reach a relatively stable level, the test cases can be compiled to solidify the requirement.

 

The above discussion focuses on external requirements close to the end user (such as ATDD). I will further explain that there are still problems with TDD at the internal unit test level. First, let's start with the requirement and find out where the unit needs come from? The answer is: the demand comes from design! For example, the demand for tires comes from the design of automobiles, and the demand for low-layer modules comes from the design of high-level modules. At the early stage of development, this internal design was very unstable and had many hypothetical components. It was difficult to explain whether the internal design was reasonable without integration tests. The actual project development usually constantly adjusts the internal design after the integration runs, that is, it affects the requirements of the Unit. Therefore, if it is test-driven, first write unit tests into unit tests based on immature internal design. In fact, the integration testing time is greatly delayed, it is not good for a stable design that can quickly identify high-level requirements. Assume that all the units are complete and then start to run the integration or acceptance test. There are two possibilities: 1. when you see the actual results, you decide to adjust the requirements. 2. it is found that the assumptions at the unit level before integration are not true or are not taken into account. In either case, unit tests that were previously written are faced with the fate of being abandoned or must be modified. In fact, most business-related unit test cases are more unstable than integration or acceptance test cases because they are affected by the needs of all their upper-layer modules and design changes. Because we waste a lot of time on unstable unit tests (writing unit tests based on my experience is more time-consuming than writing implementations ), this leads to the delay in integration to see the actual results, and there is no way to adapt to the needs in an agile manner. That is to say, ironic, The Test First concept is in conflict with the agile concept!

Therefore, I believe that Test First does not conform to the basic assumptions of agile development, but the true concept of agility is that "requirements and design depend on implementation feedback, we need to constantly explore and adjust the results in the actual operation process. It is impossible to write test cases that truly meet the final needs out of the actual operation ". So,What we really should do is to see the actual running effect as soon as possibleWhile the demand and Design of automated testing as a cure is after seeing the effect. Before integration, spending too much effort on testing the driver will only lead to a delay in failing to see the actual running effect (especially writing a large number of unit test cases based on developers' own assumptions ), to see the results, you need to adjust the requirements and discard or remove a large number of test cases. In fact, the more external demand, the greater the impact and cost of changes, the more early the need to be clear. From a macro perspective,TDD's so-called fast feedback actually accelerates internal feedback and delays external feedback.. A large number of test cases that need to be modified or voided are actually a great waste. This is also in conflict with the Lean Thinking of eliminating waste!

 

 

 

The above cost/length_of_feedback_cycle diagram is a common example to illustrate that agile methods have shorter feedback cycles and are less costly to respond to changes than traditional methods. We can clearly see that the request errors found in the acceptance test result in the highest cost. If the acceptance test is postponed a little later, the cost of the error will increase linearly. As we have discussed above, it is impossible for any method to eliminate the adjustment of requirements after acceptance testing, because this is a normal process of demand generation. The only thing we can do is to shorten the feedback period of acceptance testing as much as possible, but unfortunately, a large number of internal tests of TDD will only lead to a delay in acceptance testing, which greatly increases the cost. In actual development, I advocateDo not write unit test cases before the first integration operation test.Automated acceptance test cases depend on the cost of writing and maintenance. if the cost is high, documents and Use
Case to describe the requirement, because these two methods are easier to maintain than automated acceptance tests. Compiling unit tests must be completed after integration so that external feedback can be obtained first.Make sure you do the right thing before doing the right thing.

 

The following passage comes from the InfoQ Article Mock is not a silver bullet for testing: "After using the JMock framework, testing is easier to write, faster to run, and more stable, unexpectedly, however, the product quality is not as robust as we expected with the continuous addition of tests. Although the unit test coverage of Product Code exceeds 80%, however, when conducting a comprehensive test before release, it is often necessary to perform a round of repair and regression tests to find serious functional defects. Why do these problems frequently occur when many tests are compiled? "This is similar to the situation I encountered in practice, but unfortunately the article does not find the real cause of the problem. The real reason is that Mock is not Mock, but TDD unit tests are based on developers' assumptions that even if all the tests pass code coverage rate of 100%, when it comes to integration testing, we find that the assumption is not true at all or many situations have not been taken into account at the unit level. How can this ensure high quality? I have seen many similar cases in TDD practitioners who are very careful about writing many unit test cases and have a high code coverage rate, but they are intentionally or unintentionally First, do the right thing (unit test), and then do the right thing (integration test). This is the inverse.

 

Of course, I am not totally denying TDD.TDD is applicable to some scenarios with relatively fixed requirements, especially those with little relationship with specific business.For example, write a general data structure to implement a general algorithm. TDD's first focus on requirements and external interface design concepts also have great benefits for promoting developers' abstract thinking. In addition, TDD usually has a high code coverage rate. The main point of this article is: in actual projects, due to user demand uncertainty and external environment uncertainty, do not expect to be completely clear before implementation, the requirement is gradually clarified after the actual operation sees the effect. our development process must be agile to adapt to the changes in requirements, and the Test First concept of TDD is exactly in conflict with it. Therefore, for those who do not know TDD, I suggest learning and practicing TDD to achieve its benefits. At the same time, I also remind TDD that it has theoretical defects, this requires special attention in practice.