Three realm of white box testing of Communication Software

Three realm of white box testing of Communication Software
Source: eztester

Communication Software is widely regarded as the most difficult field for white box testing. On the one hand, communication software uses C language as the main language, and advanced white box testing technology has not been effectively penetrated into this area. On the other hand, communication Software is usually an embedded real-time system. Building a testing environment is very complicated. In addition, communication software is usually bulky and complex in structure. It is not easy to perform unit testing or integration testing of communication software. I have had the honor to work in the communication field for many years. In recent years, I have been deeply touched by the relationship between consulting and many enterprises in China. Domestic enterprises generally do not feel or pay much attention to white box testing. A few companies that pay more attention to quality have tried hard, but there is no way to hit the wall everywhere, or a lack of suitable testing tools, or the white-box test cannot be performed due to poor management. Of course, there are not a few companies that cannot find their way. Many companies have been in the wrong direction since the very beginning, and they have been wandering around for a long time without knowing it. Domestic communication enterprises are doing well and poorly in unit testing and integration testing, and there is a big difference. We divide three realms: Chaos, order, and spontaneous. These three realms refer to three stages of development. Of course, the significance of this classification is not to distinguish between high and low levels, but to try to point out the development direction of white box testing. In addition, to summarize the historical experience, communication software is complex due to its complexity, white-box testing cannot be achieved overnight. In some specific stages, we must experience it first. While dividing three development realms, we also try to explain how to implement white-box testing in various realms? Where is the focus? What problems should we avoid? One realm: chaotic stateThe chaotic state is the development phase of white-box testing. At this stage, an enterprise only has sporadic unit testing or integration testing practices and lacks successful cases, the most typical features of this phase are: Everyone is busy and busy all day. Everyone is busy in the enterprise. The testers who should have tested in the laboratory were rushed to the market. They jumped out of the customer's machine room and bowed their heads to the test center. They looked up to conceal problems with the customer. The development backbone that was supposed to stay at home encoding was also caught at the scene from time to time, and the debugging environment was shelved to solve the problem. It usually took several days to locate some difficult problems, in many cases, if the device cannot be located on the site, the device will be secretly reset, and the problem will be solved. Then, the device will quickly slide back to the company and read the code to continue positioning. The biggest characteristic of chaos is that everyone is busy with fire fighting, and the investment in system testing is not guaranteed, let alone the investment in code-level testing. At this stage, many "fire-fighting heroes" will be created, and the responsibilities of "fire-fighters" will be hard to be held accountable. According to the communication industry's rules, 60% of bugs of a product should be exposed in the white box stage, when the company is still filled with many "fire-fighting heroes", the white box stage usually finds less than 20% bugs, or even some companies are zero. The chaotic state also has an important feature. company members generally lack the concept of white box testing and generally know How to Do code review, unit testing, and integration testing. However, it involves specific scenarios, when performing unit tests on a module or cross-subsystem integration tests, we usually have no idea where to start. In a chaotic organization, it may also be popular that unstable products are the responsibility of testers. Because many people think that, even though unit and integration tests are not done or are less done, all problems can always be found as long as the system tests are done well. In fact, this viewpoint is already a thousand miles away. It is puzzling that there are not many people holding this viewpoint. Why? Just like the special phenomenon at this stage, it is clear that a developer has a poor level and the code written is always faulty, but he has saved several fires in the market, the leaders did not hesitate to regard him as a "hero" because of his conscientious attitude, selfless investment, and how much loss he had recovered from the company ". Realm 2: ordered stateAn enterprise has carried out multiple unit tests or integration tests. After successful tests, the enterprise enters the orderly state. In this phase, although the white box activities of some projects are not successful, most projects are somewhat effective, and some projects have remarkable results. At this time, a specific organization in the enterprise is responsible for promoting white box testing, and the white box testing process is gradually integrated into the R & D process. A typical example is to include problems found in white box testing into statistics, during the R & D process, the defect density (How many bugs are detected in thousands of lines of code) is used as an indicator to determine whether the white-box test is sufficient. In addition, the coverage rate indicator is used as a basis to check whether the individual is fully tested. The white box test is included in the process monitoring. It mainly controls whether a project performs white box testing, whether the implementation process is standardized, and whether the implementation results meet expectations. QA has the right to request rework for behaviors that do not meet the process requirements. Two conditions must be met to enter the ordered state: First, the white box test was successful in a few projects and became a copyable activity; second, the white box test was organized and the process was guaranteed.. If these two conditions cannot be met at the same time, it indicates that unit test or integration test still lacks assurance in this enterprise. Even if some people succeed occasionally, it is unreliable, individual behaviors cannot be converted into organizational behaviors. The orderly state is a necessary stage for white box testing of communication enterprises. Most of them are long, fast for one or two years, slow for more than ten years, and long technical accumulation is required, and optimization of organization and process. In addition, switching from an orderly state to a spontaneous state involves a significant transformation of the white-box testing concept. From the perspective of actual operations, it is very difficult for a project cycle to be crossed. During the development of an orderly state, most enterprises may encounter the following problems or phenomena: 1. I started to realize that unit testing should be done by developers. Most companies that are still in the chaotic state will think that unit testing should be done by testers, developers may feel that coding and testing by themselves will fall into inertial thinking, and the test results will not be good. However, when the tester operates several times, several insurmountable problems may occur. First, the tester is not familiar with the code and reads the source code and finds a way to perform the test, you need to know that unit testing faces numerous trivial functions and a developer can write a bunch of new functions in a single day. Therefore, if the tester completes the unit testing, it usually takes 10 times more effort than self-testing by developers, Which is fatal and unit testing is bound to be difficult to sustain. Second, when testers Perform unit tests, they often cannot determine whether a certain phenomenon is a problem. They have to find the corresponding developers to locate the problem and correct the problem, testers do not have the product code permission, which wastes a lot of time on repeated communication. After several hits, most enterprises return to this operation mode: Each developer writes their own code and performs unit testing (mainly module-level white box testing ). This is a mainstream mode of operation. It is not a mainstream mode. It may also allow two developers who are familiar with each other's code to perform unit tests. This is also an effective method. 2. we will find that it is not enough to introduce the industry tool to perform coverage testing only by using coverage evaluation tests. When an enterprise's white box testing is implemented to a certain extent, it will be confused: is the coverage rate assessment sufficient? Coverage for coverage, the goal is too easy to achieve. When one or two high-level business calls are run, the coverage rate quickly goes up, that is, if someone wants to cheat, he can write only a few use cases to meet the coverage requirements. Some people think further about this problem and have another confusion: What is the white box test? Test code? Code coverage intuitively shows whether the visible code is running, but if the specification is required and you forget to implement it, what is the coverage rate? 3. Different employees perform white box tests, which have a huge difference in results. This phenomenon is common to every company. It is particularly evident in the early stages of white box testing. What is strong is that there are few missed tests, and there are few missed problems in the subsequent stages, which are poor in ability. Although he is trying very hard to do everything well, there are still a lot of missed tests. 4. there will be white-box testing useless theories that generate white-box testing useless theories. Most of them are not against white-box testing in theory, but fail in practice. Many unit tests have been conducted, with poor results and problems discovered on the surface, deep logic problems or interface problems cannot be found, so it is determined that white-box testing is not very useful. It is also common to find that the white box test is ineffective and you will think that you have not mastered the test method. Therefore, you have tried every means to find the fatal weapon of "seeing blood and sealing your throat". After several efforts, you still have no special medicine to buy, this situation continues to develop, and white box testing is likely to become useless. The essence of ineffective white box testing is that it is difficult to break through the blind zone of "mechanical testing ". Mechanical TestingThe test is based on the available Code. For example, the tested code has a "1 + 1 = 2" statement, so we design the use case, the results also verify that "1 + 1" must be equal to 2, and the total number of test cases and coverage rate are all up to standard, that is, the number of problems cannot be found. The magic weapon to break through the blind zone of "mechanical testing" is "Designing use cases by specifications", but such a simple rule is easy to say and difficult to do, if you have a strong ability, you will not feel that you have to follow this rule. If you have a weak ability, you will not always be able to think about it. When you think about it, you will often have nowhere to start. Therefore, in many cases, it is difficult for individuals to take effective white-box tests to escalate to organizational behavior. 5. whether the white box test can be successful depends largely on the white box manager or the white box qa. This is often seen in the early stages of the white box test implementation. Managers with strong execution or QA can successfully implement the white box test, if the execution is weaker, it will fail. Many enterprises have failed to try unit tests several times, so they have abandoned the white box and only used black box tests. Some enterprises stick to it and succeed in one or two project teams, and then optimize the Organization accordingly, such as setting up a dedicated Promotion Group and building a white-box testing expert resource pool, provides test guides for various projects, further optimizes the process, and includes white-box test monitoring points in the process for control. When an enterprise's organizational structure and processes are gradually improved, white box testing is less dependent on individual normal people. 6. the white-box test is implemented in a phased manner. The test case cannot maintain the code of the centralized unit for a period of time. After the encoding is complete, the unit test is complete and Integration starts. At this time, the integrated test is conducted again, this is a white-box testing model for most enterprises. In this mode, unit testing or integration testing is only implemented within a specific period of time (for example, within one or two weeks of a version cycle, however, product code modification is always ongoing, and there is no doubt that there will be a profound problem: case maintenance and product code maintenance are not synchronized! Therefore, we often see that the first version of a product can fully implement the unit test, and then change the code to solve the problem from time to time, or change the code for the append function, unit Testing is difficult to continue. It is often the case that unit testing is performed only once in V1, and later versions such as V2 and V3 cannot be performed. Uninterrupted code modifications and phased white-box testing are obviously not in coordination. This problem does not seem to be serious on the surface, but is far from being simply resolved. This problem can be raised to the operational mode, to solve this problem, we must introduce a test mode from time to time. Naturally, you can immediately guess what mode to use? Continuous integration! Some people may think that continuous integration does not have to be introduced to keep white box testing in the version cycle? Imagine how much coding time does a developer spend independently without interference from coding to version maintenance? How long does the integration take after the integration starts? Undoubtedly, the latter occupies most of the time. If there is a test follow-up for each version modification after the integration starts, do not write test cases every day. What is continuous integration? If you do not want to supplement the use case every day, every week, every month, or every other version, how can you overwrite all the changed code with the updated Use Case? Code in the communication field is generally very large and complex. A version cycle is usually 30 ~ For 40 weeks, from the start code to the first two-to-one version, usually 2 ~ Three weeks later, we will enter the stage of continuous integration testing, which will last for a long time, usually more than 10 weeks. Therefore, if we stick to the stage test, it will inevitably lead to the failure of white box testing, the Use Cases Written by each person are difficult to maintain. They are written once and then discarded. For clarity, we call a periodic test One testIn contrast to the continuous integration mode, the test is called Continuous testing. Although many people do not think that only one test case is used for a phase test, but the actual situation is not good enough. When there are many modifications to the source code, it is very painful to concentrate on regular maintenance sessions for a period of time. 7. at the end of each R & D stage, it is a natural continuation of the test thinking to combine the use cases of each person and combine the test cases. The starting point of this attempt is to make the test set better maintained, but this does not happen, the more you merge the use cases, the more you cannot maintain them. Unless the tested system is very simple and rarely changed. Of course, if the system is rarely changed, the need to maintain the use cases is also insufficient. There is still a clear maintenance owner for the scripts before merging, but who should maintain the scripts after merging? In addition, based on experience, a full unit test, the scale of the test script should be equivalent to the tested code, and the running environment of each user's case writing is very different. It is not easy to combine such a huge system, once combined, it cannot be maintained. Therefore, merging use cases is a task performed by an enterprise at a specific stage in an orderly state. It is usually done only several times and will not be repeated if the results are not met. As a matter of fact, in an Organization that follows the test mode, it is difficult for everyone to manage themselves. 8. Some employees still try to perform unit tests on the board. Should they perform unit tests on embedded products? This problem can be further attributed to the fact that it is not feasible in practice. Theoretically, no one requires a unit test to be able to mount the board. However, in practice, most unit tests on boards of communication products fail. I have seen more than a dozen failures, especially for large companies, there are always some confident employees in each product group who are willing to believe that the single board ut is the right way. Of course, this article does not mean that you cannot succeed by running UT on a single board. It means that the probability of success is relatively low. Moreover, the definition of success varies from person to person. For example, if a developer writes code for one day and then tests the newly written code in one day, the white-box test can be maintained, however, when there are various restrictions (for example, it takes a long time to download a single board, and unit testing faces the initial code, the program will run in a few minutes without testing ), it takes 10 days to test the code written for one day, which is difficult to maintain. The critical factor for unit testing on the board is the test efficiency. Imagine that the mainstream communication software uses C language and wants to do a good job of white box testing for pure software C language engineering, compared with Java, C ++, and other projects, it is difficult to implement ut in a real-time operating system. Therefore, according to the inference of practice, the unit test of communication software should be carried out in the simulation environment in the form of pure software, but the integration test can be carried on the board and carried out in the real environment. Everything has three development directions: Forward, stay in place, and backward. The above describes the behavioral characteristics of the white box test in an orderly realm, and there are also various ways to improve the behavior between the lines, with targeted improvements, white box testing continues to advance, and vice versa, returning to chaos. For example, at the initial stage of implementation, the success of the white box test was largely affected by the execution of the project team. To improve the white box test, we need to optimize the organizational structure and process methods, successful white box practices in some projects cannot be copied to other projects. The ordered state is between the chaotic state and the spontaneous state. Because of its long time span, we can also subscribe to the initial and later stages of an ordered state. In the initial stage of an ordered state, there is no standardization, white Box tests often shake between success and failure. in the later stage of order, they should have some characteristics of spontaneous state. Specifically, the development of an orderly state to a spontaneous state involves two major challenges. First, to solve the problem of a test, we must transition to a continuous test. Second, to solve the problem of "mechanical testing" at the organizational operation level, we mentioned that the testing effects of different employees vary greatly, and the difference is normal (otherwise, excellent employees cannot be called excellent ), but can we narrow down the impact of capability differences on results to system tests? Generally, employees with strong white-box testing ability have strong coding ability and poor coding ability. It is rare to hear that the testing capability is strong but the coding capability is very poor. Therefore, to make the white box test more in-depth and effective, the key point is to solve the problem of the Way of Thinking. Training and daily exercise can also play a certain effect, but it is not obvious at all, after all, it is rare for all the Members in the project team to have an outstanding performance. Based on practical experience, the most effective way to solve this problem is to test first. If you write a test case before coding, you can only design a test based on the specification, this is the same for both powerful and weak abilities. The way of thinking is forcibly changed, and its test is undoubtedly the most thorough and effective. From the preliminary stage to the later stage, there are also great changes in the test code. In the early stage, development is the development, testing is the test, and test operations are attached together with the test code, (Optional) in the later stage of order, you will also regard the test code as a product code, which is necessary and naturally incorporated into product maintenance. Realm 3: spontaneousThe spontaneous state is the Communist realm of white box testing. At this time, the productivity is highly developed, the efficiency of design cases is improved, and testing is no longer a heavy burden. The so-called spontaneous, just as in communist society, labor is a kind of personal will rather than a means of survival, and the self-testing of developers also increases to personal will, even if the leadership is not forced, the process is not limited to white box testing, and developers still do the test voluntarily. The spontaneous state is the highest level of white box testing. Its typical characteristic is that white box testing has become an employee's Universal BehaviorAnd Spontaneous behavior. The white-box test enters the spontaneous state and must undergo two major changes. First, the test efficiency must be improved several times. This is the basis, and second, the test practice can be in-depth to effectively detect various problems. Some companies have already experienced these two transformations, the former is mainly dependent on the concepts of online testing, continuous testing, and black box testing (For details, refer to "4th white box testing method overview" to test the small cycle, pull the big cycle of R & D, and convert debugging test ), while the latter effectively discovers problems, the test first (TDD) has accepted a wide range of practical tests. When the above two major changes are completed, the white box test becomes a mandatory action for every employee, just like a debugging operation. Every employee needs to "tune" normally after writing the code ", in the spontaneous realm, "test" is a natural task for every employee. In practice, an organization in the spontaneous realm replaces "debugging" in "testing and testing" to a large extent. At this time, it is common for employees to not open the debugger for several months, because "test and test" is the most economical way for developers to run the program and the most convenient way to identify and check errors, using the debugger is not necessary. In the spontaneous realm, testing and continuous testing have become a ethos. On the other hand, employees from the leadership to the grass-roots level generally have a deep understanding of white box testing and know that white box testing should" What is done and what is not", The company has trained a group of white box testing experts who are very clear about which objects can be tested for white box testing and which are not easy to do. Even at the highest level of white box testing, not all systems are suitable for complete testing, especially those software layers that are heavily dependent on specific hardware environments, when white-box experts identify which modules are not suitable for unit or integration testing, they will consider alternative solutions, such as enhanced code review, enhanced peer review, and additional Simulator Design for specific interfaces. SummaryThis article describes the three realms of white box testing in the Communication Industry: Chaos, order, and spontaneous. These three realms reflect the general development process of communication enterprises in the white box testing field. From the chaotic state upgrade to the ordered state, the core focus is to solve the test efficiency problem. As long as the efficiency is improved, an organization is easy to advance from the chaotic state to the initial stage of the ordered state, then, through the optimization of the organizational structure and process measures, the existing results will be consolidated, and then the problems of Continuous testing and in-depth testing will be solved. After the problems are solved, they will be upgraded to an orderly advanced stage, the key to upgrading is the transformation of the concept of continuous testing. From an orderly state to a spontaneous state, the test efficiency and test quality are comprehensively improved, and the operation mode will change greatly. The test tool is the key. The tool is not only easy to use, but also extremely efficient, in addition, it is convenient for everyone to change from "tune" to "test. The above three levels of enterprise white box testing can be compared with the realm of Confucius 'life, Confucius said: I have five in ten, and I am interested in learning, thirty, forty, not confused, fifty, and I know every day, 60 and ears, 70 from the beginning, not moment. This is a chaotic realm. The word "Zhi Yu Xue" shows what to do at this stage. For white box testing, enterprises in a chaotic state must be brave enough to try; otherwise, they will always stay in place without progress; "Thirty seconds, forty seconds without confusion" corresponds to the orderly realm. At this time, the white box test has found the portal, "Don't confuse" is to strengthen your faith, continue to do, continue to optimize, so, in an orderly state of the enterprise you insist on; "70 from beginning to end, not moments", this is a spontaneous realm, the old husband does not want to do whatever he wants (or, the Sage of Confucius should not be a saint, but a man of God). The key is "don't worry". Only when you know where the rules are, can you start from the beginning, therefore, the white-box test in the spontaneous realm also needs to "do nothing ". Let's summarize: Enterprise white box testing In a chaotic state, you are trying, in an orderly state, and in a spontaneous state.. References: 1. eztester, Wayne Chan, 4th white box testing methods
2. eztester, Wayne Chan, Introduction to the 4th-generation white-box testing method (theoretical)
3. eztester, Wayne Chan, Introduction to the 4th-generation white-box testing method (vctester practices)
4. eztester: how enterprises implement white box testing
5. eztester: Several misunderstandings about white box testing

Vince Testing Technology Research Center http://blog.csdn.net/vincetest