Microsoft Software Testing Method

Software testing problems and discussions have become increasingly active in China in recent years. However, in general, there are many communication software testing technologies, but there are few methods to test software. The "technology" here refers to specific tactical problems, such as how to use a certain tool to solve a specific test problem, or what testing means a certain type of software has. The "method" here refers to macro strategic issues, or methodologies, from the concepts or concepts of software testing to the software quality control system of enterprises; from the software testing process to the setup of the testing team and the definition of their responsibilities.

As a tester, it is gratifying to be keen on "technology" discussions and exchanges. We can feel the beginning and potential of software testing in China's rapid development. But should I think about the "method" problem with the same enthusiasm as the management decision maker of an enterprise? Especially when a software enterprise's software testing starts from scratch, or when the enterprise has a certain amount of software testing investment, it finds that its effectiveness is not significant, even the introduction of tests has brought about new management chaos. At this time, the methodology is more conducive to discovering the root cause of the problem. Even a tester at the grass-roots level, after accumulating Some technical experience, should also come out from the specific daily work from time to time, review and summarize at a higher level and learn from each other, and try to propose some optimization and improvement measures, which are very meaningful for both professional and career growth.

Microsoft has a lot of experience in software testing. I would like to discuss it with you through my personal experiences and thoughts. Here, we need to note that although Microsoft's method has been proved to be successful many times by Microsoft's practice, it is very effective, but this does not mean that these methods are widely feasible in Chinese software companies. The feasibility of a method is also affected by many other factors, such as the enterprise type (Microsoft is an enterprise that produces platform software and general software products), the enterprise management system, and the enterprise culture. So my goal is to give you some ideas and references.

 

Two typical software testing methods

 

Before introducing Microsoft's software testing methods, I want to first understand what software testing is at the conceptual or conceptual level, with the aim of exporting the Theoretical Roots of Microsoft's testing methods.

Traditionally, software testing methods are divided into two categories. The first type of testing method is to try to verify that the software is "working". The so-called "working" means that the functions of the software are implemented in advance; the second test method tries to prove that the software is "not working ".

The representative of the first method is Dr, a pioneer in the software testing field. bill hetzel, on behalf of the author of the complete guide to software testing, organized the first formal software testing forum in history at the University of North Carolina in June 1972. He first tested the software in 1973 with the following definition:"It is to establish a confidence that the program can run as expected.Establish confidence that a program does what it is supposed to do. "Later in 1983, he revised his definition to:"Evaluate the features or capabilities of a program and system and determine whether it meets the expected results. Software testing is any action for this purpose.Any activities aimed at evaluating an attribute or capability of a program or system. "The" assumptions "and" Expected Results "in his definition are actually the user requirements or functional design we are talking about now. He also defined the quality of the software as "compliant ".

The first type of test can be simply abstracted into a process where the function of the software is run in the Environment specified by the design and the results are compared with the user requirements or design results, if they match, the test passes. If they do not match, the test is considered as a bug. The ultimate goal of this process is to run and pass all the functions of the software in all the environments specified by the design.

In the software industry, the first method is generally regarded as the mainstream and industry standard. The IEEE/ANSI standard of 1990 defines software testing as follows:"It is the process of running a system or building under the established conditions, observing the record results, and evaluating some of its aspects.The process of operating a system or component under specified conditions, observing or recording the results, and making an evaluation of some aspect of the system or component (IEEE/ANSI, 1990 [STD 610.12-1990] "the so-called" established situation "can also be understood as a requirement or design.

However, this method is still challenged by many industry authorities. The Representative is glenford J. Myers (representative of the art of software testing). He believes that testing should not focus on verifying that the software is working, but should first identify that the software is wrong and then discover as many errors as possible. He also demonstrated from the perspective of human psychology that "verification software works" is the purpose of testing, which is not conducive to the tester's discovery of software errors. So in 1979, he proposed his definition of software testing:"Is the process of running the program for the purpose of discovering errors.The process of executing a program or system with the intent of finding errors. "This is the second method of software testing. Simply put, it is to verify that the software is" not working "or is wrong. He even believes that a successful test must be a bug detection test, otherwise there will be no value. This is like a patient (assuming this person is ill) who goes to the hospital for a medical examination, and all the indicators are normal, it indicates that the Health Check has no value for diagnosing the patient's condition and fails. I do not fully agree with this view.

The second type of software testing method is also popular in the industry and is supported by many academic experts. Everyone familiar with Ron Patton on the 10th page of the book "software testing" (The Chinese version is published by the Mechanical Industry Publishing House. It is a classic textbook for beginners in domestic testing, there is a clear and concise definition: "The purpose of a software tester is to find Software defects, as early as possible, and ensure that they are repaired." Some software companies accept this method by taking the number of bugs as an indicator to assess the performance of testers.

 

Comparison of advantages and disadvantages of the two methods

 

Although the overall purpose of software testing is to ensure the quality of software products, it is obvious that the two types of testing methods are completely different in their specific objectives or guiding ideology. This also determines that they have great differences in their ideas, processes, and testing priorities, and they have their own advantages and disadvantages.

The first type of testing method is based on requirements and design. Therefore, it helps to define the scope of testing work, facilitate the deployment of testing focus, and strengthen pertinence. This is especially important for the testing of large software, especially in the case of limited time and human resources. The second type of testing method is not necessarily associated with the requirement and design. If the plan is improperly managed, the testing activities will easily lose the focus and go astray.

The first type of testing method can be combined with the software architecture and software development plan, so that software testing activities can be carried out on a hierarchical basis, in this way, the functions and quality of the software are gradually improved and improved in a planned manner (I will discuss the level of testing in the future ). The second type of testing method does not have the progressive nature of this process.

The disadvantage of the first type of testing method is the lack of flexibility, which is not conducive to the tester's initiative, as Mr. Myers said, it is not easy to find software errors (bugs ). This is the strength of the second type of testing methods.

 

Microsoft's strategy

 

It is precisely because Microsoft recognizes the advantages and disadvantages of the two types of test methods that Microsoft combines the two types of methods in the software test activities, based on the first type of test methods and the main clue, the second type of testing method is used in stages.

 

Microsoft's first-class test

 

In general, Microsoft's first-class tests are divided into three steps: review requirements and design-> design test-> Implementation operation test.

As mentioned above, the first type of test verifies the correctness of the software based on requirements and design. We naturally think that there are also problems with the correctness of requirements and design. Based on incorrect requirements and designs, it is impossible to develop the right software products, and testing will be futile. Therefore, verification requirements and design are the first step for Microsoft to perform the first type of testing. It is necessary to point out that the requirements and design mentioned here generally include: (1) the project manager (the information is from the marketing department, User Support Department, etc) the requirement text (Requirement Specification); (2) the function design text (Functional Design Specification) compiled by the project manager according to the requirement text; (3) the implementation design text (implementation design specification) written by developers based on the functional text ). Microsoft testers should participate in the review of all these texts. As a tester, the review focuses on checking the integrity, rigor, and testability of the text to the user requirement definition. At the same time, such review is also a warm-up activity for testers to bring them into the technical and business status as soon as possible.

Step 2: testers should prepare test plans and design test cases based on approved requirements and designs. Among the three texts mentioned above, functional design texts are the main basis. The reason is simple. This kind of testing focuses on whether the software can correctly implement the functions, rather than how these functions are implemented. We can see from this that this is a typical "black box test ". Indeed, Microsoft's testing mainly involves black box testing from the user's perspective. The completion of this step means the completion of the "test plan" and "Test Case Design" texts. The "Test Plan" text mainly describes the testing scope, fields, methods, tools, resources, and plan schedules. The "Test Case Design" text lists the test cases, the setup, execution steps, and expected results for each case. The two texts to be tested must also be reviewed by the Project Manager and developers. In this way, after various mutual reviews, we have formed a basic consensus on the project.

The implementation of the test in step 3 is the longest and most complex stage in the entire development process. In general, it is the process of putting the test cases designed in the previous step into practice as planned. This includes writing automated testing programs, repeatedly running automated testing programs, and performing manual testing cases in stages. Tests at this stage must be carried out under careful planning. As I have mentioned above, this is the characteristics and strengths of the first type of tests. This planning is first reflected in the coordination and cooperation of development and testing. Based on the dependency between the product architecture and functional modules, the planning is promoted jointly according to the overall project plan. From the perspective of the test process, a simple use case is always run or executed first, and then a complex use case is executed. a single basic function is verified first, and then an end-to-end function is integrated. First, a superficial solution is discovered, A bug that has a big impact, which is further difficult to reproduce. Therefore, with the process of project development and testing, the product functions are constantly improved and the quality is continuously improved. It should be noted that there are many test cases that need to be run repeatedly, especially for basic automated tests every day and every build. Although these tests are passed in most cases and few new bugs are found, the value is obvious to prevent quality regression. It can be seen that the theory of Myers is not applicable here. At this stage, the tester has another tedious but important task, namely maintenance of existing test cases. For example, some test cases are usually added in the following two cases: one is for areas where the test was not well designed, and the other is for external bugs (such as reports from Beta customers ), not covered by existing test cases. When the product's functional design changes (which is common at Microsoft), the test cases involved must also be modified accordingly.

 

Microsoft's second-class test

 

The second type of Microsoft testing is periodic and often carries randomness and Attack resistance as needed. For such tests, Microsoft has a special name: "bug bash (bug cleanup )".

Bug bash usually occurs at the end of each stage of project development (Microsoft is called a milestone). For example, before the beta version is released, a special time period (usually 1-3 days) is drawn ), during this period, all the people involved in the project focused on all aspects of knowledge and used all their wisdom to search for bugs in the project. This is a very interesting activity, but it is not easy to organize it. There are generally the following points: (1) Although this is a test activity, participants are not limited to testers. Project managers, developers, and even senior management personnel should participate, just like the whole people. The goal is to brainstorm; (2) encourage different departments and fields to search for each other, because new ideas and perspectives often help to discover more bugs; (3) To mobilize enthusiasm and increase interest, the competition mechanism can be introduced as appropriate. For example, when the activity ends, the individual who finds the most bugs and the most serious bugs will be given material and spiritual rewards. (4) It can be divided into topics, such as security, user interface availability, internationalization and localization.

In addition to the bug bash, Microsoft's second type of tests often have some professional tests. The most typical is the security attack testing. Generally, companies or industry experts are invited to search for product security vulnerabilities.

 

From the perspective of traditional software testing concepts, I have introduced Microsoft's strategy and the specific practices of Two Types of Traditional testing methods and their focuses. This is actually only a basic and primitive basis. The role and position of software testing in the Development of Microsoft software products is far from what these original methods can achieve, nor is it covered by traditional software testing concepts. Microsoft has many unique practices and conceptual breakthroughs in software testing, for example, "Information Service functions for software testing", "user-centered macro Quality System", "hierarchical test", "Project Quality Management System", and "third-party bug Review ", "Test automation" and "Software and Hardware-departments, teams, people and infrastructure for software testing. I will introduce these topics in future discussions.

 

November 18, 2004