Continuous integration and test automation

continuous integration and test automation

Author Huang Liang Source Csai I. BACKGROUND From graduation to now, I used to work in three companies of different sizes: private, foreign, and listed companies. But most of the past is to do projects, engaged in software development work, most companies do not attach importance to the test, even if there is no scale, not to mention the establishment of testing system. In short, the management of the development of light testing in China for decades, and continue to see their low wages to test engineers and teachers in the classroom when the test was a pass to know the test was ignored by Chinese bosses. In the last two years, I have been engaged in testing and project management of CRM software products. Because the company's software quality requirements are particularly high, this will inevitably cause a lot of attention to the testing work, not only requires a strong testing team, the team must have a professional level in the business, testing skills, and in the software development process often due to the testing and continuous adjustment. Fortunately, with the improvement of software development technology and tools, software engineering and software process practice, software testing has been paid more and more attention and specialization. During my testing work, I have been studying CMM, test theory, automated testing tools, and set up a complete test system. This does not introduce the entire test system, but introduces the most worthwhile part of testing: continuous integration and test automation. The aim is to make progress together with everyone. There are, of course, a lot of introductions about continuous integration and automated testing, but I'm not going to just introduce continuous integration, not just automated testing, but how to automate tests. Second, test automation Automated testing is the hope that automated testing tools or other means, according to the test engineer's scheduled program for automated testing, to reduce manual testing of the labor, so as to improve the quality of the software. The purpose of automated testing is to detect old defects. The purpose of manual testing is to discover new defects. Test automation involves the integration of test processes, test systems, automated compilation, continuous integration, automated release of test systems, and automated testing. This means that testing can be automated, not only as a matter of technology and tools, but also as a cultural issue for companies and organizations. First, the company from funds, management to support you, followed by a dedicated testing team to establish a suitable test process for automated testing, testing system; The second is to remove, compile, integrate and publish the original code from the controlled library, to automate unit tests, and to automate the functional testing process. (i) The benefits of automated testing 1. Perform regression testing for new versions--Test each feature For product-type software, each release of a new version, most of which features and interfaces are similar or identical to the previous version, which is particularly suitable for automated testing, allowing the test to achieve the purpose of testing each feature. 2. More and more frequent tests--tedious, time-consuming Our product release cycle to the market is 3 months, that is, our development cycle only a short period of 3 months, and during the test is every day/every 2 days to release a version for testers to test, a system has thousands of function points, manual testing is very time-consuming and cumbersome, which will inevitably make testing inefficient. 3, to replace the difficulties of manual testing-300 users have some non-functional aspects of the test: stress testing, concurrent testing, large data test, crash testing, employing to test is impossible to achieve. Prior to the introduction of the Automated Test tool, in order to test concurrency, one or No. 200 of the Research and Development Center in the development manager's password: 1-, 2-, 3. , everyone presses a button at the same time. It was interesting to recall the situation. 4. Consistent and repeatable Since the scripts for each automated test run are the same, it is difficult to achieve consistency in each test execution. Because of the consistency of automated testing, it is easy to find any changes to the software being tested. 5, better use of resources-weeks/night The ideal automated test can be run completely and automatically as planned, and the automated tests are capable of performing the task in a way that is not possible for developers and testers to implement shifts, and can be tested on weekends and evenings. This makes full use of the company's resources and avoids the wait between development and testing. 6, solve the conflict between testing and development Typically, at the end of development, into the integration testing phase, the developer has time to wait for the tester to test for errors due to fewer errors in the test system at the beginning of each release. In fact, there are more contradictions in the development model with short iteration period, but the main contradiction can be solved by automated test. 7, increase the software trust degree In short, the benefits and benefits of automated testing are obvious, but there is only a smooth fact that automated testing can reap its benefits. (ii), automated testing-misunderstanding, limited automation testing benefits a lot, but there are a lot of limitations, but also because many bosses to automated testing is too high, so there are many examples of failures to perform automated tests. 1. Expect automated testing to replace manual testing Automated tests cannot be expected to replace manual tests, and the tests are mostly manual. 2, the expected automatic test found a large number of new defects It is also impossible to expect automated tests to uncover more new bugs, and the more new defects prove, the greater the chance that automated tests will fail. Discovering more new flaws should be the main purpose of manual testing. Test expert James Bach summed up 85% of the defects by hand-found, while automated tests found only 15% of the defects. I actually think that automated testing can be a good spot for old bugs. 3, the tool itself does not have the imagination After all, tools are tools, there are some need to think, experience, interface aesthetic aspects of testing, automated testing tools powerless. 4, technical issues, organizational issues, script maintenance The implementation of automated testing, there are a lot of resistance, such as whether the Organization attaches importance to the establishment of such a Test team, whether there is such a technical level, for the test script maintenance workload is also very large, it is worth maintaining and so on issues must be considered. (iii), not suitable for automated testing Automated tests are not suitable for all companies, all projects. 1, custom type project (one-time) Customized projects for customers, maintenance period by the client side, and even the use of the development of language, operating environment is also the special requirements of customers, that is, the company in this regard the accumulation of less testing, such a project does not cooperate with automated testing. 2, the project cycle is very short project The project cycle is very short, the test cycle is very short, it is not worth the effort to invest in automated testing, and finally set up a test script, can not be repeated use is unrealistic. 3, the Business Rules complex object The complexity of the business rules of the object, there are many logical relations, operational relationships, tools are difficult to test. 4, beautiful, sound, usability test People's Sense of view: the appearance of the interface, sound experience, usability test, and only people to test 5, the test is rarely run: one months only run once Testing is rarely run and is a waste of automated testing. Automated testing is to make it take the trouble, repeatedly running to be efficient. 6, the software is not stable Software instability can result in automated test failures due to these uncertainties. Only when the software is relatively stable, there is no interface critical error and interrupt error to start the automated test. 7. Involving physical interaction It is difficult for tools to complete interactions with physical devices, such as swipe-card testing. (iv), what kind of situation for automated testing automation testing can be implemented in many large companies, it is suitable for the characteristics of automated testing and high return on investment. 1. Product-Type Project Product-type projects, each project will only improve a small number of features, but each project must repeatedly test those that have not changed the function. This part of the test can be fully automated testing, and can also be added to the new features of the test will slowly add to the automated test. 2. Incremental development and continuous integration project Since this development pattern is frequently released for testing, automated tests are required to test frequently so that people are freed from testing new features. 3, can automatically compile, automatic release of the system To be able to fully implement automated testing, you must have the ability to automate the compilation and automated release system for testing. Of course, failure to meet this requirement can also be automated testing under manual intervention. 4. Regression test Regression testing the strengths of the test automation test, it is good to ensure that you have introduced a new defect, the old defects have been modified. To some extent, automated test tools can be called regression testing tools. 5, repeated, mechanical action Automated testing is a favorite test: multiple repetitions, mechanical actions, and such tests will never fail. For example, to enter a large number of similar data to the system to test the pressure and report. 6, need to run the test frequently In a project requires frequent running tests, the test cycle by day, you can maximize the use of test scripts to improve efficiency. 7, the cumbersome task into automated testing Iii. continuous Integration and its automated compilation The concept of continuous integration (continuous integration) comes from a practice of XP (extreme Programming), our development model is based on the CMM, introduced some XP concepts, so our idea is to take the essence of each to fit ourselves. Continuous integration refers to the ability to automate the integration of already submitted (Check-in) code until it is published to the test server for testing. 1, the implementation of Automation day construction needs to do the following parts of the work: 2, all the source code to save in a single development server, so that everyone can get the latest source code (need to use configuration management tools to store sources: such as vss/cvs/clearcase). 3, so that the creation process is fully automated, so that anyone can enter only one command to complete the creation of the system. 4, so that the test fully automated, so that anyone can enter a single command to run a complete set of system testing. 5, to ensure that everyone can get the latest, best executable file. 6, automated compilation: In order to be able to provide automated testing, so all the code must be able to achieve automated compilation. In fact, a lot of companies in the continuous integration of the implementation of the change function: such as Java programs can be used on Ant + Junit to add their own functionality can achieve continuous integration--we call this tool: Day build But many companies do not realize the automatic compilation of JSP, for the use of JSP-written web pages, it is compiled to execute the language, because the first execution to compile first, that is, the first time a bit slow, if you want to use automated testing tools WinRunner to perform functional testing, it will fail. Because the most basic requirement of an automated test tool is that the entry and exit conditions must be exactly the same as when recording and playback. 2, continuous integration of the most benefits: Can replace the manual release, in the Java EE has a role called Deployer., its main job is to regularly release new systems for development, testing, usually at least one hours per release, such as a few problems a morning to spend, but the use of "day build" can be fully automated, Time is almost equal to compile time. It completely avoids the developers ' "bug-Meeting"-which developers often need to drive because someone has stepped into someone else's field at work and influenced the code of others, the person affected doesn't know what happened, and the bug appears. Most of these bugs can be found on the same day that they were introduced. Because there are not many changes in the day, you can quickly find the location of the error. Continuous integration can be found by the author of the source code, in the form of mail and log distribution to the author, the next day the first thing to work is to modify the error first. Continuous integration reduces the time it takes to "catch bugs" in the integration phase, and frequently releases new versions, which ultimately improves productivity and software quality. 3, the ideal continuous integration of the implementation method: (a) The same development server in which the same software product is to be centralized, that is, the most recent and compiled source code for everyone is in the configuration management tool such as VSS. B, there is a running master-created machine, a planned run day build, and a creation process in the day build, which is done in a Java class that is always running, and if no task is created, the creation process loops around waiting. C, the daemon extracts all the code (including the original program and configuration files, database scripts, etc.) into a directory where the machine is created. After the extraction is complete, the daemon will invoke the ant script in this directory. D), Ant takes over the entire creation process and makes a complete creation of all the source code. The ant script takes care of the entire compilation process and puts the resulting class file into six jar packages and publishes it to the EJB server. Once created, the Create daemon sends an e-mail to all developers who have returned the code to the latest creation, reporting the creation. E, when Ant finishes compiling and publishing, the creation daemon starts running the new jar on the EJB server and starts running the BVT test suite: Unit Testing with JUnit. After the unit test is complete, the day build sends the unit test report to the wrong developer. F), in order to use automation tools (WinRunner) for functional testing, you must compile the JSP, using the JSPC command to pack a layer, you can automatically compile all the JSP files, but because the time to compile the JSP is very long (more than the compilation of Java code longer), Therefore, a separate compilation server is generally used for compiling. Publishing a compiled JSP file has a high success rate for automated testing (because it is very slow to run the JSP file for the first time, while automated tests are most taboo to run and wait for recordings). The functional automation test also needs to be executed sequentially according to the plan, which requires the TestDirector test management system to dispatch WinRunner for testing. Automate all repetitive and tedious tasks, and integrate them frequently to automate repetitive testing. Test suite implementation Test process. With the ability to continuously integrate and test automation, a testing system is needed to support and maintain your testing process to ensure that the testing process conforms to the process, standard, and continuous improvement. (i), why a process is needed. Many companies are spending a lot of money on testing, but still not getting the expected benefits. This may be due to lack of adequate test plans, prioritization of tests, duplication of work, lack of use of tools to match manual testing, no use of test automation tools, inadequate testing automation, or improper use of tests, and so on. Therefore, a test suite implementation process is required. (ii) Why tools are needed. Tools can speed up the progress of tests, introduce control and management into the entire testing process, such as the Mi Company's testdirector is a very good test management system, but also the web version. The test management system has many functions: Test management and reporting: The test management system is able to ensure system development and testing processes you do not solve problems as soon as possible. Credential for audit trail: TestDirector stores all test results, all changes are written into an audit tracker, such as: time, date, modifier, error authorization, can clearly see the error when the ball kick not responsible for the whole process. Improve test coverage: test functionality with data driven by automated test tools to improve test coverage. (iv), test suite--the main objectives of the test system (5W3H) The test system is established to ensure that all the activities of the software test are carried out according to the plan and standard, which is the guide of the Tester's action program and responsibility. That is, there is a system, a process to guide their work, cultivate their sense of ownership. Let the test work be carried out methodically. The main contents are: Testing process, test guidelines, test procedures, document templates, quality standards, testing tools, testing techniques and methods and other content. The main objective of the test system (5W3H): The purpose is to tell the people associated with the test activities at what time, what kind of place, who to do, what to do, why, how to do, how to complete, defects in any analysis and prevention. Can be short: 5w3h. 1, why to test the system (WHY). Testing new features: Each release of a new version, the first to test its new features. Create a regression test test suite to verify defect modifications: In this test cycle, verify the defect modification of the last test cycle. Verifying system performance Detection new hardware 2, how to test the system (how). System test: Check System overall function Stress testing: Detect the response of the software under repeated operation, or under other pressure conditions, such as low memory space/low disk space, etc. Installation test: Verify that the system is installed correctly and that there is no conflict with the installed software. Security testing: Testing the level of system access permissions and authorization Boundary testing: Using data boundary and system boundary inspection procedures 3, when to test (when). At what stage of the development process do you start testing? As soon as a requirement specification comes out, or a project management plan comes out, testers begin to do something: write test plans, write test cases, perform tests, test reports, and defect analysis. Many bosses think that the code is finished before starting to test work, so refused to have full-time testers, afraid they do not have anything to do in the early stages of the project. What are the prerequisites and ancillary conditions? How long does it take to test? What is the delivery schedule? When does the test stop? When to stop testing is very learned, many companies are in the absence of time, no money is, the boss or the project manager said stop. In fact, it is determined by the time curve of bug prediction, bug discovery rate and error correction rate. This curve can only be stopped if it reaches a horizontal line. 4, who will implement the Test (WHO). Hardware: What kind of server, client, and network environment you have. Software: What software environments are installed best suited for these tests. Architecture: Test categories are many, different people carry out different tests, such as developers to do unit testing, testers for functional testing, integration testing, non-functional testing, and let market, demand personnel, customers to do acceptance test Data: What kind of test data is needed to implement this test, the design of these test data. HR: Arrange related HR According to the requirement of test plan. 5, where to test (where). Test on the development server. Developers may ask you to test on a test server, which in fact has a significant impact on the efficiency of the test and on the emotional performance of the testers because the development server is an extremely volatile environment. And there's no obvious test phase. Build a test lab? For companies with many projects, it is necessary to establish a test lab, which is mainly used for environmental compatibility testing, stress, performance testing, acceptance testing, etc. To reduce the load on the tester's local machine so that it can perform other tests while testing, The mechanism for remote timed execution of tests. 6, test what (What). What are the main features of an application in automated testing? Sort these features by importance. The relative importance of each part of the automated test? What are the overall quality objectives (availability, functionality, reliability, performance, etc.)? 7, how to calculate the completion (how)? To define test completion conditions and completion criteria, you should stop testing immediately after you meet these conditions and criteria, otherwise it is not allowed in economics and time, because the test can go on forever. 8. How to analyze and prevent defects (how)? After the test should be tested on the wrong category, the error characteristics of the analysis and put forward preventive measures, in order to be consciously in future projects to avoid, which is CMM5 said in the defect prevention. V. Automated testing tools (WinRunner) In addition, this article briefly introduces the principle of automated testing tools. 1, WinRunner basic principle--recording/playback function --recording Add-in selection before recording: it develops different add-in for different languages Settings for parameters before recording Recording mode selection: Context Sensitive Analog Recording Tips Save the recording script and GUI --Debugging Modify the recorded script. Add sync points and wait times. Add Checkpoint Checkpiont. Modify Gui-map to improve readability and maintainability. The prerequisites for playback. Execute test mode: Verification method: Check the application is correct. Debug mode: Add new features and features Update method: Updates the expected results with the results of the run in the new version of the application. Analysis results. 2. Parametric Data driven test Features: Perform different test benefits with the same test script: improve test coverage Steps: 1. Convert your test to data-driven test: Datadriver 2. Add data to the datasheet 3. Correction scripts use the correct expression 4). Custom result Information (Tl_step) 3, the use of WinRunner risk Product software, there will be a lot of their own development of components, controls or introduce new technologies such as XML,HTC, which may make automated test tools do not recognize, resulting in the entire automated test failure, the accumulated test scripts will be discarded. In short, because the commercial society to the software quality requirement is more and more high, the software development process continuous improvement, the software project continuous integration and the test automation development is inevitable, its function also will become more and more obvious. Different technologies and development environments have different requirements for how to automate tests, and there are many areas worth studying.