An ecosystem implementation for Web site automation testing

This is an "ecosystem" that I built in my work on Web site automation testing. The concept of "ecosystem" is what I learned from the company's predecessors, and he has always believed that automated testers should not only be confined to writing test code, but should also automate the entire process of automated testing (test code continuous integration, distribution, execution, etc.) to form a "system" The higher the automation of this system, the more effort the automation testers will have.

First, the concept

Here I drew a picture of:

It is called "ecosystem" because there is very little human intervention needed after completion, and the rest of the time is the internal cycle of the system. As an automated tester you only need to commit the code, and then you can see the results of the run on the Automationdashboard, and the rest is digested inside the system. Of course, the results of the analysis still need to be completed, the machine is not smart enough to be able to flexibly analyze a variety of cases fail to drop the reason.

We can think of the whole system as a black box, so the graph above can be turned into:

In fact, the people who draw here are not confined to automated testers, but can also be:

(1) Product managers, such as product managers need to know from the automated regression test release There is no risk of delay;

(2) Team managers, such as development managers and QA managers, need to know the latest code quality from an automated daily/weekly regression;

(3) Developers, they may want to know that the code they just submitted has broken the basic functionality of the system by using quick regression (a test that was run after the submitted product code was deployed to the test environment);

(4) Other developers who help with automated testing, who are just beginning to learn how to write automated test code, don't care about the internal implementation of the ecosystem.

Second, the realization

Speaking of the concept, the next step is to talk about concrete implementation. I'm talking about what I think is best for the implementation of the product I'm testing, more than one tool, and more than one way. Jenkins can be replaced with teamcity or other ci, git can also be svn or tfs,automationdahsboard can be implemented with. NET, Springmvc, Ror, and so on, the slave of running tests can be windows/ Linux/mac (Local tyrants!) ), choose the one that best fits the product you are testing. Another point is that the automated test code is implemented with the keyword-driven idea, which is a different topic, there is time to write another article.

Okay, get to the chase. Let's talk about every important part of the system in turn:

1, SCM (Source Code Management). I chose Git, which could be a git server (the company built a git server itself) or a bare repo (http://www.saintsjd.com/2011/01/what-is-a-bare-git-repository/).

2, CI (continuous integration). I chose Jenkins, which is easy to deploy and rich in plugins.

Its role is to:

(1) Remove the code from Git, build (. NET corresponds to MSBuild, if Ruby is not build, direct deployment can be);

(2) Deploy the build *.dll (here is a copy) to all the slave;

(3) Start or stop the automationservice on all slave (also referred to Automationservice later) to control the execution of the test.

I am in the Jenkins of these job configuration is still more cumbersome, to be fine and can write another article. Here is a special mention of two very useful plugins:

(1) Parameterized Trigger Plugin (https://wiki.jenkins-ci.org/display/JENKINS/Parameterized+Trigger+ Plugin): You can trigger the build of other project in a build step.

Its most useful is this "Block until the triggered projects finish their builds" option, On the hook, Jenkins will be able to continue to the next build step after all trigger project has completed the build (rather than just trigger other project builds, and continue to the next build, without waiting for them to finish). Do each build step in the real sequence.

(2) Nodelabel Parameter Plugin (https://wiki.jenkins-ci.org/display/JENKINS/NodeLabel+Parameter+Plugin): In all " Possible nodes "The specified label (" label ") on the Jenkins node (which is the Jenkins master or Jenkins Slave) triggers the designation project (the triggered project is parameterized).

For example, I have a project called "Startclassicrolatserviceonallnodes", which has a build step that is set up like this:

Take a look at the "Startclassicrolatserviceonasinglenode" project setting:

This project has a node type parameter, the parameter name "Nodex" corresponds to "Nodex" in the previous label factory, "Possible nodes" is selected "All", then all of the listed node (master, 10.107.122.152, 10.107.122.153, 10.107.122.154) are within the scope of judgment (determine if there is a "Node" tag, and then execute project).

In addition, all of the listed node I added a "node" tag.

In this way, when I trigger "startclassicrolatserviceonallnodes", I will be in master, 10.107.122.152, 10.107.122.153, 10.107.122.154 These 4 node to execute "Startclassicrolatserviceonasinglenode" simultaneously.

3, Automationdashboard, here is the translation of "Automated Test Control Panel" bar. In fact it should be called the control Panel with Jenkins, but since Jenkins has APIs to call, the two things you want to do can be unified into a single Web interface. This dashboard is completely used. Net+iis+sqlserver a little bit from the database design and construction, data access layer, the business layer, the performance layer to do up, to speak carefully ... The amount of ... It will be another article (Oh man, not again!). Anyway, I think, although I do automated testing work, but should not confine myself to testing. To better automate testing, develop Web sites, install virtual machines, and other tools to use, you should take the time to learn and master them.

Well, say this dashboard. Here are only two main components, one website (hereinafter referred to as dashboard), a Windows Service (hereinafter referred to as Atservice), and a console application (hereinafter referred to as Consolerunner):

(1) Dashboard, its main function:

A, show the health of the test: How many are running/executed, respectively, on which slave to execute, and so on.

b, through the call Jenkins API to trigger Jenkins job, indirectly control the execution of the test.

C. Show the results of the test: which case, error time, error message, Code backtracking (stack trace), or even an error occurred.

The main interfaces are as follows:

A, Summary, as the name implies is a summary of information, case how many pass how many fail, case by classification of each class how many and so on. (In fact, I do not have a very important picture here, is the coverage pie chart)

B, queue, test queue, contains the currently running, running, run test fixture or test case (depending on the test tool, NUnit, JUnit, RSpec, and so on, fixture may be called differently, in short it contains multiple test The collection of case). Can be started, stopped, terminated (can be emptied after termination) the test executes or empties the current queue.

C, TestCase, all the tests in the ecosystem are shown here, you can see the time and status of their last execution (Pass/fail), click on a case to jump to all test results for that case. You can filter the use Cases by status (Pass/fail/other), and you can tick some of the use cases to re-execute, or to re-execute all fail case. The "Reload test Cases" is primarily concerned that the Test case in the *.dll file may change after a deployment and needs to be reloaded. But then I modified the job in Jenkins to reload automatically after each deployment, so this button doesn't really work.

D, TestSuite, a collection that contains multiple fixture is a suite. Check multiple suite clicks "Run Suite" to add the fixture included in these suite to the queue.

The suite here is a complement to the category in NUnit, click on "New Suite" and you can choose fixture to compose your desired suite:

E, TestResult, show the results of all test case, can be filtered by the test case ID, click the ID of the tc# column to show only the results of this case.

The blue "I" icon to the right of the point can jump to the detailed page of the result, the function is not enabled, according to Runnermessage and Runnerstacktrace can know the error code location, and then try to reproduce the problem.

(2) Atservice. The Windows Service (slave windows, later) was installed on each slave to ask dashboard, "Is there a test fixture/case assigned to me now?" ", if there is and the current slave idle, then grabbed to run, run finished report results.

Remember the queue? Whether you choose Test Case/suite on the TestCase or Testsuite page, you simply add them to the queue (and, to be precise, insert records into the database table of the queue) without assigning them slave. Only after Jenkins initiates the Atservice on the slave, does atservice go to the queue table to fetch itself (marking that the fixture/case already has a master, Other slave will not be caught) test fixture/case that have not been run and have not been assigned a slave.

(3) Consolerunner, the first figure was not drawn. This console program is primarily intended for Jenkins calls. Isn't Jenkins allowed to run the job on a regular basis? Right, timed call this console application, pass a few parameters, you can fill the queue at a specified time fixture/case, and then start Atservice start to perform the test. This will enable the unattended operation of the Quick/daily/weekly/full regression.

4, Slave

I choose to run tests on Windows:

(1) Company it generally only provides virtual machines for Windows operating systems

(2) The majority of users of products on Windows (in fact, this is a bit of nonsense, desktop OS Windows is still the world's King.) Admittedly, I use Linux for my own development, Mac is quite popular at home and abroad, but Googleanalytics shows that most of the visits come from windows. What, you say ios/android? The amount of ... The mobile side is still the short board of the product ... ）

If you choose Linux, pay attention to the Selenium Webdriver native event setting (http://code.google.com/p/selenium/wiki/AdvancedUserInteractions #Native_events_versus_synthetic_events).

With regard to browsers, Firefox, Chrome, ie all, Webdriver's browser compatibility is already very good. Browser compatibility is a bit of a headache, to support a lot of browsers can sometimes add a lot of development, testing costs, I usually run on Firefox is enough. What the? Digital? Toilet? Penguins? You think more, selenium official does not support.

How many slave can you find to perform the test? Oh, more! Can't find so many physical machines to own virtual machines, distributed operation can give your automated test ecosystem loaded with rockets! Run more tests in less time to get feedback faster from your tests!

Well, that's pretty much all. There are a lot of details left in the later text to say. Self-perception This ecosystem still has a lot of functions that can be perfected and added, and this implementation and operation mechanism may not be applicable to the product you are testing, but now it is enough for the product I have measured.

Regardless of how it is achieved, I would like to express the following core points:

Do automated testing is not confined to the writing of automated test code, we want to automate not only the manual test case, but also the entire automation test process! Test code continuous integration, deployment (distribution), execution, results display, the more aspects of automation, the more thorough, to save you more time, you can use these saved time to do more meaningful things. Human invention of the computer, code programming, in fact, is an automated process. In the past to rely on manual labor is now done to the computer to do-the server is not diligent to repeat the implementation of the program we wrote it? Building an automated test ecosystem is the same thing, because machines can do repetitive work more reliably than people do.

If you're still copying *.dll manually and need to open NUnit to perform tests manually and run tests on 1 machines, now is the time to increase productivity!