Running RF cases in parallel using Pabot

First, the problem introduced

In the interface automation with the increase in cases, especially the number of process cases, especially the WebForm type of ASP. NET, and the database validation is also time-consuming, resulting in RF execution cases more and more long, encountered such a problem, 705 interface test Cases (case contains process class cases, A process class case may contain 3, 4 individual cases, a single case is also included in the login), through Jenkins on a single machine to run 13 hours and 30 minutes, and currently not the entire project all the interface test cases, the number of cases continue to grow, a system if there are 3, 4,000 interfaces that would be scary if you had to run for a day or two at this speed.

Second, the solution

So is there any way to shorten the running time? One is the large project into several sub-modules, test cases are divided into several sub-modules, so that you can do distributed testing, the second is to carry out parallel testing, on a machine to open a number of processes to test, the best solution is a combination of one or two, in the online search, did not expect to have open source class library Pabot. Pabot runs RF test cases in parallel with suite by opening multiple processes, and there is a lock mechanism for resource sharing issues between processes. It is also compatible with RF command-line parameters so that the RF commands are easy to modify in Jenkins, just add a few parameters.

Third, testing

There's a new Pabotdemo, with 3 Suite, 3 test in each suite, and each test performs the same steps sleep 10s, pausing 10s. If you run 9 test in RF, each test takes 10s, then 90s is required. Test results for running in RF.

If you use Pabot to turn on multiple processes running in parallel, it will reduce the elapsed time, which is 2 processes and 3 processes respectively.

2 processes: Pabot--processes 2 Pabotdemo, the original case of 90s is time consuming 63s.

3 processes: Pabot--processes 3 Pabotdemo, the original case of 90s is time consuming 32s.

The above simple test of using Pabot to open multiple processes in parallel to execute the RF case, there is no inter-process resource sharing, so no lock, specific use can refer to: Github.com/mkorpela/pabot. Using Pabot to open 2 processes or the 705 test cases mentioned above in the original single execution machine, it took 5 hours to reduce, the pass rate was increased, and the running time decreased to 8 hours 30 minutes.

Iv. further optimization

Open 2 processes in parallel to run 705 cases to reduce the running time of 5 hours, if more open several processes or there is a drop of space, in addition to open several processes, but also to optimize the case. We should all download files, some large files a few g, some only a few m, small files are quickly downloaded, large files may just start, the impact of download time of the key factors may be a few large files, in order to achieve the purpose of parallel download, large files can be divided into several small files, so that parallel download effect better. Because Pabot runs in a suite, the case structure of the project has more than 100 cases in the suite, and there are only a few, which leads to fewer cases of the suite that may have run out, and the case-based suite may have just begun, And can not play the maximum effect of parallel operation, distribution is uneven, so need to adjust the case, the case of more than the suite to split, so it can open more than one process, or a total of 3 suite to open 5 processes, it is useless, or only 3 processes are running, the above is just a download file example, There is a lot of relationship between the whole and the part, a bit similar to the cask principle.

V. Advantages and Disadvantages

This is just an interface case execution using Pabot, and for a UI test case it might kill the browser process, and there is no test that can be tested with Pabot.

Vi. Summary

The above optimization case structure is also for parallel execution of the optimization of the case, in fact, there are places to optimize, such as login, now each case in the execution of the first login, is more than a get and post requests, especially the kind of JSON, may be first get, post login, Then only the POST request is finished, if you can simulate the operation of people, the same user only logged in once, the efficiency should also be improved, the case is less likely to be less obvious, the case of more words should be obvious.