Thoughts and Reflections on QCon2015

QCON2015 Special topics on architecture optimization, special areas of tuning, but can systematically describe product testing direction only "Ctrip Wireless app Automation test practice."

 

(i). Ctrip's Wireless App automation

"Ctrip Wireless app Automation test Practice" mainly introduces Ctrip in the direction of app automation.

1. First summarize what they have done:

(1). Mobile device real-machine automated test cluster, and cluster monitoring.

(2). The device in the cluster can provide remote manual control services (remote operation of the phone screen).

(3). Write a case based on Appium.

(4). Establish a basic automated test model (Automation case: Manual case is 3:7).

2. Output effect of app Automation test:

,

(b). Reflection

1. What can be improved by Ctrip:

(1). PC utilization is low: Ctrip 1 PCs only hang 2 mobile devices.

The ADB find device/emulator port is 5555~5585, each device/simulator occupies 2 ports, so the PC can generally hang 15 devices/simulators. When we were doing mobile automation testing, it was a PC hanging 6~8 mobile device.

Of course, Ctrip does not do this for certain reasons:

A) Adb-server and mobile device I/O, there will be occasional card death.

Cause: The mobile device does not return, causing Adb-server to read () without data. In addition, adb-server and device I/O are locked and not timeout, which will result in: One device read the card dead, other equipment also do not want to work ...

Solution: ADB monitor, find abnormal adb, kill and restart. Restarting ADB will have side effects (the execution case will fail), but can be resolved by the power of the case execution and the recurrence failure case.

b) ADB is unable to withstand large amounts of I/O concurrency

ADB design goals are not designed for high concurrency (visible from lock + child process limit) ... So when the 15 devices at the same time high-frequency command transmission, is very slow ... But single PC load 6~8 mobile device, still can.         So, under the malicious speculation, Ctrip only hang 2 sets of purpose is to save trouble.

(2). The test results do not have the function correctness conclusion

Some of Ctrip's test types (e.g., accessibility testing, and so-called smoke testing ...) ) cannot be judged by the function correctness. This is controversial in the industry, some people think: can test the activity page crashes/webpage whether the load is successful, the other part is that: manual rework test to ensure that the function is correct, so it does not need to exist. Individuals are more inclined to the latter. In addition, this also reflects a problem from the side: UI-level automation testing implementation is not easy, here is not far away.

2. Ctrip's merits:

(1). Provides device remote manual control Service (remote operation of the phone screen).

This feature is not new, Testin has been providing similar services in 2012, except 13 years after shielding the external. It has the advantage of pooling resources to reduce equipment redundancy.

(2). Write a case based on Appium.

I was in charge of it. YY Mobile End test Framework "testagent" Implementation and maintenance, the reason is that based on appium implementation of automated testing is desirable, for two reasons:

A) to provide a good case writing/programming style.

UI Automation Testing Framework implementation, I think there are 3 priorities:

1. Manipulate the method that is being tested to apply UI elements.

2. Provide easy-to-read, easy-to-use APIs.

3. Provide convenient, complete element index way.

2, 3 is the basis for writing elegant test cases. Of course, for both 2,3,appium and testagent.

b) Open source community support.

Appium and other testing frameworks (including: Ali, Baidu, and yy ...) ) The difference is here.

Open Source Community provides: A lot of experience exchange, technical support and continuous improvement;In addition, the widespread popularization of open source tools has also reduced the difficulty of recruiting and transferring personnel in the future.

(3). Ctrip has an independent foundation tools research and Development department to support the top business testing

The above-mentioned advantages and disadvantages, are technical aspects of the problem. But this non-technical problem is my greatest feeling.

since the test center has been spun off to various project teams, bringing a more efficient project team, I feel there are some side effects: knowledge and tool sharing are harder than ever. I think it could be one of the reasons that the breakthrough in automated testing has slowed down after the test center split, and it's difficult for business testers to take into account tool development + Automation platform operations and case writing + maintenance of 2 responsibilities in daily testing. Ctrip's automated testing has made this progress, with the support of the basic tools sector being a big reason, which is precisely what the company lacks.    

Thoughts and Reflections on QCon2015