Mobile App Automation Testing Framework

Brief introduction

UI Automation testing for mobile apps has long been a difficult problem, with the "change" of the UI changing, resulting in extensive maintenance of automated use cases. From the perspective of layered testing, automated testing should be done on a level-by-layer basis. The largest number of automated tests should be unit testing, the easiest to implement and the easiest to detect problems early, followed by interface-level testing, to verify the logic for the purpose of automation, due to the relative stability of the interface, automated testing costs are relatively acceptable; the biggest automation cost is UI-level automation testing, However, the UI interface is a direct feedback to the user's performance display, moderate, especially the BVT level of automated testing is also very necessary. This paper analyzes the similarities and differences of several automation frameworks, which makes the testers have a reference in selecting the Automation framework.

Android Automation Framework

1. Instrumentation

Https://developer.android.com/reference/android/app/Instrumentation.html

Instrumentaion is a test framework that comes with Android, and is the basis for many other test frameworks that can load the tested component in the same process. It has many rich high-level packages, and users can use other frameworks based on instrumentation to avoid excessive two development. However, instrumentation does not support cross-application, resulting in a instrumentation-based framework that inherits this disadvantage.

2. Robotium

Https://github.com/robotiumtech/robotium

Robotium is a stronger framework developed based on the instrumentation framework. Easy-to-use encapsulation of commonly used operations. For the development of functional, system and acceptance test scenarios. It runs when it is bound to a GUI component. It installs a test case suite as an application on an Android device or emulator and provides a real-world environment for performing tests.

Advantages: Easy to write test scripts in the shortest time, high ease of use. Automatically follows the current activity. Because the runtime is bound to GUI components, its test execution is faster and more powerful than appium. You can also work without accessing the code or not knowing the app implementation. Supports activities, Dialogs, Toasts, Menus, Context Menus, and other Android SDK controls.

Disadvantage: You cannot handle flash and Web Components. It can become very slow on older devices. Because iOS devices are not supported, tests are interrupted when automated tests cover both Android and iOS. There is no built-in recording and playback capabilities, and the use of recording functions requires testdroid and robotium Recorder such as charging tools.

3. Uiautomator

https://google.github.io/android-testing-support-library/docs/uiautomator/

Uiautomator is a testing framework provided by Google, which provides advanced UI testing for native Android apps and games. This is a Java library that contains APIs to create functional UI tests, as well as execution engines for running tests. The library comes with an Android SDK.

Advantage: It gives junit test case privileges when running access to different processes. The library is supported and maintained by the Google community.

Cons: only support android4.1 (API level 16) and above. Script logging is not supported. The focus of support is Java. You cannot get current activity or instrumentation. Web views are not currently supported. Libraries only support the use of Java, so it is difficult to mix cucumber with ruby. To support the BDD framework, it is recommended to use Java's own BDD framework, such as JBehave.

4. Espresso

Https://google.github.io/android-testing-support-library/docs/espresso/index.html

Espresso is Google's Open source Automation testing framework. Compared to Robotium and uiautomator, it is characterized by a smaller size, more concise, more accurate APIs, a simple code for writing tests, and easy to get started quickly. Because it is based on instrumentation, it cannot cross the app.

5. Calabash

Https://github.com/calabash

Calabash is a cross-platform app testing framework for iOS and Android developers that can be used to test screens, gestures, and real-world function code. Calabash Open source is free and supports cucumber language, cucumber allows you to express your app's behavior in the natural English language, realizing BDD (Behavior driven development, behavior-driven development). All statements in cucumber are defined using ruby.

Pros: Large community support. The list item is simple, like the English expression of the test statement supports all the actions on the screen, such as sliding, zooming, rotating, tapping and so on. Cross-platform development support (the same code applies to both Android and iOS devices).

disadvantage: after a test step fails, all subsequent steps are skipped, which can lead to missed more serious product issues. Testing is time-consuming because it always installs the app first by default. Requires the Calabash framework to be installed in the iOS IPA file, so testers must have iOS app source code. In addition to Ruby, it is unfriendly to other languages.

6. Appium

http://appium.io/

Appium is an open-source, cross-platform, automated testing tool that supports iOS, Android, and Firefoxos platforms. With Appium, developers can test the mobile app without recompiling the app or making any adjustments, allowing the test code to access the backend APIs and databases. It is supported by a dual platform that drives Apple's UIAutomation and Android Uiautomator framework, while binding the selenium Webdriver for old Android platform testing. Developers can write test scripts in any language compatible with Webdriver, such as Java, OC, JS, Php,python, Ruby, C#,clojure, and Perl languages.

7. Selendroid

Https://www.gitbook.com/book/lihuazhang/selendroid/details

Selendroid is a framework based on the instrumentation. Fully compatible with the Webdriver protocol. The selendroid can be used on both the emulator and the actual device, as well as the integration of grid nodes for scaling and parallel testing.

8. Robolectric

http://robolectric.org/

Robolectric is an Android unit testing framework, but it does not rely on the testing capabilities provided by Android, it implements a set of Android code that the JVM can run, and then intercepts the Android-related code calls as the unit test runs. Then go to the Robolectric implemented code (shadow objects) to execute the call process. So it's not like a simulator or device that requires dexing (the android Dex compiler compiles class files into the format used by Dalvik VMs on Android devices), packages, deploys, and runs, dramatically reducing the time it takes to test execution. The pivotal lab claims that using Robolectric can run 1047 tests in 28 seconds.

In addition to implementing the existing interface of the class within Android, Robolectric also adds a number of additional interfaces to each shadow class to read some of the status of the corresponding Android class. For example, it provides the Getimageresourceid () method for ImageView, which the tester can use to determine if the desired image is displayed correctly through the Getimageresourceid () interface.

9. Robospock

http://robospock.org/

Robospock is an open-source Android testing framework that provides a simple way to write BDD behavior-driven development specifications, using the groovy language and supporting the Google Guice library. Robospock incorporates the functions of robolectic and Spock.

Ten. Cafe

http://cafe.baidu.com/#panel1

Cafe is a robotium-based test framework from Baidu, which provides a cross-process testing solution.

Athrun.

http://code.taobao.org/p/athrun/wiki/index/

Athrun is a mobile test framework Taobao, which supports Android and iOS. The android part is based on the instrumentation, which is extended on the basis of Android's original ActivityInstrumentationTestCase2 class, providing a complete set of object-oriented APIs. Automated testing on IOS includes the injection-based automation framework AppFrameWork, and the Athrun_ios, Instrumentdriver, of the automated framework for recording.

12. Other

Other automation frameworks include the Monkey series (Monkey, Monkeyrunner, Monkeytalk), which are used for stability testing, where Monkeytalk supports iOS and Android, and it enables real, functional interaction testing for applications. Monkeytalk provides a simple "smoke tests", complex data-driven test suite. Monkeytalk supports native, mobile and hybrid applications, real devices or simulators. Monkeytalk makes scene capture very easy and can record high-level, readable test scripts. There are also selenium webdriver for automatic browser testing, which can be used to test user behavior, user interaction such as touch, finger scrolling, long-press, etc., also support some features of HTML5, such as local storage, session storage, application cache and so on. The CTS is an automated tool for compatibility testing, most of which are based on JUnit and instrument panel technology. It also extends the automated testing process to automate use cases and automatically collect and summarize test results. The CTS uses XML configuration files to divide these test cases into multiple test plans (plan), and third parties can create their own plan.

Summary (Android)

The inheritance relationship of each test framework is as follows, and the inheritance relationship determines the innate superiority or congenital deficiency of some frameworks. Multiple frameworks can be integrated in real-world applications.

• Instrumentation-based testing frameworks, such as espresso,robotium,selendroid, cannot support cross-app use. such as automated testing in a cross-app operation, can be combined with uiautomator implementation.
• The automation framework that supports BDD is relatively small and can be selected between Calabash and Robospock and JBehave.
• If you want to support both Android and iOS, the optional frames are appium and calabash, or Athrun.
• If you select a frame for the unit test, you can choose instrumentation or robolectric. Robolectric implements the Shadow object class, which takes a short time.

iOS Automation framework

1. Xctest

Https://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptual/testing_with_xcode/chapters/01 -introduction.html

Xctest is a simple and powerful test framework introduced by Apple in iOS 7 and XCODE5, its tests are very simple to write and follow the Xunit style. The advantage of Xctest is that it integrates with Xcode, has a dedicated test navigation bar, but is not very rich because it is restricted to the official Test API.

2. UIAutomation

Https://developer.apple.com/library/ios/documentation/DeveloperTools/Conceptual/InstrumentsUserGuide/UIAutomation.html

UIAutomation is an apple-provided UI Automation testing framework, written in JavaScript. Based on the uiautomation has an extensible tool framework and a driver-oriented framework. The extended framework provides many useful JS tools with the JavaScript extension library approach, and the injected framework typically provides some lib or framework that requires testers to import the content into the code project for the application under test, and the framework can drive the app through them. In this way, the language of the editing script is no longer limited to JavaScript because the driver UI Automation uses the UI Automation library at the bottom of the automated test to drive the UI Automation through TCP communication.

3. Frank

http://www.testingwithfrank.com/

Frank is a very popular app testing framework on the iOS platform that uses the cucumber language to write test cases, and Frank includes a powerful "app inspector" –symbiote that you can use to get detailed information about your running apps. Enables developers to conduct test reviews in the future. It allows you to write a test scenario for structured English sentences using cucumber. Frank requires that the test be compiled inside the application, which means that changes to the source code are mandatory. This is done by using the cucumber and JSON combination commands, sending commands to a server running inside the local application, and running the command with Uispec.

Pros: The test scenario is written in understandable English sentences with the help of cucumber. Powerful symbiote real-time inspection tool. Active community support. Constantly expanding the library.

Cons: support for gestures is limited. It's a little hard to run tests on your device. Modifying the configuration file requires running on the actual device. The logging feature is not available.

4. KIF

Http://www.oschina.net/translate/ios-ui-testing-with-kif

KIF is an abbreviation for the Keep It functional project, an iOS app functional testing framework, written in objective-c language, easy to use for Apple developers, and a widely recommended test tool for developers. KIF Tester uses the private API to understand the view hierarchy in the app. But the disadvantage is that it runs slower.

5. Calabash-ios

See calabash-android description.

6. Subliminal

http://inkling.github.io/Subliminal/

Subliminal is another framework that integrates with Xctest. Unlike Kif, it is written based on UIAutomation, which is designed to hide some of the complex details of uiautomation in developers.

7. Kiwi

https://github.com/kiwi-bdd/Kiwi/wiki/Getting-Started-with-Kiwi-2.0

Kiwi is a complete replacement for xctest, writing tests using the Xspec style. Kiwi comes with its own set of toolsets, including expectations, mocks, stubs, and even asynchronous testing. It is a behavior driven development (BDD) library for iOS development, with its simple interface and usability, easy to set up and use, and ideal for novice developers. Kiwi is written in objective-c language and is easy for iOS developers to get started with.

Summary (IOS)

The iOS automated test framework inherits the following relationships. Xctest integrates directly with Xcode's IDE and is simple to use, but it does not support stubs and mocks, so fewer xctest frames are used alone. Kiwi is an iOS platform that is very useful for behavioral-driven development of BDD's test framework, with a very nice syntax that can write highly structured, easy-to-read tests. UI Automation is the solution for Apple's official UI Automation tests, but the interface is not rich enough.

• KIF, Frank, and Calabash simulate event firings in the form of code, so that the code being measured acts as if it were triggered by user behavior. But the cost is the complexity of inserting an extra layer.
• There are calabash and Kiwi that support BDD in the iOS test framework.
• The optional Unit test framework is kiwi,specta,quick and so on, while Kif,subliminal and Calabash are more suitable for UI-level acceptance testing.

Some interesting automated test frameworks

1. Sikuli Graphic Programming Technology

http://www.sikuli.org/

Sikuli is a new graphic programming technology released by MIT's research team. Based on image retrieval technology, it provides a set of Python-based scripting languages and an integrated development environment. The user can use the screen to directly refer to the GUI elements to program, complete the interactive operation. Sikuli scripting follows the Python syntax specification. Because Sikuli is based on Python, its core code is written in Java and can be referenced in a user-defined Java project as a Java standard class library.

Its script is in this style:

Sikuli directly refers to the GUI object's screen as a function parameter, and the whole code is very clear and readable. During the execution of the script, the image retrieval algorithm is used to analyze and match the corresponding control in the current screen, and the corresponding mouse or keyboard action is applied to it. This way, when scripting, we don't have to worry about cumbersome application-related APIs or get Web content objects.

Disadvantages:

1. Only Windows, MACOSX, and Linux platforms are supported, and mobile platforms are not supported.

2, rely on the screen, make 1) in different platforms, different resolutions, different operating systems need to maintain a set of graphic source files, not conducive to cross-platform transplant; 2) if the interface occlusion outside the program logic, the program execution is affected.

However, as an addition to the existing automated testing tools, especially for projects that cannot get the API, such as Flash animations, it is very effective.

2. A/b test framework Appgrader

Https://www.utest.com/articles/utest-launches-appgrader-for-android

Although Appgrader is not a first-class testing framework, it also has a director. It can help developers to compare their applications with many other types of applications in many different forms and functions. By comparing the results, developers can improve and improve their applications more specifically. Currently Appgrader only supports Android platforms.

3. IOS A/b test framework Fliptest

http://www.fliptest.co.uk/

Fliptest is an excellent iOS app A/B testing framework that lets you choose the best UI for your app. Fliptest will return test results based on many factors, such as appearance and ease of use, to help developers solve UI problems. Testing with Fliptest eliminates the need to resubmit the app to the app Store or drastically change the code by adding a single line of code to the app, saving you a lot of time.

Reprint: http://tmq.qq.com/2016/09/mobile-app-test-automation-framework/

Mobile App Automation Testing Framework