Spoon + robobench + jenkins automated continuous regression testing, robotiumjenkins

Spoon + robobench + jenkins automated continuous regression testing, robotiumjenkins
Significance of Automated Testing: Not to mention it as a layman. Even those who are engaged in automated testing have such doubts as today or once. They have worked hard to write automated testing cases, but basically cannot find problems, what is its significance? Before explaining this meaning, let's take a look at the definition of quality.
Definition of quality:Definition of Quality in Wikipedia: the Chinese mainland is also known as "Quality". It can refer to the characteristics, character, and nature of an item, or the level and Quality of a product or service. Factors that affect quality include the reliability and security of items, whether the functions are complete, and whether they can meet the needs. Definition of software quality: Software Quality refers to the quality of software in a software system or system, that is, the degree to meet user needs, including functional and performance requirements. The quality of software includes functions, performance, reliability, security, scalability, maintainability, and other quality features, this also guides most quality-related tests. From the above definition, we can also see that for quality, people's cognition is more about "quality, in terms of "quantity", quality should be a combination of quality and quantity, focusing both on "quality", that is, quality assurance, and on "quantity", that is, efficiency. With the rapid development of the Internet, especially the arrival of the mobile Internet, enterprises hope to respond to market changes more quickly and ensure the quality of their products, therefore, various software projects, such as agile, continuous integration, continuous delivery, automation, and other technologies, were born. That is to say, for quality, our goal is not just quality, but the true "quality", that is, moving fast and don't break things !. After knowing the nature and purpose of quality, let's look at the previous problems to explain more. Compared with manual testing, automated testing is far more likely to discover problems than manual testing, however, as mentioned in the previous article, the significance of quality lies in "quality" and "quantity". Therefore, the greater significance of automated testing lies in "quantity" and in improving efficiency, in actual projects, automation is not just about automating testing, but automation should be performed for high efficiency and automation. Of course, in theory, automated testing should still be able to discover about 20% of the problems. For automated testing that cannot find any problems, you need to think about the test case design and test framework establishment issues. This article will share how to perform test automation better, faster, more stable, and more reliable. Before getting started, let's take a look at the meaning of quality.
Significance of quality:For users, higher quality means higher software quality:Imagine an example: One day, Daming was dreaming about eating a delicious roast duck and was suddenly woken up by an alarm on his cell phone. The night before Daming knew that there would be an important meeting tomorrow morning, therefore, it was only half an hour earlier than the original time that the phone had triggered an alarm .. Daming had to go out to work without waiting for breakfast in a hurry, and quickly called a car using taxi software. Daming thought that he should be in a hurry and waited patiently for the arrival of the taxi, in the twinkling of an eye, the past 10 minutes have passed. Daming started to be in a hurry. Why didn't he get a taxi .. After another 10 minutes, the taxi finally appeared. Because the novice taxi driver did not know Daming's residence, he used the car navigation. As a result, the navigation system did not update the map data in time, the driver went to the wrong place .. Unexpectedly, because he failed to attend the meeting in time, Daming received a bid. It's almost noon. due to bad mood, I don't want to go out for a meal. Daming called a takeout with a takeout software. He's already hungry, Daming, and listened to the music and hoped to relax, as a result, the music was inexplicably interrupted several times, making the mood worse... As for the afternoon, I won't talk about it. In the end, Daming spent a depressing day. Although the above Daming is a little miserable, it is by no means impossible to happen. With the arrival of the mobile Internet, everyone's life needs mobile phones and their software, everyone's happiness is greatly influenced by various software. If all the software can run well, how beautiful the world will be ~
For enterprises, quality will mean the true "quality ":With the explosive growth of information, the pace of our times is getting faster and faster. Especially on the mobile Internet, a software project cycle cannot be as half a year or even several years as traditional software. Imagine if your software project's continuous delivery capability is weaker than your competitors, it will mean that the other party can respond to market changes faster and put ideas into reality faster than you do, obviously, you will lose the market and on the mobile Internet, in which case, the huge marketing operation cost will not be recoverable. Therefore, it is imperative to improve the overall efficiency of software projects, it is also necessary to constantly think about how to speed up the project, improve the capability of continuous delivery of products, and more effectively ensure the quality of products. "To make better software available to people all over the world", presumably this should be the task and mission of quality, testers, and even all software-related practitioners ~ Based on the above background and practice, we will go to the topic spoon + robobench + jenkins for automated continuous regression testing ..
Reasons for automated testing:As mentioned in Agile testing, manual testing takes too long, manual processes are prone to errors, automation gives people time to do more valuable work, automated regression testing provides a safety net, automated testing can be provided early and frequently feedback, test documentation, etc. In short, automated testing can improve testing efficiency. After repetitive activities are automated, more time can be spent on more creative and Exploratory tests that require human brain thinking. Necessity of automated testing based on UI: for software, even if unit test cases and component test cases are compiled, these tests do not combine the software into a whole for integrated testing, therefore, it is difficult to find the most similar integration test problems to users. Moreover, many project teams have no unit tests at all. For android, the system version fragmentation is serious. Compiling UI-based automated test cases can also be used for compatibility testing.
App automated testing should include the following basic capabilities:Multi-host parallel execution: automated test cases are executed on multiple mobile phones at the same time to perform regression testing and compatibility testing in different versions of Android operating systems. Retry and error: when the test case is not passed, automatic re-run and record can be performed to reduce the real-time log records of failed use case execution due to accidental factors: The test execution process should be able to record runtime logs, this allows you to describe in detail the test execution across applications. You can test the cross-application situation to generate a Junit form test report. A detailed Junit form test report is generated, you can easily view the test case execution result code coverage report: generate the code coverage report to further guide the test policy's continuous and fast feedback capability: for test running status, quick feedback on easy-to-access reports should be provided: The test report details can be easily accessed.
Spoon IntroductionProject address: Workshop. After downloading the project, go to the spoon/website/sampledirectory and upload index.html to see the following example:
After you click "View", the following functions are displayed:

The directory structure of the report generation is as follows:
Junit-reports collects xml reports in junit format. Using the junit plug-in of jenkins, you can easily generate a unit test report and run the following command:
Java-jar spoon-runner-1.1.1-jar-with-dependencies.jar-apk example-app.apk-test-apk example-tests.apk
Detailed parameters:
Options: -- apk Application APK -- fail-on-failure Non-zero exit code on failure -- output Output path -- sdk Path to Android SDK -- test-apk Test application APK -- title Execution title -- class -name Test class name to run (fully-qualified) -- method-name Test method name to run (must also use -- class-name) -- no-animations Disable animated gif generation -- size Only run test methods annotated by testSize (small, medium, large) -- adb-timeout Set maximum execution time per test in seconds (10 min default)
Note: 1. Since the static pages in the spoon report use the online fonts in googleapis, the report may be open rather slowly <link href = "http://fonts.googleapis.com/css? Family = Roboto: regular, medium, thin, italic, mediumitalic, bold "rel =" stylesheet "> therefore, we recommend that you access ingress <link href = "static/fonts.css" rel = "stylesheet">
2. spoon does not provide the Option spoon that enables code coverage. Because it only encapsulates the am instrument command, the final execution is still the command of the am instrument execution case. Therefore, the code coverage function can be enabled, to add the-e coverage true option to enable it to collect code coverage, you need to modify the spoon source code. The most convenient option is to write SpoonDeviceRunner in the source code. add the following two lines of write dead parameters in java:

LogDebug (debug, "About to actually run tests for [% s]", serial); junitReport = FileUtils. getFile (output, JUNIT_DIR, serial + "_" + I + "_" + ". xml "); runner = new RemoteAndroidTestRunner (testPackage, testRunner, device); runner. setCoverage (true); // Note: After set is set to true, You need to insert a pile to generate the ec code coverage Statistical File runner during application packaging. addInstrumentationArg ("coverageFile", "/sdcard/robotium/spoon" + I + ". ec "); runner. setMaxtimeToOutputResponse (adbTimeout );

In this way, after the test is completed, the ec file for coverage rate statistics of emma will be generated in the SD card, upload the ec file pull to the CI Server java-cp/usr/local/program/emma. jar emma report-sp $ source_path-r xml-in coverage. after em, $ ec_files generates the corresponding xml file, you can generate the coverage report based on the corresponding plug-in jenkins.

3. spoon cannot execute the test case set separately.

Adb shell am instrument-w-e class com. android. foo. FooTest, com. android. foo. TooTest com. android. foo/android. test. InstrumentationTestRunner

As shown above, we know that multiple use case sets can be specified when the use case is executed using am instrument. After the application is started, all the use case sets to be executed will run in the same process, if the crash application is tested when FooTest is executed, TooTest will not be executed, and the test results collected by you will not include the results of TooTest. When you have a large number of use cases, it is clear that you want to execute all the use cases whenever possible during each regression test, rather than failing to execute all the subsequent use cases due to the previous use case crash. The separate execution will be similar to the following: adb shell am instrument-w-e class com. android. foo. fooTest com. android. foo/android. test. instrumentationTestRunneradb shell am instrument-w-e class com. android. foo. tooTest com. android. foo/android. test. instrumentationTestRunner wants to make spoon support this mode of operation can only modify the spoon source code for the above points are not good, the blogger fork made some modifications to the source code, see: https://github.com/hunterno4/spoon so far, through spoon can be completed Multi-host Parallel Execution, Real-time Logging, Generate Junit form test report and collect code coverage reportAnd other functions
Error retry and errorFor automated test cases, if they are executed only once, they are often unable to pass the test case because of accidental factors, obviously, no one will be willing to read such a report soon. Error retry general practice: 1. collect the console output during test execution, and analyze whether the output contains certain specified keywords to determine whether the test failed and re-run the task. This method is cumbersome. 2. According to the Android junit built-in @ FlakyTest implementation mechanism, rewrite runTest override runTest () method can refer to the http://qa.baidu.com/blog? P = 985 Note: 1) When rewriting runTest, note that the execution process of the test case is as follows: setUp () --> runTest () --> tearDown () here, the re-run only repeats the test * () method. Therefore, for many cross-Activity test cases, such as redirecting from Activity A to Activity B, however, the assertion in Activity B fails. When re-running, the test *** () method will continue to be executed and click the code to jump to B, however, the interface is still in B, and the controls in A are obviously not found. Therefore, when rewriting the runTest () method. automatically jump to the initial Activity before runTest. This can be done through solo. goBackToActivity () completion 2) when a failure occurs, we often not only want to capture the picture when an error occurs, but also want to capture a series of running processes, this can be done through solo. the startScreenshotSequence () method is implemented as follows:

@ Override protected void runTest () throws Throwable {String testMethodName = getName (); String currentTestClass = getClass (). getName (); LogUtils. logD (TAG, "currentTestClass:" + currentTestClass); boolean isScreenShot = true; boolean isScreenShotWhenPass = false; long startTime = 0; long endTime = 0; holo holo = new Holo (getInstrumentation (), getActivity (); // here, holo is the added and deleted robotium, which can be understood as a String by solo. CurrentActivity = getActivity (). getClass (). getSimpleName (); LogUtils. logD (TAG, "currentActivity:" + currentActivity); Method method = getClass (). getMethod (getName (), (Class []) null); int retrytime = 3; if (method. isAnnotationPresent (RetryTest. class) {retrytime = method. getAnnotation (RetryTest. class ). retrytime (); isScreenShot = method. getAnnotation (RetryTest. class ). isScreenShot ();} LogUtils. log D (TAG, "isScreenShot:" + isScreenShot); int runCount = 0; do {LogUtils. logD (TAG, "runCount:" + runCount); try {holo. goBackToActivity (currentActivity); if (runCount> 0) {// when the first execution of the use case fails, enable the sequence. You can design holo based on the actual situation as to how many images are to be captured. stopScreenshotSequence (); holo. startScreenshotSequence (endTime, 5, testMethodName, currentTestClass);} startTime = SystemClock. uptimeMillis (); super. runTest (); endTime = SystemClock. u PtimeMillis ()-startTime; LogUtils. logD (TAG, "run test" + testMethodName + ", testcase pass with time cost:" + endTime); if (isScreenShotWhenPass) {holo. takeSpoonScreenShot (testMethodName, currentTestClass, testMethodName, DEFAULT_QUALITY);} if (holo! = Null) {holo = null;} break;} catch (Throwable e) {if (retrytime> 1 & runCount <retrytime-1) {runCount ++; endTime = SystemClock. uptimeMillis ()-startTime; LogUtils. logD (TAG, "run test" + testMethodName + ", testcase failed with time cost:" + endTime); continue;} else {if (isScreenShot) {LogUtils. logD (TAG, "takeScreenshot:"); holo. takeSpoonScreenShot (testMethodName, currentTestClass, testMethodName, DEFAULT_QUAL ITY);} if (holo! = Null) {holo = null;} throw e ;}}while (runCount <retrytime );}

In addition, the client of spoon itself does not support sequences. Therefore, it can be combined with the sequence function of robotium itself, as long as the directory can be understood by spoon runner.
Cross-application capabilityFor the instrumentation-based framework, the cross-application capability is weak. To cross-application without using the server, the root mobile phone is needed. However, with the continuous improvement of the android system version, android4.3 and above will soon become the mainstream, and after 4.3, based on Uiautomation, we can easily cross-application. In addition, for the instrumentation-based framework, our testing project can also freely call the android api to complete functions not available in robotium, for example, text message interception, network switching, unlocking, and lock screen. Take network switching as an example: when we implement related functions, we may need to add user-sensitive permissions, such as: <uses-permission android: name = "android. permission. CHANGE_NETWORK_STATE"/>
Such permissions may not exist in our tested applications, and we do not want to add such permissions to the tested applications. Therefore, we can only add these permissions to our testing projects. Because the tested application does not have the corresponding permissions, our test cases are in the same process as the tested application, therefore, if you want to switch the network, you will obviously report permission issues if you call the relevant APIs directly. In this case, we can encapsulate the relevant methods and then call them in the test case through broadcast. We register a receiver in our own test project. When we receive the broadcast, we call the api for network switching, in this case, the network-related api is run in the test project process, and the process has relevant permissions. Terminate this process. In the process of testing project .apk, We can do most of the things that common apk can do, or even do things that common apk cannot do. For example, you can write an activityin the test project .apk, and then write a testcase in the test project .apk to test the Activity. We can also enable the service to monitor cpu, memory, traffic consumption, and other performance indicators in the test project when the tested application runs automatically. We can silently connect to the network, send http requests, and switch settings through Uiautomation or even without the need of root. We have almost omnipotent power, and we are willful...
Continuous and fast feedback capability
After the test is completed, you can send feedback via Email. For jenkins, you can use Email-ext plugin to expand the mail function. This plug-in also provides many email templates that can be directly displayed in emails, such as svn change set and junit report details. Before selecting the mail template you want to use, you can test it:
Note: To use a mail template in jenkins, you need to copy the template that comes with the plug-in to the email-template under your jenkins workspace directory (which does not exist by default and needs to be created by yourself ).
Easy-to-access reports
The result of the tests after spoonexecution is collected to the spoon-outputdirectory and can be accessed through the index.html static page. Therefore, you can deploy a Web Container such as tomcat to conveniently access each test result. However, considering the maintenance and spoon-output are static, there is no need to create another container. jenkins has a userContent directory, all files stored in this directory can be directly accessed through jenkins. Therefore, we can automatically copy the result file generated after each test to userContent and store it according to the id of each build. Example: http: // 192.168.10.111: 8080/jenkins/userContent/spoon/162/spoon-output/index.html
In this way, after each test is completed, you can directly access the test result through the link in the email when you receive a feedback email.
Summary:So far, spoon + robobench + jenkins's automated continuous regression testing can meet basic core requirements. When the project checks in every day, we use these automated test cases to perform regression tests on multiple mobile phones. After the test is completed, we receive an email. If there is any problem, visit the test result details page, check whether a bug exists based on the current running logs. If any, immediately report the bug fix. In this way, we can complete the regression test and compatibility test on multiple mobile phones at the same time. The more mobile phones, the more benefits they will be. Of course, automated continuous regression testing is only part of improving the "quality" and "quantity". For the testing team, there are many other aspects such as interface testing automation, testing environment O & M automation, and performance monitoring. How to improve project efficiency is an eternal topic. for traditional testing teams that focus more on quality, it will become increasingly difficult for enterprises to pursue efficiency in the new era, presumably this is also the essential motivation for the birth of google's engineering productivity team. Vision: for the software industry, the current automated testing is still relatively preliminary, but as the litter brother invented the plane, it can only fly a small distance, however, after years of continuous efforts and efforts from several generations, flight tools have brought huge benefits to humans. Even if we look closer, the automation of the manufacturing industry has changed the pipeline of many industries into automation, which greatly improves the productivity of the manufacturing industry. Therefore, we believe that the automation of the software industry will greatly improve the productivity of software projects.