Android compatibility test _cts-Environment setup, test execution, results analysis

To ensure that Android apps work correctly on all Android-compatible devices and maintain a similar user experience, Android provides a set of compatibility test cases at the time each release is released (Compatibility Test Suite, CTS) to certify that the device running the Android system is fully compatible with the Android specification, with the relevant Compatibility standard documentation (compatibility Definition document, CDD).

First download the latest set of compatibility test cases from http://source.android.com/compatibility/downloads.html (Internet needs to be able to enter Google) and unzip. As shown in 1.

Figure 1 The CTS test environment download in the Android website, etc.

Most are written based on JUnit and dashboard technology. It also extends the automated testing process to automate use cases and automatically collect and summarize test results. The CTS uses XML configuration files to divide these test cases into multiple test plans (plan), and third parties can create their own plan.

The preparatory work to be done before the CTS test is as follows:

1. Download the compatibility test case package and extract the extracted file name as "Android-cts". At the bottom of the http://source.android.com/compatibility/downloads.html page, there is a link called "Compitibility Test Suite (CTS) User Manual", Is the latest version of the Android Compatibility test case execution method, it is recommended to read through the document before execution.

2. Brush machine is the version that needs to be tested.

3. When the phone is powered on, if there is a Google account settings, Cancel: Start->not now->next->next->next->finish.

4. Set the phone language to English: Setting->language&input->language->english (states).

5. Insert the SIM card and the external SD card (the SD card needs to be formatted: settings->storage->erase SD card->erase SD card->erase everything).

6. Plug in the USB, connect the phone to the computer, can use the ADB devices check, whether the connection is correct.

7. Turn on WiFi and connect to available WiFi.

8. Turn on Bluetooth without pairing.

9. Make sure the LCD off:settings->display->sleep->30 minutes after 30 minutes without operation.

10. Remove the screen Lock: The value of Settings->security->screen lock is "None".

11. Open settings->location services-> "Google Location Services", "GPS Satellites", "Location & Google search".

12. Open Settings->accessibility->developer Options->usb Debugging (USB Debug).

13. Open Settings->accessibility->developer Options->stay awake (keep awake).

14. Open the Settings->accessibility->developer options->allow mock location (allow analog locations).

15. Install the "Text to Speech" file (com.svox.langpack.installer-1.apk) via Settings->speech synthesis->install voice data If there is no such file in android-cts/repository/testcases/, this step is omitted.

16. If you need to perform compatibility testing on accessibility, install "ctsdelegatingaccessibilityservice.apk" (ADB install–r */android-cts/repository/ TESTCASES/CTSDELEGATINGACCESSIBILITYSERVICE.APK), and will settings->accessibility->delegating Accessibility The service option is turned on. If there is no such file in the M directory, this step is omitted and is generally not available.

17. If you need to perform a device Management compatibility test, install the "ctsdeviceadmin.apk" (ADB install–r */android-cts/repository/testcases/ CTSDEVICEADMIN.APK), and will setting->security->devices administrators-> Android.devicesadmin.cts.CtsDevicesAdmin options are turned on. This file must be in the directory, please look it up, 2.

Figure 2 ctsdeviceadmin.apk

18. If you need to perform a multimedia compatibility test, you need to perform:

1) Download Android-cts-media-x.y.zip from http://source.android.com/compatibility/downloads.html and unzip.

2) into the unpacked folder, and execute bash copy_media.sh, the test required files to copy to the phone memory, if copy failed, may be the phone path is not correct, please use gedit open copy_media.sh file, while ADB The shell enters the phone terminal to see if the phone memory directory matches the directory in the copy_media.sh file. If not consistent, please change the copy_media.sh file, you must ensure that the copy to the phone memory (after the copy can open Gallery for viewing), or it will affect the subsequent android.media and other media-related test package execution, 3-1, figure 3-2 shows.

Figure 3-1 Copy Media

Figure 3-2 Copy Media

19. Make sure the phone is in the home screen, i.e. press the "Home" button.

The CTS test officially begins:

1. Enter the "*/android-cts/tools" directory, perform bash cts-tradefed, first identify the device, then appear cts_host, then prove that you have entered the CTS command line interface, you can enter the CTS related command to perform the CTS test, As shown in 4.

Figure 4 CTS command-line interaction interface

2. To test the default CTS, which includes all packages, you can enter the following command:

Run CTS--plan CTS (this is used for all two run tests, please consult the relevant personnel for specific use)

or run Cts–disable-reboot--plan CTS (during the test run, the phone will not reboot, so it is easy to connect adb logcat), run up first according to the date and time to create the test results folder, and then appear "Start Test run O f XX packages, containing XX tests "explained that the test has started to run, at this time as much as possible to observe more than 10 minutes, the appearance of" Installing prerequisites "and after the show case pass, then ensure that the CTS did start run , shown in 5.

Figure 5 Run CTS

CTS Test Results Analysis:

After the test is finished in the */android-cts/respository/results folder, you will see that the date and time folder is used to save the executed test results, as shown in Figure 6-2.6-1.

Figure 6-1 Results

Figure 6-2 Results

And there is a zip file of the same name that holds the same content. The log is automatically recorded during the test, and the log is automatically saved in the */android-cts/respository/logs folder with the date and time name in the end of the test, as shown in Figure 7-2.7-1.

Figure 7-1 Logs

Figure 7-2 Logs

In the test results folder, all the test results are saved as XML. The test results page is typically divided into "Device information", "Test Summary", "Test Summary by Package", "Test failures (XX)" and "Detailed test Report" And so on four areas. "Device Information" lists the specific hardware and software of the device under test, as well as the functional configuration information, as shown in 8.

Figure 8 Device Information

"Test Summary" lists information such as the CTS version number, the number of status case, and so on, 9.

Figure 9 Test Summary

The "Test Failures (XX)" will record the output when the assertion fails, as shown in 10.

Figure Ten Test Failures

Each test to ensure that the CTS test case all run, with "L R" view, the CTS test is all run, that is, not executed a column value is 0, if the value is not 0, then there is no run end of case, It is possible that the phone freezes or reset causes the ADB to not recognize the device, so the case behind it is not executed state, 11-1, figure 11-2 shows.

Figure 11-1 Not executed

Figure 11-2 Devices offline

At this point need to reconnect the phone, and then use the command "run CTS--continue-session session_id" can continue above the case of not executed, session_id with the command L R can be seen, 12-1, figure 12-2 shows.

Figure 12-1 Viewing session ID

Figure 12-2 Continue not executed case

First, ensure that the entire case is run again, not executed value is 0. After that, the case run in "failed test Cases" is three times, which excludes the stability of the mobile phone system, especially the case fail caused by reset and freeze. The goal is to determine that case fail is due to a problem with the CTS case itself, and not to any other factor. To re-run the fail case, you need to create a new test plan on top of the case where the last run was completed, and then execute the new test plan. New test test Use command "Add Derivedplan--plan plan_name-s sessionid-r [pass/fail/notexecuted]" Add a new plan, and then use the command "run CTS--plan Plan_ Name "runs to test items that are not tested. Such as:

To test all fail entries for SessionID 2, the input command should be:

>add Derivedplan--plan cts_fail_1-s 2-r fail

>run CTS--plan Cts_fail_1//Cts_fail_1 is defined above, you can make your own name.

If the fail is still a lot, it is recommended to do a third time, on the basis of the Cts_fail_1 test plan, new and execute the test plan, if the "L R" to view the Cts_fail_1 test plan SessionID is 3, then execute

>add Derivedplan--plan cts_fail_2-s 3-r fail

>run CTS--plan Cts_fail_3

As shown in 13.

Figure Add Derivedplan

Three times after run, three result folders with date and time names are generated in the */android-cts/respository/results folder, and the three failed test cases are pasted into the Excel table for summarization. and count the number of fail case numbers in each test package.

CTS tests some common commands, 14.

Figure Help

Some common commands related to host:

help:cts Command List

Exit: Exit CTS Terminal

......

Some common commands related to run:

Run CTS--plan test_plan_name: Execute a test plan

Run CTS--package/-p: A package in a separate run CTS test

Run CTS--class/-c [--METHOD/-M]: The class specified by run, or a method specific to a class

Run CTS--continue-session session_id: Continue run to specify a case with a status of not executed on the session

Run CTS [option]--serial/-s device_id: Run CTS on the specified device_id [option]

......

Some common commands related to Java packages:

L/list d/devices: List The status of all connected devices and devices

L/list Packages: List all test packages for CTS

L/list P/plan: List all the CTS test plans

......

Some common commands related to the test plan:

Add Derivedplan--plan plan_name--session/-s session_id–r [pass/fail/notexecuted/timeout]: from the specified session The ID generates a new test plan based on the various states of the case

......

Option-Related commands:

Run CTS--disable-reboot [option]: No need to restart the phone during testing

CTS FAQs Summary

1. If the test results show that Android.media and android.mediastress two packages the case of fail is more, and when viewing details, 14 of the log appears, the media does not copy into the specified directory, this can also be adb Shell into the phone terminal, in the Fail log path to view, whether there is a need for files, I guess there must be no.

Figure Media Fail Log

You will then need to copy the specified file to the directory specified in the fail log, and you can refer to each push file path in the */android-cts-media-1.1/copy_media.sh file, as shown in 15, paying particular attention to the Internal_ SD, which is going to be based on the phone version is likely to change, if changes, update the script according to the changes.

Figure copy_media.sh

2. The entire round of tests runs, and if a case of rerun fail is required, a status of 16 appears when a new test plan is added.

Figure 16 A plan with the same name already exists

Depending on the hint, you can see that the plan exists with the same name, and you can change the name, or delete the plan with the same name in the directory shown in 17.

Figure Map Plan Folder

3. If you need to run a specified number of packages during the test, instead of the full CTS, you can modify the Cts.xml file in the Plan folder, as shown in 18.

Figure 18 Modifying plan

Android compatibility test _cts-Environment setup, test execution, results analysis