1
, IntroductionRobust testing is a fault-tolerant test of the function and system of each module, which detects whether the software module can maintain its normal operation under abnormal input and harsh environment conditions, and includes error data processing, exception handling and illegal operation processing. Robust testing greatly improves the error coverage, and the test terminal must meet the requirements of the test specification, but also have higher maturity, fault tolerance and ease of recovery, so as to better improve the software quality.
2
, testing purposesTo ensure that the terminal software in the processing of error data and abnormal problems, the functional modules work properly, improve the fault tolerance of the terminal software. The purpose and basis of the anomaly testing are as follows, and our previous test cases are validating these three features: maturity: the ability of the terminal software to avoid failure due to errors in the software: the ability of the software to maintain the specified performance level in case of incorrect data or illegal operation Recoverability: In the event of a failure, the terminal software rebuilds the specified performance level and restores the ability to directly affect the data
3
, Test principleThe purpose of robust testing is to observe the robustness of the terminal software. It is critical to the survival of the terminal software in exceptional and dangerous situations. For example, the terminal software in the input error, network anomalies or illegal operation, can not crash, not collapse, testing and improve the terminal software fault tolerance, to ensure that user data will not be compromised and lost. If the terminal software can not handle the wrong input, it may result in: garbage data into the terminal software, affecting the follow-up; Because the terminal software can not control the running process, the terminal software may be in an unknown state, run unstable situation, or error state, affect the normal business, also may occur security issues, Make illegal users gain benefits, or terminal software can not provide normal service. Therefore, robustness testing is absolutely necessary, but less than the normal operation of the test priority.
3.1
Error Data processingError data processing Test principle is based on the specification definition of manual input of the errors to test, check the test terminal related function module fault tolerance, in the case of input illegal data module function is abnormal. Error data is a collection of meaningless, unreasonable input data, as required. Error data includes unsupported characters, unsupported files, error numbers (passwords, phone calls, etc.), blank data, duplicate data, error settings, out-of-bounds data, and more. To judge whether the information is correct, first of all to conform to the norms, and secondly to be friendly, reasonable, easy to understand, such a hint can be accepted by the user.
3.2
Handling of exceptionsException handling testing principle is based on the definition of the exception handling part of the specification of non-human factors caused by the anomaly test, check the test terminal related function module retry mechanism and automatic recovery ability. Non-human factors include: network anomalies, server anomalies, terminal software anomalies and so on. To judge whether the information is correct, first of all to conform to the norms, and secondly to be friendly, reasonable, easy to understand, such a hint can be accepted by the user. After the abnormal situation occurs, the terminal software automatically initiates the retry mechanism, and the terminal software recovers automatically after the exception condition disappears.
3.3
Illegal operation processingThe principle of illegal operation processing test is to test the abnormality caused by the illegal operation according to the specification definition, check the recovery ability of the related function module of the test terminal, whether there is data loss, check the integrality of the data and so on. Illegal Operation example: when editing MMS, the processing of power-down, SD card data use Plug and unplug SD card, data transmission when the USB cable. To judge whether the information is correct, first of all to conform to the norms, and secondly to be friendly, reasonable, easy to understand, such a hint can be accepted by the user. After the illegal operation can not generate garbage data, no data loss, interrupted operation did not cause terminal software anomalies. After the abnormal condition disappears, the terminal software can recover automatically.
4
, test methodsRobustness testing methods include error data processing, exception handling, and illegal handling of test methods for three types of tests, and the test methods for these three types of test content are as follows
4.1
Error Data processingThe test method of error data processing is to input illegal data to the specified module, and check the response and prompt information of the terminal software is normal. The specific test method is as follows: 1) According to the test code to determine which values are illegal data, 2) to enter the corresponding illegal data for testing; 3) Check the test results. Usually the test results will automatically adjust the data and give the correct information; Note: A specific test device or test environment is generally not required. The specific steps for error data processing are listed below: Requirements: The duration of a single-page MMS message is 1~999 readiness condition: 0 for illegal data test steps: 1) Create a new MMS 2) to write a single-page multimedia message. MMS content including picture and Text 3) set duration to 0 test results: 1) New MMS Success 2) MMS writing success 3) Setup failed with popup warning dialog box
5
, abnormal situation handlingException handling is the test of non-human factors caused by the exception, check the test terminal on the abnormal situation of normal processing. Exceptions include network anomalies, server anomalies, terminal software anomalies, and so on.
5.1
Network ExceptionsNetwork anomalies mainly include: Network congestion/network interference, 2G/3G network switching, circuit domain/Data domain business conflict, communication network is not available and other network anomalies. The specific test method is as follows: 1) using a shielding box to simulate a broken or weak signal, or to set the terminal software network and the actual network does not match the situation, 2) test the corresponding network-related functions. For example: The terminal software Network is set to "TD only" case in the absence of TD signal area to test. Requires that the terminal software does not generate garbage data when a network exception is generated and can continue operations that were previously interrupted when the network is available.
5.2
Server ExceptionSimulates a server exception by using a test platform. Test as far as possible with the current network card through the current network WAP Gateway test. When some test items are unable to complete the test with the current network environment, the lab card and the lab network environment are used for testing. The specific test method is as follows: 1) Modify the data on the test platform; UA modification Tool (Uatools): Modify the error UA to see the push capability of push information fault tolerance 2) test the relevant data.
5.3
Terminal Software ExceptionsTerminal software anomaly mainly test low-power situation, the use of about 15% of the battery power into each module to test, check the battery under the condition of the module, low-power alert should not have an adverse effect on the operation in progress. The specific test method is as follows: 1) when the battery charge is about 15%, the relevant module is tested, and 2) the power is lower than 10%, there will be a low power alert, reminding you not to interfere with the operation in progress. 3) power below 0% after the terminal software automatically shut down, check whether the shutdown will lead to data loss problems, and then see if you can continue to charge the problem. For example: When shooting a video, the terminal software automatically exits the camera when the charge is less than 15%, and the terminal software prompts "low power, please recharge after use", the video has been taken automatically saved to the specified path. The specific steps for exception handling are listed below: Test steps: 1) the terminal receives MMS. 2) shutdown after receiving the push message from the terminal being tested. 3) 2 days after the boot, the received push notification message to extract the download. Test Result: 1) receiving terminal MMS message push messages successfully 2) The terminal is shut down. 3) Prompt the user that the message has expired.
6
, illegal operation processingIllegal operation processing is the basic functions of the module at the same time as human factors caused by the terminal software abnormal operation, including the removal of batteries, but also in the execution of illegal operations, as a step to modify the deletion of synchronized data, large-capacity data transfer to modify the deleted data, Terminal software upgrade by external factors interference check the next terminal software upgrade and so on. For illegal operation terminal Software should not produce garbage data, and can save the edited data, ensure no data loss. Commonly required test equipment are: SD card, SIM card, USB cable and so on. The specific test methods are as follows: 1) Establish the necessary test environment or preset conditions, 2) perform illegal operations, 3) Check the test results, should not produce garbage data, and can save the edited data, to ensure that no data loss. The specific steps for exception handling are listed as follows: Test steps: 1) Enter information 2) new MMS, add picture and Text 3) unplug the battery 4) Reboot 5) Access information View Test results: 1) Enter information Success 2) new MMS success 3) terminal software shutdown 4) boot success 5) Edited MMS is automatically saved in the Drafts box
7
, test case Design
7.1
Error data processing case DesignError data processing test Case design method is based on test specifications and requirements, analysis of terminal software prone to abnormal data, such as unsupported characters, unsupported files, blank data, duplicate data, error settings, cross-border data and other information, based on error data analysis results design test cases. For example: Browser error data "Blank bookmark name/address, blank home address, duplicate bookmark name/address, wrong network settings, error password and so on." ”
7.1.1
analysis principles for error dataThe first step is to analyze the exception data classification according to the reasonable data required in the specification (that is, a set of meaningful data), and then design the test case around the exception data (that is, an unreasonable number of data sets that are not meaningful). The error data can be analyzed by combining equivalence class and boundary value test method.
7.1.2
examples of data typesRequired field input test: The test specification indicates the fields on the screen that must be entered and each of the fields on the screen that are described as must be entered to ensure that you are forced to enter data in the field. The error data test should consider whitespace if there are no prompts and subsequent actions to enter the relevant data; Special field type test: Test Case for the field of the Special data entry requirement (name, date, phone number, zip code, etc.) specified in the test specification. The data entered in the error data test should include data types that it should not accept, test the software for incorrect input prompts and follow-up, such as passwords that only support numbers and English, and should not support Chinese. Special Character type test: A field that is required by a test specification to support characters, error data test input data should be focused on it does not support character sets, test software for error input prompts and subsequent operations; For example: folder name illegal characters/\: *? "< > |. Fetion nickname cannot be guest, friend name cannot be stranger and so on. File type testing: The error data test considers the support of files other than the file types specified in the specification, as well as the hints and follow-up actions given by the test software to handle unsupported file types; Data validation: Enter incorrect URLs, usernames, passwords, or phone errors, The test software handles the prompts and subsequent actions given by the error data, for example, entering a URL that does not exist in the browser, unlocking the input error password, and so on. Example of error data processing test case: Initial Condition:
MMS settings has been set properly
Key Test areas of concentration:9 i* ' 3 o+} ' ~% v:g, R V
1. Launch Messaging/u& i! p9 w# F7 l; N4 S. v* \
2. Compose message$ L6 @-w e% l
3. Attach a Nonsupport media
4. Send MMS Scenario Result
1. Show Conversation List
2. MMS is added
3. The Nonsupport media is added
4. Successfully send
7.2
exception case processing use case designThe design method of the exception handling test case is to design the test case of non-human factors such as network anomaly, server anomaly and so on in the test environment. The client should be handled correctly, and the terminal software should give a clear reason for the error according to the specification requirements, and some modules start the automatic retry and recovery mechanism after the exception condition disappears. First of all, the non-human factors generated by the type of abnormal conditions, including server anomalies, network anomalies, terminal software anomalies and other situations. The test case is then designed around the exception type.
7.2.1
Server Exception TestThrough the artificial means, the terminal software interacts with the data between the server, the test server simulates the abnormal state, check the reaction of the software and the remedial prompt or operation after. This includes server timeouts, server response errors, and so on. The server time-out test case includes all server-related modules, and when the terminal software makes a request to the server and discovers that the server does not return a normal response, the retry mechanism should be initiated. For example: Download MMS Timeout, login fetion timeout, update dynamic content timeout, and so on. The server response error test case includes all server-related modules, and the terminal software initiates a retry mechanism when the terminal software receives the packet content or format error from the server. For example: Dynamic content Download response error or sync corresponding error, etc.
7.2.2
Network anomaly TestingThrough the artificial means to adjust the network signal strength, network conflict and other conditions, detection of the terminal software response and subsequent remedial tips and operations. Including network is in use, circuit domain/Data domain business conflict, communication network is not available and so on. The network is using the test case to include all network-related modules, after the specified 2G network module to the network after the request, found that the GPRS PDP is being occupied by other applications, and the application uses different network connection parameters (APN or proxy), then start the retry mechanism. After the module sends a request to the network under the specified 3G network, the terminal software Multi-APN concurrent processing should be supported. Circuit domain/Data Domain Business Conflict test cases include all network-related modules, in the 2G network terminal software in the use of data domain business connection, the terminal software receives the circuit domain voice call, the client stops the data domain business connection, to the circuit domain after the voice call to restart the data domain business connection. For example: When downloading files or synchronizing data, entering the phone, etc. In the 3G network, the terminal software supports the processing of circuit domain and data domain service concurrency. Communication Network unavailable test cases include all network-related modules, specifying that the module sends a request to the network and discovers that GPRS is not available, then initiates the retry mechanism. For example: Flight mode into the browser, no signal when the custom terminal software TV, etc.
7.2.3
Terminal software anomaly testingCheck the operation under low power conditions. Including the operation of low-power, low power after entering the specified module and other tests. Low-power test cases need to cover all functional modules, the terminal software under low power conditions for the basic functions of the operation, check the low power alert and low power status of the impact of the module. For example: Low-power calls, low-power photos, low-power listening to music, low-power watch video and so on. Exception handling test cases are listed as follows: Initial Condition:! B0 l-x* P (u# J!) {
1. MMS settings has been set properly
2. MMS has expired in multimedia Message Center9 M1 C0 E8 p$ G5 g/k9 h:j
3. The Auto-retrieve is active Key Test areas of concentration:7 M5 w u+ m$ K
1. Receive MMS
2. Manual Retrieve MMS7 E/e5 * \, h) |
3. View this notification Scenario Result
1. Receive a notification
2. Receive a prompt that MMS have expired"w$ D" s ' b-w+ V
3. Successfully view notification
7.3
illegal operation processing use case designThe design method of illegal operation processing test cases is mainly from the aspects of demand, related business and so on. For different functional modules to analyze interoperable illegal operations, the most common illegal operation is to unplug the battery test, transfer and download files, Plug and unplug USB data cable, etc. Illegal operation of unplug battery: main test When an application is running, unplug the battery, interrupt the ongoing operation, detect the response of the terminal software and subsequent remedial tips and actions. Illegal operation of Plug and unplug SD card: Main test function module interacts with SD card when inserting and unplugging SD card, or removing SD card after terminal software to carry out operation related to SD card, detecting the reaction of terminal software and subsequent remedial prompt and operation. Illegal operation of plug-in SIM card: including Test terminal software and SIM card for data interaction with the SIM card, or remove SIM after the terminal software for SIM card or network-related operations, detection of the terminal software response and subsequent remedial tips and actions. Illegal operation of Plug and unplug USB: Test terminal software to remove USB when transmitting data with external device, or connect USB when terminal software runs the application and change the corresponding connection mode, detect the reaction of terminal software and the remedial prompt and operation after. For the illegal operation of the terminal software upgrade: Test in the terminal software upgrade by external factors interference, upgrade the terminal software normal interruption, check the terminal software can start the recovery mechanism, the terminal software to revert to the non-upgraded version and normal use, and the next terminal software upgrade has no impact. The illegal operation test cases are listed as follows: Initial Condition:' H. Z: {) {-H7}! Y6 D
1. There is at least one push e-mail accountKey Test areas of concentration:' [, T6 M8 o/@ (H! [
1. Power off when sending an email9 L. s& J6 N n+ C
2. Power on and then go on sending the e-mail from Outbox/B (J8 L0]5 H, X
3. Take battery off when receiving emails: O9 m% Q. ~) X2 B. | 5 c) N
4. Restart phone and then go on receiving the emailsScenario Result
1. Verify that the unsent email should is put into Outbox
2. Verify that the email should is sent out.+ 9 N2 b/r# F (v& o ' t3 R; B
4. Verify that received emails should not being received again,unreceived email should be received this time
What is robustness testing
