Understanding of testing

We will hear many terms about testing at work, such as ut, integration test, and end-to-end test.

In my opinion, most people only have a vague understanding. They are not really clear about the meaning of each test.

When leaders stress that they want to perform the XX test, they will randomly throw the concept to their subordinates. In fact, they do not know why or how to do it.

As far as I know, most Chinese companies are even worse. Every developer is fully engaged in writing code, and is unwilling to spend time writing the most basic unit tests.

At the same time, the importance of testing is not only to ensure quality. In today's popular agile development, testing has become an important part of agile development.

Therefore, do we really understand the meaning of the test. Their necessity, how should they be defined, are there strict boundaries between them, and what are appropriate tools for each test type.

These are all issues that we must think about.

Now, I will discuss common tests. If you have any suggestions or do not agree with them, please click here.

Unit Test

Unit test definition:

A unit test is a test written by the programmer to verify that a relatively small piece of code is doing what it is intended to do. they are narrow in scope, they shocould be easy to write and execute, and their implements tiveness depends on what the programmer
Considers to be useful. the tests are intended for the use of the programmer, they are not directly useful to anybody else, though, if they do their job, testers and users downstream shocould benefit from seeing less bugs.

The above definition emphasizes

1. ut should be written by developers and used by developers.

2. ut testing should be a small piece of code, so that developers can determine that the source code has done what they want it to do.

Let's take a look at how the author of testng summarized ut:

Unit testing (testing a class in isolation of the others)

First, he stressed that an ut case can only be targeted at one class. You should not test more than one class in an ut case.

The second point is that the UT of a class must be completely independent and should not interact with other code. This is also why we need to use mockito and other mock frameworks. In this way, when testing code and interacting with other objects, we will use the mock object to maintain ut independence.

In addition, ut can be automatic or manual.

Unit test tools:

Well-known JUnit and testng;. Net-specific nunit; vs2010 started to provide a test framework directly.

Cppunit for C/C ++ and phpunit for PHP.

Mockito, easymock, and other mock frameworks are also important components.

 

In addition, unit testing has a coverage rate problem. It refers to the number of codes in the tested class called by an ut. The ut written by an irresponsible programmer may only test one method in the class, and the UT written by the responsible programmer should overwrite all the main methods in the test class.

Cobertura is an open-source tool that automatically detects ut coverage and generates reports. It is a good practice to integrate this tool into daily build.

It is unreasonable and meaningless to require the coverage rate to reach 100%, usually 80 ~ 90%. There are two reasons:

 

1. Does UT have to overwrite all the code?

Let's take a look at the JUnit official website's answer:
No, just test everything that cocould reasonably break.

Be practical and maximize your testing investment. remember that investors in testing are equal investors in design. if defects aren't being reported and your design responds well to change, then you're probably testing enough. if you're spending a lot
Of time fixing defects and your design is difficult to grow, you shoshould write more tests.

If something is difficult to test, it's usually an opportunity for a design improvement. look to improve the design so that it's easier to test, and by doing so a better design will usually emerge.

 

2. For general projects, it is basically impossible to reach 100% coverage.

Although we now have various testing frameworks and various mock frameworks, we may encounter code that is difficult or even unable to be tested in many possible cases.

For example, the private method, the only test method is to use the reflection mechanism.

If the test coverage rate becomes a goal, instead of improving the code quality and preventing bugs, it will put the horse upside down.

 

However, modifying the source code for testing is not unreasonable. If you find that the code is difficult to test, there may be some problems with the code.

For example, as mentioned in another article about mockito, In order to pass the mock object to the tested object, the source code must provide the get/set method, otherwise, testing will be very troublesome.

In practice, we will gradually find that the Code is not independent enough, there is a lack of proper get/set methods, the structure is unreasonable, and the methods should be restructured, which will lead to difficulties in testing.

So ut is a review of the source code. A responsible programmer should see more from ut. Always remember one: we are not testing for testing, and we should think more.

 

3. What is independence for each test method?

Recently I encountered a question about ut: How independent is each test method "?

For example, there are two methods in a tested class:

Public class app {

Public void Method1 (){

String teststring = method2 (); // method 2 is called in method 1.

}

Public String method2 (){

}

}

So the question is, when we write the test method method1test of the first method Method1, what should we mock? Should we use mock method 2? Or do you not need mock?

That is to say, when we test the method, we should regard method 2 as an external thing. Through mock method 2, we only test method 1.

Or call method 1 directly without using mock (actually method 2 is also called ). Therefore, the behavior of method 1 and method 2 is tested simultaneously.

I would have preferred the first practice, taking method 1 as a completely independent tested part, and using mock method 2, let method 1 be tested, it is completely unrelated to other classes or methods.

However, my views have changed since I discussed it with my colleagues. It seems that the second approach, both method 1 and method 2 are tested. However, in a class, each method is not completely independent.

Therefore, a member variable exists. The value of a member variable can be passed in each method by nature.

Therefore, the independent ut range is always a class. During testing, when code of other classes is called, mock should be used to isolate the test class from other classes.

During testing, mock is not required if other methods of the same class are called. Because the code in the same class is called.

Integration Test

Definition of integration test:

An integration test is done to demonstrate that different pieces of the system work together. integration tests cover whole applications, and they require much more effort to put together. they usually require resources like database instances and hardware
To be allocated for them. the integration tests do a more convincing job of demonstrating the system works (especially to non-programmers) than a set of unit tests can, at least to the extent the integration test environment resembles production.

Therefore,

1. Generally, the integration test is run in the specified environment.

2. The purpose of the integration test is to test whether each component can work normally. Like ut, integration testing also aims to see if the code works in a "design or expected way.

3. The scope of integration testing is relatively broad: the scope can be small. If multiple classes are involved in an ut case, this can be considered an integration test. From the background to the front-end. Integration tests can be saved med at the unit level or the system level.

Integration testing often involves external components, such as databases, hardware, and networks.

4. Integration testing should usually be completed by dedicated testers.

 

HP's (Mercury) qtp or Borland's silktest can be used for automated integration testing and regression testing.

In today's agile development, a concept becomes more important:Ci (continuous integration)

This article does not intend to discuss CI, because it is a big topic. CI and integration testing are often closely integrated.

CC is one of the commonly used CI tools.

Hudson is another new Ci tool.Http://hudson-ci.org/

 

Smoke test

I like this name very much, and its source is very interesting. Smoke test was developed from an electronic hardware test. It indicates that the first time the prototype of the hardware is powered, if the hardware smoke, it indicates that there is a problem with the product.

From the reason for the name of smoke test, we can see that smoke test is a relatively basic test (a quick test), just to check whether each component can work together, and does not go into the functional correctness.

A simple integration test where we just check that when the system under test is invoked it returns normally and does not blow up.

Note that the smoke test is generally an integrated test of a wide range. Generally, it can be a whole system/end-to-end test.

For example, a project: the customer service personnel enter user information on the terminal, and end users can query their information on the Internet.

In this case, the smoke test can input the information of a user on the terminal, and then query whether the user information can be found through the web program, without worrying about the correctness of the user information.

 

Therefore, some people think it is the first step in the test: First tests on which testers can conclude if they will continue testing. But it is not necessary.

 

Regression test regression test

A test that was written when a bug was fixed. It ensure that this specific bug will not occur again. The full name is "non-regression test ".

A regression test re-runs previous tests against the changed software to ensure that the changes made in the current software do not affect the functionality of the existing software ."

The two concepts above are inconsistent: one is that regression testing is to overwrite the Fix bug, and the other is that regression testing is to overwrite the newly added features.

However, we can see the unification of the two statements:

Regression testing is a test to cover new changes in the system.

Regression testing can be written by testers or developers.

The form can be a ut that overwrites new functions or bugs, or a dedicated testing tool written by testers.

 

End-to-End test

Validates the entire application to ensure that it satisfies previusly established acceptance criteria and performs as an integrated system. The purpose of system testing is not to test all possible
Positive and negative conditions (reserved for functional and integration testing), but to instead execute business functions and infrastructure management (I. E .; batch Processing, system security features, backup and recovery etc .) in an isolated and controlled
Environment to validate that a quality system is ready for production.

Therefore, end-to-end testing emphasizes comprehensive testing, including hardware environments.

The last sentence is very important: it is for the production function.

Because of its wide coverage, it is generally a manual test. Generally, there is no dedicated tool for end-to-end testing.

 

Function test functional test

Functional Tests are related to integration tests, but refer more specifically to tests that test an entire system or application with all of the code running together, almost a super integration test.

Non-functional test

In our company, non-functional testing mainly refers to ancillary functions other than the main functions of the program. For example, GMI, log, report, and Failover are added to monitor program running and program stability.

Acceptance TestsI don't know what Chinese is.

In general,Acceptance TestsAndFunctional TestIs very close, and sometimes the two are considered completely the same. The similarities and differences between the two are:

Functional Testing: This isVerificationActivity; did we build the thing right? Does the software meet the business requirements?

For this type of testing we have test cases that cover all the possible scenarios we can think of, even if that scenario is unlikely to exist "in the real world ". when doing this type of testing, we aim for maximum code coverage. we use any test environment
We can grab at the time, it doesn't have to be "production" caliber, so long as it's usable.

Acceptance testing: This isValidationActivity; did we build the right thing? Is this what the customer really needs?

This is usually done in cooperation with the customer, or by an internal customer proxy (product owner ). for this type of testing we use test cases that cover the typical scenarios under which we recommend CT the software to be used. this test must be conducted
In a "production-like" environment, on hardware that is the same as, or close to, what a customer will use. This is when we test our "ilities ":

• Reliability, availability: Validated via a stress test.

• Scalabilitiy: Validated via a load test.

• Usability: Validated via an inspection and demonstration to the customer. Is the UI configured to their liking? Did we put the customer branding in all the right places? Do we have all the fields/screens they asked?

• Security (Aka, securability, just to fit in): Validated via demonstration. Sometimes a customer will hire an outside firm to do a security audit and/or intrusion testing.

• Maintainability: Validated via demonstration of how we will deliver software updates/patches.

• Scalability: Validated via demonstration of how the customer can modify the system to suit their needs.

In short, functional testing tends to test all the functions of the software. Whether all requirements are satisfied.

Acceptance testing tends to test the user experience as much as possible. Test whether the user's actual needs are met in an environment that is identical to or as similar as possible to the production.

At the same time, both of them should be completely black box tests.

The following sentence is very important:

Acceptance Tests/criteria (in
Agile Software Development) are usually created by business MERs and expressed in Abusiness domain language. these are high-level tests to test the completeness of a user story or stories 'played' during any sprint/iteration.

A
Business Domain language is a natural language that is not understood by developers. Jbehave is a popular framework.

Acceptance Test cards are ideally created during sprint planning or Iteration Planning meeting, before development begins so that the developers have a clear idea of what to develop. Sometimes (due to bad planning !) Acceptance
Tests may span multiple stories (that are not implemented in the same Sprint) and there are different ways to test them out during actual sprints. one popular technique is to mock external interfaces or data to mimic other stories which might not be played
Out during an iteration (as those stories may have been relatively lower business priority). A user story is not considered complete until the acceptance tests have passed.

That is to say, in a sprint, how can we determine whether a developer has completed a function? The standard is that the corresponding acceptance test passes.

This is the reason for acceptance-test driven development.

Summarize the white box and black box tests:

White box testing means that you know the input, you know the inner workings of the mechanic and can inspect it and you know the output.

With black box testing you only know what the input is and what the output shocould be.

 

Therefore, ut is usually a white box test, but sometimes it can also be a black box. For example, in the popular test-driven development, ut is created before the code as needed. At this time, ut only knows what we have and what we want to see, so it is a black box.

Most other tests are black box tests. However, integration testing and regression testing can also be a white box.

 

As mentioned above, the test of source code coverage is theoretically only meaningful for white box testing. For testing with the black box, the code coverage does not make sense at all.

For some leaders, "we need to cover the source code 100% for this integration test", which is basically because the brain is congested.

But it should be noted that, like the black box test, code coverage can be calculated, although I think it makes no sense.

After cobertura injection, you can easily obtain the coverage of source code for any integration test. For details, refer to the introduction of cobertura I wrote.

Finally, let's take a look at the popular open-source mock framework (I pay special attention to UT-related tools because I am a developer)

• Easymock was one of the most popular tools earlier.

Now, easymock has fewer functions due to its early appearance. When easymock is used, it is difficult to use mock.

• Mocktio

A popular mock tool after easymock. The syntax structure is clear and easy to understand. Documentation is comprehensive, so it is easy to find answers to problems due to their popularity.

Mockito is sufficient for most ut mock demands. In general, it is a very good mock tool.

• Powermock

In fact, this tool is extended on easymock and mockito to solve problems that easymock and mockito cannot solve.

In my introduction to powermock, I explained in detail what special mock requirements this tool can be used.

The disadvantage is that the tool has few documents and the examples are incomplete. Problems Encountered during use are not easy to solve. At the same time, powermock must be used with easymock or mockito.

Therefore, easymock or mockito must be mastered first.

• Jmokit

Http://code.google.com/p/jmockit/

Another tool similar to powermock. I am not going to introduce this tool (I recently read the mock tool for Aesthetic Fatigue ).

This tool implements the same functions as powermock. Therefore, you can select this tool if you do not like powermock.

Compared with powermock, this tool documentation is comprehensive and suitable for learning.

In addition, there are many mock tools, such as jmock. There are also some dedicated mock tools, such as mockftpserver.

The Open Source world is very interesting ~