Expand the JUnit test parallel program

Gan Zhi
Ganzhi@cn.ibm.com
), Senior Software Engineer, IBM China Software Development Center
Dai Xiaojun
Daixiaoj@cn.ibm.com
), Software engineer, IBM China Software Development Center
Qi Yao
Qiyaoj@cn.ibm.com
), Software engineer, IBM China Software Development Center
Luo Zhida
Luozd@cn.ibm.com
), Software engineer, IBM China Software Development Center

 

December 19, 2008

TDD (test driven
Development) has already become a widely used agile practice. For Java, JUnit
It is a unit test tool favored by developers. It even affects testing frameworks in other languages. From the perspective of xunit
You can see the framework. With the advent of the multi-core era, developers will need to conduct more and more parallel program tests. Thanks to the scalability of JUnit, we can use
Annotation allows JUnit to better support parallel testing. In this article, we will introduce a pair
So that we can easily perform unit tests in parallel.

What is the difference between testing parallel programs and the past?

With
With the popularization of multiple cores, parallel program development has been put on the agenda. Compared with serial programs, parallel programs are more likely to encounter errors. On the one hand, the execution sequence of parallel programs is very random, and the thread-staggered execution sequence
It may be different each time, and the entire program is incorrect as long as there is a problem with a sequence. On the other hand, parallel programs are a new field for most programmers and have little experience, which is prone to errors.
Another factor.

In this case, we need to carefully test our parallel programs and components. Currently, some JUnit instances are available.
Extension can create multiple threads and run multiple test cases at the same time to speed up the execution of test case sets, such as P-Unit
. However, for testing parallel programs and components, these functions cannot meet all requirements. Because developers usually want to precisely control the synchronization between multiple threads.

Similar to the test sequence program, we hope to prepare some test data before testing parallel operations. Then, start multiple threads to test and perform different operations. Finally, wait for all threads to end and verify the correctness of the results. In the second stage, multiple threads can be staggered in any order. The correctness check of the results should not be related to the execution of the thread sequencing.

Mark
JUnit only captures exceptions of autonomous threads. Exceptions generated by other threads
It will be quietly ignored, so that we can still get the "green bar" when the sub-thread fails to run.
". This is obviously not a test behavior that programmers like. We hope that the test results will correctly reflect the running results of all threads.

This testing mode for parallel programs will be tested and
The row program will be repeated constantly. If developers need to re-create these frameworks every time, it is not only tedious, but also easy to introduce errors. The following simple extension can be used to test parallel programs and
Ordered programs are as simple as those. This extension does not affect other features of JUnit and the use of JUnit plug-ins for various ides.

Download and use the extension framework

First, we provide an example of using the new extension for parallel testing.

Example 1. Use JUnit extension for parallel testing

/** * @author Zhi Gan * */ @RunWith(Parallelized.class) @ParallelSetting(threadNumber = { 1, 2, 4, 8 }) public class TestThreaded { Set<String> strSet; @Before public void setUp() { strSet = new LockFreeSet(); } @Test public void doNothing() { } @InitFor("testThread") public void putSomeData(int size){ strSet.add("putSomeData"); } @Threadedpublic void testThread(int rank, int size) { // every thread adds element to set strSet.add("abcde" + rank); } @CheckFor("testThread") public void checkResult(int size) { assertEquals(size+1, strSet.size()); } public static void main(String[] args) { for (int i = 0; i < 10; i++) JUnitCore.runClasses(TestThreaded.class); } }

 

If we run the test in eclipse, the JUnit view after the test is completed is as follows:

Figure 1. Completion result of parallel test

Next, we simulate the subthread to throw an exception at runtime.

Example 2. Exceptions during sub-thread running

/** * @author Zhi Gan * */ @RunWith(Parallelized.class) @ParallelSetting(threadNumber = { 1, 2, 4, 8 }) public class TestThreaded { Set<String> strSet; @Before public void setUp() { strSet = new LockFreeSet(); } @Test public void doNothing() { } @InitFor("testThread") public void putSomeData(int size){ strSet.add("putSomeData"); } @Threadedpublic void testThread(int rank, int size) { // throw a runtime error in spawned thread throw new RuntimeError(); } @CheckFor("testThread") public void checkResult(int size) { assertEquals(size+1, strSet.size()); } public static void main(String[] args) { for (int i = 0; i < 10; i++) JUnitCore.runClasses(TestThreaded.class); } }

 

If we run the test in eclipse, the JUnit view after the test is completed is as follows:

Figure 2. Completion result of parallel test

We can see that our parallel test cases have passed the test, and they can work normally when running in different threads. The execution results of the serial test method (donothing) work completely as before. That is to say, serial and parallel tests can appear in a test class at the same time.

Annotation

Table 1. Extended annotation description

Name	Parameters	Objects used	Remarks
Parallelsetting	Threadnum: used to specify the number of threads. We usually want to use an array to specify multiple values. In this way, we can check whether the program works normally in a single thread, with fewer threads, and with a large number of threads.	Used for the entire testcase to specify the parallel settings used for testing	Used to Specify parallel settings for the entire testcase
Initfor	Specify the testing method of this method Service	The parallel test method used to specify the service of the initialization method
Threaded	No parameter, or define a parameter compatible with @ Test	This is a test method with two int types. During the test, the testing framework passes the thread serial number and total number through the method parameters. This is similar to the MPI convention.	Specify a method as a parallel test method, which is equivalent to the original @ test comment of JUnit.
Checkfor	A string parameter is used to specify the parallel test method to be verified.	A threaded modification method	Specify a method to detect the results of parallel execution. We cannot directly check the threaded method, because the testing of other threads may not be completed yet.

Description of JUnit extension process

Gui
Thanks to the flexible internal architecture of JUnit, As long as JUnit standards are followed, we can easily expand JUnit
. Compliance with standards also means that our scaling can seamlessly leverage the extensive community support for JUnit. For example, we have not compiled any eclipse
Plug-ins, but our test results can be naturally presented in eclipse through a well-designed GUI.

To put it bluntly, the process of expanding JUnit mainly consists of the following processes:

1. Generate annotation definitions, including: @ threaded, @ initfor, @ check, @ parallelsetting
2. Generate the testclassrunner subclass parallreceived and implement the logic for running the custom test.
3. Generate a subclass of testmethodrunner for use by the parallreceived class.

In
When threadedmethodrunner is implemented, we first use
Setdefaultuncaughtexceptionhandler to capture exceptions. Then encapsulate the exception to the main thread. The current version uses
Executor. Because the future in JDK already provides similar capabilities, we do not need to worry about the correct transmission of exceptions.
JUnit can accurately print the exception information generated in the parallel test, which also means that we can use the assert function provided by JUnit.

Conclusion

With
As multi-core platforms gradually become mainstream, developers inevitably need to develop and test parallel applications. This article uses annotation Extension
JUnit makes it easier to support the three-phase parallel test mode of "prepare data -- multi-thread running -- check result", reducing the tedious work of developers to manually create threads and synchronize. In addition
JUnit supports capturing test errors from child threads and correctly displaying test results in eclipse and other ides.

References

Learning

• Understanding P-Unit: an open-source Performance Testing Tool
  "(Developerworks, May 2007): P-Unit is an open-source performance testing framework. Unlike JUnit, P-Unit not only focuses on the correctness of test cases, the performance parameters of the test case are also collected.
• "Use contest for multi-thread Unit Testing
  "(Developerworks, May 2006): This article explains why parallel bugs are so hard to capture and introduces a new solution from IBM research.
• In the author's blog
  There are more parallel articles.
• Download the multi-thread correctness analysis tool from alphaWorks.
  To check for potential deadlock and data race errors in the program.
• Use "contest" from the IBM Haifa Experiment
  "Faster testing of parallel-related bugs.
• P-Unit
  : Open-source unit testing and performance testing framework

Obtain products and technologies

• Download JUnit parallel test Extension from SourceForge
  Complete code.
• Open-source projects from amino
  Download more parallel components from the website.

Author Profile

		Gan Zhi, IBM China Software Lab (csdl bj) China emerging technology institute Member, mainly engaged in parallel computing, UML, Model Driven Development, and SOA . He obtained a doctorate degree in network security from the computer department of Shanghai Jiao Tong University, during which he published many papers and technical books. You can use The http://ganzhi.blogspot.com or ganzhi@cn.ibm.com contacts him.

 

		Dai Xiaojun, IBM China Software Lab (csdl bj) China emerging technology institute Member, mainly for concurrent programming and agile software development methods. He earned a master's degree in Computer Software and theory from the Institute of software, Chinese Emy of sciences. You can use The daixiaoj@cn.ibm.com contacted him.

 

		Qi Yao, a member of the IBM China Software Lab (csdl bj) China Emerging Technology Institute, is mainly engaged in dynamic program analysis and static program analysis. He earned a master's degree from Beijing Institute of Technology. You can contact him via qiyao@cn.ibm.com.

 

		Luo Zhida, a member of the IBM China Software Lab (csdl bj) China Emerging Technology Institute, specializes in dynamic program analysis and static program analysis. He earned a master's degree in software engineering from Peking University. You can go through the luozd@cn. I