Understanding P-Unit: an open-source Performance Testing Tool

Understanding P-Unit: an open-source Performance Testing Tool

Level: elementary Zhang Huang Qianwei, Software Engineer May 31, 2007 P-Unit is an open-source performance testing framework. Unlike JUnit, JUnit focuses on the correctness of test cases, while P-Unit focuses not only on the correctness of test cases, the performance parameters of the test case are also collected. By default, P-Unit collects the time and memory consumption of the test case, and reports in file, image, and PDF format can be generated. In addition, P-Unit also supports parameter testing, multi-thread testing, and comparison of performance between different Java virtual machines. Introduction to P-Unit Maybe we are used to using JUnit to write unit tests to ensure code quality (I have always done this), but we may often encounter the following problems: What is the correctness of multi-thread programs? How to test program performance? How can we compare the performance of different solutions technically when multiple solutions are available? For question 1, we may be resigned? Or does it rely on manual analysis or user feedback? Many software single-threaded unit tests have a high coverage rate, thus ensuring code robustness. However, multithreading testing is often ignored, which does not mean that multithreading testing is not important. On the contrary, fixing a user-reported multithreading bug is often much higher than that of a single thread, because the test cases are often not 100% reproducible. This requires programmers to fully pay attention to the development stage. One important reason why multi-thread unit testing is insufficient is that there is not a testing tool that is as easy to use as JUnit, and repeated test cases are often not accepted by programmers. For question 2, a mature performance-oriented product often has a performance testing platform. This test platform should focus on the test business logic itself, without the need to care about how to run test cases. Are you suffering from writing such a test platform? And spend time generating some intuitive reports? For question 3, we often write a prototype to compare the performance between different products. How can we compare the execution speed and memory consumption? Or choose the most suitable Virtual Machine? P-Unit is such an open-source performance testing software that can help you solve the above problems. P-Unit can: Multithreading support: the same test case can be executed in a single thread or multiple threads. the test case developer only needs to write a set of test cases. Parameterized test cases: many test cases require testing the performance of the same function on different orders of magnitude. Performance Tests for different virtual machines: you only need to specify the Virtual Machine path to test the performance of the same test case on different virtual machines. The report shows the performance differences intuitively. Event mechanism architecture: punit is based on the event mechanism architecture. to customize a report, you only need to implement the event responder and register the responder to the punit core.   Multi-thread execution test case Before learning how to run a test case with multiple threads, let's take a look at how to use the P-Unit single thread to execute the test case. Unlike JUnit, the P-Unit test case does not need to inherit any test classes or implement interfaces to execute the method starting with test. Although JUnit 4 has added the Annotation Feature, the prefix of the test method is still the first choice for testers. Therefore, if your JUnit test cases follow the test naming rules, p-uni t can be compatible with running JUnit test cases. The following code list 1 is the most common test case: Listing 1. Test Case 1 public class SimpleTestClass {public void setUp() { SampleUtil.doSomething();}public void tearDown() { SampleUtil.doSomething();}public void testA() { System.out.println("testA"); SampleUtil.doSomething();}public void testB() { SampleUtil.doSomething();}public void testC() { SampleUtil.doSomething();}}public class SampleUtil { private static Random _random = new Random(); public static void consumeMemory(int length) {byte[] data = new byte[length];for(int i = 0, j = 0; i < data.length; ++i) { ++j;}}public static void consumeTime(int time) {ThreadUtil.sleepIgnoreInterruption(time);}public static void doSomething() {consumeTime(Math.abs(_random.nextInt()) % 500);consumeMemory(Math.abs(_random.nextInt()) % 100000);}} This is a common test case, but it is noted that this is only a test case and does not contain any other logic. This is also the idea of separating the business logic pursued by P-Unit from the test running environment. In the same test case, you can choose different test environments to run, instead of binding them to a specific test software tool. Now let's take a look at how P-Unit runs this test case. You only needmainWrite a line of code in the function to run it: List 2. Single-thread running test cases public static void main(String[] args) {new PUnitSoloRunner().run(SimpleTestClass.class);} Listing 3. Results of a Single-thread running test case [solo] Started running samples.SimpleTestClasssamples.SimpleTestClasstestAtestA() - [287.0ms]testB() - [27.0ms]testC() - [213.0ms]total: 3, failures:0 (GREEN) - 2025.0ms Is it the same as imagined? Next we will look at how to run this test case with multiple threads. You may have guessed from the above example:mainThe function still requires only one sentence of code.PUnitSoloRunnerChangePUnitConcurrentRunnerYou can! Listing 4. multi-thread running test cases public static void main(String[] args) {new PUnitConcurrentRunner().run(SimpleTestClass.class);} Listing 5. Results of multi-threaded Running Test Cases [concurrent] Started running samples.SimpleTestClasssamples.SimpleTestClasstestAtestAtestAtestAtestAtestAtestAtestAtestAtestAtestA() - [405.0ms]testB() - [469.0ms]testC() - [503.0ms]total: 3, failures:0 (GREEN) - 1447.0ms Is it the same as imagined? By default, P-Unit starts 10 threads for execution. to specify the number of threads, you only need to input the number of threads as a parameter.PUnitConcurrentRunnerYou can. P-Unit even supports different test cases with different threads, which requires the test case to implementConcurrentInterface, which is defined: Listing 6. P-Unit concurrent Interface public interface Concurrent {public int concurrentCount();} I believe there is no need to explain the meaning of this interface. The number of threads required for this test case is returned. Parameterized test case Performance testing, unlike unit testing, often requires tests of different magnitude in the same test scenario. JUnit is a very good unit testing tool, but it does not cover this aspect. For example, we compare class libraries.Foo1Methodbar()And class libraryFoo2Methodbar()Which is more suitable for our own application, we need to test the performance of this function within the range of the application's possible magnitude. Experienced developers know that small orders of magnitude A is better than B is better. Therefore, comprehensive testing is very important to understand the code performance and can help developers make correct decisions. P-Unit supports passing parameters to the test method.parameterizableInterface, the main method of this interface is to return a list of parameters, the parameters of this list will be passed to the test method one by one. Listing 7. P-Unit parameterized test case public class ParamTestClass implements Parameterizable {public static void main(String[] args) {new PUnitSoloRunner().run(ParamTestClass.class);}public Parameter[] parameters() {return new Parameter[] { new ParameterImpl(10), new ParameterImpl(20) };}public void testA(ParameterImpl param) {SampleUtil.doSomething();}public void testB(ParameterImpl param) {SampleUtil.doSomething();}public void testC(ParameterImpl param) {SampleUtil.doSomething();}public void setUpAfterWatchers(Parameter param) throws Exception {}public void setUpBeforeWatchers(Parameter param) throws Exception {}public void tearDownAfterWatchers(Parameter param) throws Exception {}public void tearDownBeforeWatchers(Parameter param) throws Exception {}static class ParameterImpl implements Parameter {private int _count;ParameterImpl(int count) {_count = count;}public int count() {return _count;}public String toString() {return String.valueOf(_count);}}} The execution result of the above Code is: Listing 8. P-Unit parameterized test case output [solo] Started running samples.ParamTestClasssamples.ParamTestClasstestA(10) - [57936.0bytes,447.0ms]testA(20) - [33128.0bytes,61.0ms]testB(10) - [24832.0bytes,137.0ms]testB(20) - [0.0bytes,63.0ms]testC(10) - [83560.0bytes,468.0ms]testC(20) - [16528.0bytes,47.0ms]total: 6, failures:0 (GREEN) 1450.0ms The above results show that each method is executed twice and different parameters are input each time. How can I run a parameterized test program in multiple threads? I believe that the reader has understood how to implement it. You only need to replace punitsolorunner with punitconcurrentrunner. Test Cases of running environment With Java open-source, more Java runtime environments have emerged. Apart from Sun's Reference implementations, Bea and IBM have their own Java runtime environments, for example, Apache Harmony's open-source Runtime Environment (although Apache harmony is not currently called a Java Runtime Environment ). Running environment test cases can help developers in the running environment and select the running environment. For example, the following example is a test.java.util.ArrayListAndjava.util.VectorPerformance in two different runtime environments. The test case writing method is exactly the same as the normal test case. We only need to tell P-Unit the Java path of different runtime environments and the correct classpath, and then callrunVMsFunction: Listing 9. Test Cases of P-Unit running environment public static void main(String[] args) {PUnitSoloRunner runner = new PUnitSoloRunner();runner.addPUnitEventListener(new OverviewReporter(new ImageRender()));runner.runVMs(ListTestClass.class, new VM[] { VMConfig.HARMONY, VMConfig.SUN });}public class VMConfig {private static String CLASSPATH = " -cp correct_classpath_including_all_jars_and_path"; |-------10--------20--------30--------40--------50--------60--------70--------80--------9||-------- XML error: The previous line is longer than the max of 90 characters ---------|private static String HARMONY_PATH = "harmony_path//bin//java" + CLASSPATH; private static String SUN_PATH = "sun_path//bin//java" + CLASSPATH; public static VM HARMONY = new VM(HARMONY_PATH, "HARMONY"); public static VM SUN = new VM(SUN_PATH, "SUN");}public class ListTestClass { private static final int LIST_COUNT = 100000; private static Object element = new Object();private Random indexGenerator = new Random();; public void testInsertArrayList() { ArrayList arrayList = new ArrayList(LIST_COUNT); insertSequence(arrayList); insertRandom(arrayList); } public void testInsertVector() { Vector vector = new Vector(LIST_COUNT); insertSequence(vector); insertRandom(vector); } public void insertSequence(List list) { for (int i = 0; i < LIST_COUNT; ++i) { list.add(element); } } public void insertRandom(List list) { for (int i = 0; i < LIST_COUNT; ++i) { list.add(indexGenerator .nextInt(LIST_COUNT),element); } } } The running result of the above Code is as follows: We can intuitively see that the harmony version used by the author is faster in this test case (picture on the left), but consumes more memory (picture on the right ). The next section describes how to output a report, but you may have noticed that the code is very simple. From the above example, we can see two forms of P-Unit output results: console and report image. By default, P-Unit is output to the console. P-Unit uses the event mechanism, which provides notification events on each node of the runtime. All outputs are implemented by registering the event responder. This also indicates that the result output is completely isolated from the runner, and you can customize your own reports. P-Unit has four built-in outputs: console, file, image report, and PDF report. In the previous example, we have seen the image report. The code is: runner.addPUnitEventListener(new OverviewReporter(new ImageRender())); The built-in reports of P-Unit have three different granularities: overviewreporter, testsutie, and testclassreporter ). These three levels can be output in either the image format or PDF format. Therefore, there are a total of six types of output. The above code is to output the overall picture. Because event listeners are independent from each other, you can select both the output image and the output PDF file. You only need to add the event listener: runner.addPUnitEventListener(new OverviewReporter(new ImageRender()));runner.addPUnitEventListener(new OverviewReporter(new PDFRender()));   So far, have you basically understood the concept of P-Unit? Simple, easy to use, focus on multithreading, focus on performance, this is P-Unit. In addition, P-Unit has many small features, such as the alphabetical interface, to ensure the sequence of execution of test functions. Use P-Unit to make your code more robust!