In the previous issue of "CLR thorough resolution," I emphasized that to reliably create high-performance programs, you need to understand the performance (msdn2.microsoft.com/magazine/cc424899) of the various components used in the early stages of the design. This requires performance data. Therefore, measurement is an integral part of the design process.
I also introduced a tool called MeasureIt in that period, which makes it easy to create new benchmarks to quickly gather the data needed to make good design decisions. The original figures provided by tools such as MeasureIt are of paramount importance, and it is also important to enable people to understand the basic meaning of these numbers. Based on this understanding, you can predict some of its performance before you actually measure it. That's what I'm going to talk about here.
MeasureIt Overview
If you have not yet downloaded the MeasureIt tool, I strongly recommend that you download it now. The tool is located in the MSDN® magazine website in the download code of this column, composed of an EXE file. Running it will generate a Web page that shows the results of running a benchmark suite. After installation, you can access other documents by running the following command:
Measureit/usersguide
MeasureIt comes along with its source code and can be easily decompressed using the/edit qualifier. This makes adding a new benchmark as easy as writing one or two lines of code (and providing the code to be timed). For more detailed instructions on how to do this, see the user's Guide.
The MeasureIt datum is associated with different performance zones and is indicated on the command line when the tool is started. By default (that is, no command-line arguments), MeasureIt runs a set of datums (about 50), including various basic microsoft®.net Framework runtime operations. The simplified sample output is shown in Figure 1.
Figure 1 Sample Benchmark
Name |
Medium value |
Average |
Standard deviation |
Minimum value |
Maximum Value |
Example |
Nothing [count=1,000] |
0.000 |
0.037 |
0.110 |
0.000 |
0.366 |
10 |
Methodcalls:emptystaticfunction () [count=1000 scale=10.0] |
1.000 |
1.103 |
0.496 |
0.857 |
2.577 |
10 |
Objectops:new Class () [count=1000 scale=10.0] |
5.060 |
10.223 |
13.927 |
3.340 |
51.215 |
10 |
Objectops:new Finalizableclass () [count=1000 scale=10.0] |
78.552 |
155.408 |
168.595 |
64.997 |
629.243 |
10 |
Objectops: (Class) Activator.CreateInstance (ClassType)] |
102.510 |
102.949 |
4.076 |
96.876 |
109.819 |
10 |
Arrays:localintptr[i] = 1 [count=1,000 scale=10.0] |
0.713 |
0.664 |
0.076 |
0.574 |
0.773 |
10 |
Arrays:string[i] = astring [count=1,000 scale=10.0] |
3.402 |
3.405 |
0.012 |
3.397 |
3.442 |
10 |
Delegates:ainstancedelegate () [count=1,000 scale=10.0] |
1.235 |
1.205 |
0.111 |
1.094 |
1.475 |
10 |
MethodReflection:Method.Invoke emptystaticfunction () |
472.283 |
472.744 |
5.409 |
466.291 |
482.094 |
10 |
P/invoke:fulltrustcall () [count=1,000] |
6.184 |
6.254 |
0.793 |
5.469 |
7.599 |
10 |
P/invoke:10 Fulltrustcall () (call average) |
2.669 |
2.688 |
0.061 |
2.665 |
2.870 |
10 |
P/invoke:1 Partialtrustcall [count=1,000] |
27.806 |
30.440 |
8.735 |
26.343 |
56.582 |
10 |
MeasureIt runs each datum 10 times and calculates the statistics based on the results. These report values are then quantified so that a single call to an empty method takes a unit of time. For example, Figure 1 shows a median time of 5.06 for assigning a small object, which means that usually assigning a small object is only a little more than five times times the time it takes to invoke the method. But this is not the case in all cases. Note that the maximum time that an object is allocated is more than 51 units. As a result, it often takes longer than the average. In fact, if the benchmark unfortunately forces a large collection of fragments, its time spent in the method is likely to be much higher than the maximum reported here.
Even so, you should be able to discover the value of the MeasureIt tool. With almost no work to do, you can roughly learn about the cost of small allocations. Because the tool collects multiple examples and calculates statistics, you will also learn that some operations, such as object allocations, can vary considerably, which is the point. By understanding the minimum, maximum, and standard deviation information, you can determine whether the measurement is reliable enough.