CLR comprehensive and thorough parsing: Early and often rating performance, part 2nd

In the previous issue of "CLR thorough resolution," I emphasized that to reliably create high-performance programs, you need to understand the performance (msdn2.microsoft.com/magazine/cc424899) of the various components used in the early stages of the design. This requires performance data. Therefore, measurement is an integral part of the design process.

I also introduced a tool called MeasureIt in that period, which makes it easy to create new benchmarks to quickly gather the data needed to make good design decisions. The original figures provided by tools such as MeasureIt are of paramount importance, and it is also important to enable people to understand the basic meaning of these numbers. Based on this understanding, you can predict some of its performance before you actually measure it. That's what I'm going to talk about here.

MeasureIt Overview

If you have not yet downloaded the MeasureIt tool, I strongly recommend that you download it now. The tool is located in the MSDN® magazine website in the download code of this column, composed of an EXE file. Running it will generate a Web page that shows the results of running a benchmark suite. After installation, you can access other documents by running the following command:

Measureit/usersguide

MeasureIt comes along with its source code and can be easily decompressed using the/edit qualifier. This makes adding a new benchmark as easy as writing one or two lines of code (and providing the code to be timed). For more detailed instructions on how to do this, see the user's Guide.

The MeasureIt datum is associated with different performance zones and is indicated on the command line when the tool is started. By default (that is, no command-line arguments), MeasureIt runs a set of datums (about 50), including various basic microsoft®.net Framework runtime operations. The simplified sample output is shown in Figure 1.

Figure 1 Sample Benchmark

Name	Medium value	Average	Standard deviation	Minimum value	Maximum Value	Example
Nothing [count=1,000]	0.000	0.037	0.110	0.000	0.366	10
Methodcalls:emptystaticfunction () [count=1000 scale=10.0]	1.000	1.103	0.496	0.857	2.577	10
Objectops:new Class () [count=1000 scale=10.0]	5.060	10.223	13.927	3.340	51.215	10
Objectops:new Finalizableclass () [count=1000 scale=10.0]	78.552	155.408	168.595	64.997	629.243	10
Objectops: (Class) Activator.CreateInstance (ClassType)]	102.510	102.949	4.076	96.876	109.819	10
Arrays:localintptr[i] = 1 [count=1,000 scale=10.0]	0.713	0.664	0.076	0.574	0.773	10
Arrays:string[i] = astring [count=1,000 scale=10.0]	3.402	3.405	0.012	3.397	3.442	10
Delegates:ainstancedelegate () [count=1,000 scale=10.0]	1.235	1.205	0.111	1.094	1.475	10
MethodReflection:Method.Invoke emptystaticfunction ()	472.283	472.744	5.409	466.291	482.094	10
P/invoke:fulltrustcall () [count=1,000]	6.184	6.254	0.793	5.469	7.599	10
P/invoke:10 Fulltrustcall () (call average)	2.669	2.688	0.061	2.665	2.870	10
P/invoke:1 Partialtrustcall [count=1,000]	27.806	30.440	8.735	26.343	56.582	10

MeasureIt runs each datum 10 times and calculates the statistics based on the results. These report values are then quantified so that a single call to an empty method takes a unit of time. For example, Figure 1 shows a median time of 5.06 for assigning a small object, which means that usually assigning a small object is only a little more than five times times the time it takes to invoke the method. But this is not the case in all cases. Note that the maximum time that an object is allocated is more than 51 units. As a result, it often takes longer than the average. In fact, if the benchmark unfortunately forces a large collection of fragments, its time spent in the method is likely to be much higher than the maximum reported here.

Even so, you should be able to discover the value of the MeasureIt tool. With almost no work to do, you can roughly learn about the cost of small allocations. Because the tool collects multiple examples and calculates statistics, you will also learn that some operations, such as object allocations, can vary considerably, which is the point. By understanding the minimum, maximum, and standard deviation information, you can determine whether the measurement is reliable enough.