IOC container benchmark-Unity, Windsor, structuremap and spring. net

There are a number of inversions of control containers out there so I thought it wocould be an interesting experiment to do a simple benchmark. there are different ways that one can instantiate a type in. net, for example via the new operator, activator, getuninitializedobject and dynamic method. the performance difference between these methods are in some cases quite high, maybe the same is true These IOC containers? Granted IOC containers do more than just create objects so other factors will probably play a big role in the results.

So here are the contestants:

• Castle Windsor
  • Part of the popular Castle project.
  • First introduced in 2004
• Structuremap
  • SourceForge Project maintained and created by Jeremy D. Miller
• Spring. net
  • Replicated red by the spring Java framework
  • First introduced in 2004, www.springframework.net
• Unity
  • Created by the Microsoft patterns Practices Team
  • First released in release L 2008
  • Project homepage on codeplex

I have been using Castle Windsor since 2005 and I think it is the best of the bunch, so I guess I am unconsciously biased toward Windsor. however I will try to make this benchmark as objective as I can.

The scenario for this test:

• Have each IOC container resolve a usercontroller 1000 000 times
• The usercontroller will have two constructor Dependencies
• Run the test with transient (new instance for each resolve) and Singleton Components

The usercontroller looks like this:

Public class usercontroller

{

Private iuserrepository repository;

Private iauthentificationservice authservice;

Public usercontroller (iuserrepository repository, iauthentificationservice authservice)

{

This. repository = repository;

This. authservice = authservice;

}

}

I have also a general container interface that the benchmark engine will use. Each container will implement this interface.

Public interface icontainer

{

String name {Get ;}

T resolve <t> ();

Void setupfortransienttest ();

Void setupforsingletontest ();

}

All tests used the latest released version of each library. before you interpret these charts please observe that the measurement is for one million component resolves which means the actual time difference between each container is actually very small.

Here are the results when all components were setup as singletons:

Here are the results when all components were setup as transient:

So what does these charts tell us? Lets take the biggest difference in the transient case, spring. Net took 44.149 seconds and Unity took 8.164 seconds, what is the actual difference when resolving a single instance?

Spring. Net: 44.149/1000000 = 0.000044149 seconds

Unity: 8.164/1000000 = 0.000008164 seconds

So the actual difference is only about 36 microseconds. Another way to put these values into perspective is to compare against the new operator. I created a newoperatorcontainer with a resolve method that looks like this:

Public t resolve <t> ()

{

Object o = new usercontroller (New ldapuserrepository (), new defaultauthentificationservice ());

Return (t) o;

}

OK, comparing the above with an inversion of control container is like comparing apples to oranges, an IOC handles so much more than just object creation. also an IOC cannot use the new operator directly but must use one of the other methods. my guess is that all IOC containers in this test uses an approach which involve il generation which if cashed comes close to using the new operator directly. anyway I think it will show just how small the difference between the real IOC containers are. in order to visualize this I needed to invert the values so that high means fast and low means slow.

Bytes

UPDATE: The above chart can be very misleading. the X-axis is not seconds but 1/s. I hope it shows that the difference between the containers are very small compared to instantiating the objects manually.

OK, can we draw any conclusion from the test? Well I think we can say that performance shoshould not be an issue when choosing one of these IOC containers. the difference is too small. when you choose which container to use you shoshould consider other aspects, like how invasive the container is to they way you want to work.

For the complete code: iocbenchmark.zip

Posted by torkel ödegaard at AM

Labels: Benchmarks, C #, Castle, IOC

13 comments:

Anonymous said...

IMHO your last graph is very misleading. Why do you use the inverse values? Use the usual values and point out how tiny the "new" value is-or if you feel like the inverse is beneficial at least label your axis correctly.
Otherwise nice article

2017l 14,200 8 pm

Torkel ödegaard said...

Yes, you are right. I also feel that the last graph is too misleading. I will correct it when I get home from work.

2017l 14,200 8 pm

Joshua said...

I wocould interested to see how ninject compares.

2017l 14,200 8 pm

Torkel ödegaard said...

I guess it shoshould have been in the test having a slogan like "lightning-fast dependency injection for. Net"
Hopefully I will have time to do it later in the week, thanks for the tip!

2017l 14,200 8 pm

Bil simser said...

This is interesting but IMHO a bit of a waste of time. I mean, what application wocould * ever * need to try to create a million objects all at once? Even in a tight loop you * might * Create a few thousand objects. I just can't think of any use case where this situation wowould ever manifest itself so to me the numbers are pretty meaningless. interesting to look at, but not of any value to judge something. if the differences were significant with say a few hundred objects then maybe this exercise wocould be worth something.

2017l 16,200 8 pm

Torkel ödegaard said...

Yes, I agree that this is not a realistic scenario. doing one million resolves is not something that do in one request. that was just to accumulate the performance difference to see if there were any substantial difference.
I don't think it was a waste of time, because it cocould have potentially been a big meaningful difference between them. now it turned out that there weren't but that didn't make the test meaningless, because now you know

2017l 16,200 8 pm

Nate kohari said...

Nice post. I 'd also be interested to know how ninject does in comparison.
Bil's right, though, that it wouldn't typically make a difference, because most applications don't need to create 1,000,000 instances. however, faster IOC also means that you can use it in places that you wouldn't otherwise be able to -- for example, devices that support the Compact framework.
Still a worthwhile study, and it's interesting to see the results!

2017l 16,200 8 pm

Ruurd boeke said...

I'm sorry, but if you are going to do a test and find a huge relative difference, you shoshould not disregard it by looking at the absolute difference. what's the point in doing the test?
I think it's pretty clear that no container takes a very long time, so what were you hoping to find?
(No disrespect meant, I clicked on the post because I was curious as well)

2017l 16,200 8 pm

Torkel ödegaard said...

Well the point was to check if there was any relevant difference, not to check which was the fastest.
And I think you can actually disregard a relative difference if that difference is still not relevant when you look at the absolute performance.
My conclusion to the test was that the relative difference was too small to make any relevant difference in real applications.

2017l 16,200 8 pm

Anonymous said...

Nice benchmark. A Functional Comparison for the used IOC containers wocould be interesting. Where do the performance differences come from?

2017l 17,200 8 AM

Nick said...

I think the comment about functional differences is important-what are you getting for you time?
In a real app with more components, the factors affecting container performance will shift from the expense of creating instances, to the expense associated with algorithms that are affected by the number of instances (e.g. which instance from snapshot shocould be returned ?) And algorithms that take a time proportional to the size of the dependency graph (e.g. Circular dependency checking .)
You can never tell where that performance bottleneck is going to be until you measure it

2017l 20,200 8 AM

Sharkboy said...

I think what this page shows is that using IOC containers will not make your app significantly slower. What makes apps slow are poor design, bad databases, and network latency.
As to which one to use, you have to try and figure out which one has a future.
I wocould rule out spring. Net because it is a Java port and will always be several steps behind in supporting the latest. NET Framework features.
Structuremap is cool but since only one guy is supporting it I wowould perform CT the project to die once that dude gets tired of dealing with it.
Unity may be the safe bet being that it has "official" support from Microsoft. the MS guys where starting to promote this at teched this year. on the other hand, don't expect CT any innovation from Microsoft.
If you want a supported component, go with unity. If you want innovation, go Windsor.

June 16,200 8 6:24 AM

Luke said...

You clearly don't know Jeremy Miller very well if you think that's what'll happen with structuremap.

July 6, 2008 pm