My Thoughts on ORM comparisons

UPDATE 1: I made clear that NH actually performs better in those posts, when a simple optimization such as Enabled Batching.
UPDATE 2: Please check Ayende's latest post for how those benchmarks can be made fair.

I am a bit late to the party but there we go.

Last several days, there have been a hot debate on the performance and capability of NH vs some other commercial/noncommercial ORMs. This issue has been discussed both in NHibernate Developer List and in Ayende’s blog. I have recently noticed another benchmark between NH and EF, which I hope to address, too.

Fallacies of benchmark

What I have seen in those benchmarks is the wrong use of ORMs. This statement has two aspects: Wrong use of ORM and wrong use of NH itself.

In ORM, you don’t use it to do bulk operations. And 0.001 second difference will not differ anything in your application. If it does, you don’t also use RDBMS, you use something else.

There is no point on making a loop of thousands of iterations which saves an entity. Davy has well pointed out that this has a lot of implications in NHibernate, unlike some other ORMs. First of all, NHibernate first-level cache will be in action and it will fill the cache with each item saved. Secondly, as NH makes use of POCO it has no mechanism of things like INotifyPropertyChanged, it has to compare the old to the new value. As you may guess, the more entity you have, the longer it will take. No exception. If you want to leave first-level cache out of the game, you use StatelessSession feature of NHibernate, no exception. Actually when you check Fabio Maulo’s post you’ll see it can even become faster without Stateless Session.

Another point I have noticed in both benchmarks is that they are/were not fair. If I compare it to EF, lets say, ctx.SaveChanges() will batch the chages and send the query to the database in one go. NH has this feature, as Ayende already pointed out, since 2006. However, NH by default doesn’t make use of it, and this is not a bug, it is a feature. You have to use setting to enable it.

Gergely Orosz has made his own benchmark and didn't use batching support and NH fell well behind of EF. I contacted him and he was kind enough to discuss the issues. I showed him how to enable batching and why not to flush everytime, as it wouldn't be fair. Now NH performs much better and he said he will update his post.

Benchmark the community, too

NHibernate has a big community, and great minds as committers. It really does matter. Let me give you some statistics about NHibernate development and its community

Here is the commit statistics.

image 

It is one of the most active projects in .net environment. The image above doesn’t include the contrib projects. A great kiss goes  to Fabio Maulo for his massive efforts.

Let’s look at discussion group, shall we?

image

Do you know any other group with that activity? I don’t know.

 

In addition, NHibernate gives you more than _ANY_ ORM can give. It gives you a lot of extensibility points which can be used in a lot of scenario. Validator, Search, Spatial, Shards to name a few.

In conclusion

NHibernate is the obvious winner.


Posted 08-21-2009 7:27 PM by Tuna Toksoz
Filed under: , ,

[Advertisement]

Comments

Rob Cecil wrote re: My Thoughts on ORM comparisons
on 08-21-2009 4:34 PM

Here. Here. Your final point should speak volumes. As a very early adopter of ORM in .Net (2002/2003), I had to sustain two *poor* implementations: ORM.NET and Objectz.Net. Eventually migrated off of those to an OSS ORM : OJB.Net. But unfortunately that had a single committer, and we eventually settled on a Nhibernate 1.x.

Rob

Chris Nicola wrote re: My Thoughts on ORM comparisons
on 08-21-2009 9:56 PM

You hit the nail on the head there  The entire issue of benchmarking is overshadowed by the community support offered by choosing nHibernate.  Support the goes above and beyond even the ORM domain.

Nigel Spencer wrote re: My Thoughts on ORM comparisons
on 08-21-2009 10:23 PM

I agree with your thoughts on benchmarking. However, the rest of your argument seems to boil down to NHibernate is being actively developed and has a large active community. Both are important considerations, but by those measures alone I would say EF is the clear winner. I imagine the EF team easily beats those commit statistics, and I would guess that the EF community is growing faster than NH (albiet still much smaller). In my opinion it is still easier to find better documentation/definitive posts on EF than it is for NH. NH has IMHO always been difficult to learn, terribly documented and whilst there of lots of opinions its difficult for a novice to determine which of them are worthwhile. 'A large fan base alone doesn't a great product make.'

Tuna Toksoz wrote re: My Thoughts on ORM comparisons
on 08-22-2009 5:14 AM

@Nigel

You're right, actually I wanted to show how benchmarks I have seen so far aren't fair, by code. I actually did by Gergely's benchmark, and we improved NH results by comparing apples to apples. NH has batching support and without enabling it and flushing everytime a thing is added while EF batches things, is not really fair.

Gergely(gregdoesit.com/.../nhibernate-vs-entity-framework-a-performance-test) will update his results soon, I hope.

EF has the advantage of Big MS support and an inherent increasing support, but what people see so far, EF lacks a lot of thing. I am pretty sure that EF will become mature enough and take NH down after a couple of years perhaps.  I am totally fine with it. It is one less work for us to do.

What I cannot stand is unfair attacks on NH, so unfair that it became obvious for anybody. Look the O R M Battle piece, the author keeps stating he doesn't care MS for the sake of increasing their product's profits. They do it wrong, and NH fails. They don't enable batch just to make NH fail.

The community support I have been mentioning is a clear advantage over several MS. People says that we are more responsive than many technical support. This is not to say we offer the best support, we are open source developers and we do all those things in our free times. NH has many times wtih low activity, this is expectable. The thing is NH offers the best support at no cost.

NH also has the advantage of being opensource. Our release cycle is pretty much short. We do release early, we do release often. You see a bug and it is well documented for us to see? We will most probably be fixed in next cycle.

IF you need something really fast? Pay NH developers so that they will dedicate more time on your issue.

This is what I wanted to say by "obvious" winner. It's performance is already fast and even faster than MS alternative (no offense on EF, just as an illustory examle, I hope to do some benchmark with other frameworks and perhaps prepare another post with details), and NH, in addition to performance, has the advantage of opensource-view. It is extensible, there is a huge community. It can provide you a lot, if you use correctly.

Sorry, I thinkl this comment has become to the size of a post :)

Tuna Toksoz wrote re: My Thoughts on ORM comparisons
on 08-22-2009 5:25 AM

@Chris

Please read the post, I have updated the issue. If you have been following this O R M Battle thing, you have already noticed that it is unfair, as they don't enable batching for NH while EF and their own product does batching.

With a fair comparison, such as we did with Gergely Orosz (http://gregdoesit.com)  but it is not yet published, shows what happens when you enable it. NH actually wins in storing and updating. I think i well pointed enough to why NH is good in benchmarking when you do it "fair".

Then I mentioned about community. NH is not a one man show, NH has community support means more minds in an issue, and the better it can become (let me emphasize, it is already better as pointed out by their benchmarks).

Anonymous wrote re: My Thoughts on ORM comparisons
on 08-25-2009 12:45 AM

I agree nHibernate is real good alternative, but it's very slow to my liking. Extensibility etc is very good but I think for people who are ready to corrupt their class design a little in favor of speed by inheriting from a base class, they should be able to do that in nHibernate.

Reflection and Proxies too slow.

I 100% agree with you there is NO alternative yet for nHibernate in .NET world. I hope EF v4 is as good as touted in team blogs.

There is enough space for 2 good products in ORM space in .NET. I bet nHibernate and EF will eventually prevail.

James wrote re: My Thoughts on ORM comparisons
on 08-27-2009 7:12 PM

I'll stick to Lightspeed. I tried nHibernate a long time ago and the amount of up front effort doesn't pay off. Lightspeed on the other hand. Even if Lightspeed was the slowest performing ORM i would still pick it over any other.

About The CodeBetter.Com Blog Network
CodeBetter.Com FAQ

Our Mission

Advertisers should contact Brendan

Subscribe
Google Reader or Homepage

del.icio.us CodeBetter.com Latest Items
Add to My Yahoo!
Subscribe with Bloglines
Subscribe in NewsGator Online
Subscribe with myFeedster
Add to My AOL
Furl CodeBetter.com Latest Items
Subscribe in Rojo

Member Projects
DimeCasts.Net - Derik Whittaker

Friends of Devlicio.us
Red-Gate Tools For SQL and .NET

NDepend

SlickEdit
 
SmartInspect .NET Logging
NGEDIT: ViEmu and Codekana
LiteAccounting.Com
DevExpress
Fixx
NHibernate Profiler
Unfuddle
Balsamiq Mockups
Scrumy
JetBrains - ReSharper
Umbraco
NServiceBus
RavenDb
Web Sequence Diagrams
Ducksboard<-- NEW Friend!

 



Site Copyright © 2007 CodeBetter.Com
Content Copyright Individual Bloggers

 

Community Server (Commercial Edition)