Unit tests are overrated

Something is rotten in the state of Development. It seems to me we (developers) either ignore testing altogether, leaving it to the Q&A team when we throw them the app over the wall, or we concentrate on unit tests only. Let me make one thing clear before we go any further – unit tests are a fantastic and extremely valuable tool in many cases, and I by no means am trying to discourage anyone from writing unit tests. They have their place in the development pipeline (especially if you’re doing TDD).

While I still usually write a fair amount of unit tests I find the percentage of unit tests in the code I write is getting smaller. I just find unit tests are not the best value for the buck in many cases where previously I would write unit tests unconditionally, not really giving it any thought.

Where would that be?

Let’s stop and think for a second – what unit tests are good at. Unit tests exercise single units of functionality, and they’re the most fine grained tests you write. They tests a method produces the right output. They test a class responds in expected manner to results of invocation of another method, potentially on another object. And they really shine if those algorithms and small scale interactions are complex and/or part of API that is going to be used extensively where they do a really great job at being an executable documentation.

However, for quite a few scenarios unit tests just aren’t the best approach and that’s my goal with this blog post – to make you stop and think – should I write a unit test, or perhaps an integration test, with real, not stubbed out dependencies?

To illustrate the point let me tell you a (real) story. I was called to a client recently to have a look at the issue they were having in their system. The system was composed of an administration website, and a client application that was communicating with the server side via a WCF service. The issue they were having is that some information that was entered on the administrative website, wasn’t displayed properly in the client application. However all the unit tests were passing, for all three elements of the system. As I dug into the codebase I noticed that some transformation was being done to the information coming from the web UI on the web-app end before it was being saved, and them what was supposed to be the same process in reverse was happening on the WCF Service side. Except it wasn’t. The transformation being done changed, and was updated everywhere except for the website (this also reinforces the point I made in my previous blogpost, that changes in application code should almost always be accompanied by changes to tests). So the unit tests between two parts of the system were out of sync, as were their implementations yet unit tests weren’t able to detect that. Only a grander scale, integration test, that would cover both ends of the spectrum, the saving of the information on the website site, and reading it from the same repository on the service side would have caught this kind of change.

At the end, let me share a secret with you – Windsor, has very few unit tests. Most tests there are (and we have close to 1000 of them) exercise entire container with no part stubbed out in simulations of real life scenarios. And that works exceptionally well for us, and lets us cover broad range of functionality from the perspective of someone who is actually using the container in their real application.

So that’s it – do write tests, and do remember to have more than just a single hammer in your tool belt.


Posted 02-28-2011 10:00 AM by Krzysztof Koźmic
Filed under:

[Advertisement]

Comments

Konrad wrote re: Unit tests are overrated
on 02-28-2011 4:43 AM

Your point is not to stick to unit tests alone, which is quite different from overrating unit tests, therefore makes the title misleading. Just saying :)

Patrick Smacchia wrote re: Unit tests are overrated
on 02-28-2011 5:08 AM

More and more, I find myself writing less tests, to cover more code (most of the time 100% coverage) BUT my code is full of code contracts that fails when violated at testing time.

Sooner or later, the community will realize that having tests in unit test code or having code contracts in code, is pretty much the same thing, except that contracts are cheaper to write. Contracts are also good code embedded documentation.

codebetter.com/.../code-contracts-and-automatic-testing-are-pretty-much-the-same-thing

TrackBack wrote Story added
on 02-28-2011 6:13 AM

Your story was featured in ! Here is the link to vote it up and promote it: www.readof.com/.../1482

Krzysztof Koźmic wrote re: Unit tests are overrated
on 02-28-2011 7:11 AM

@Konrad

I may have gone for a catchy title but that's not just it. I meant the unit tests are overrated as a way of testing everything. They have their place obviously but other ways of testing may yield better value. I thought I made that pretty clear in the first paragraph.

Krzysztof Koźmic wrote re: Unit tests are overrated
on 02-28-2011 7:15 AM

@Patrick Smacchia

+1 although I usually don't rely on the Code Contracts API and use good old Debug.Assert (mostly because most of the code I write these days is still .NET 3.5 sadly)

Patrick Smacchia wrote re: Unit tests are overrated
on 02-28-2011 7:51 AM

Krzysztof, I indeed still used much more Debug.Assert than the new MS contract API. (not sure abou that, I thing the MS contract API+tooling is supported on .NET 3.5, isn't it?)

Once I wrote about making your assertions fail while running tests, it is unfortunately not trivial:

codebetter.com/.../unit-test-vs-debug-assert

John DeHope wrote re: Unit tests are overrated
on 02-28-2011 8:50 AM

A thousand times yes. A vast majority of my bugs aren't bad code per se. Instead they are integration issues. It's just too easy to code for, and write tests for, the happy path. Given a choice between integration tests and unit tests, I'd choose integration tests any day.

jdn wrote re: Unit tests are overrated
on 02-28-2011 9:42 AM

+4.  Some of us have been saying this for *years*.

andrew wrote re: Unit tests are overrated
on 02-28-2011 10:49 AM

Some of the rules of unit tests say that they must not require any external dependencies and they should be pretty fast, because otherwise nobody will run them. Additionally, this make possible to safely run tests during the build process on a build server. Integration tests in turn may require you to deploy a database, configure IIS, create some registry entries, etc. You wouldn't want to do all these things on your build server, right? So while I agree with what you're saying, I still don't see how to make this process really smooth.

Thomas Weller wrote re: Unit tests are overrated
on 03-01-2011 1:33 AM

It's true what you say: Tests are quite often not the best solution in terms of 'getting the most bang for your buck'. But more often than not they are. The problems that you avoid by having tests are usually much bigger than the few minutes that you occasionally can save. And what's even more important: You cannot reliably know that in advance, so there will be a certain amount of wrong decisions - which in turn then will cost you additional time and effort. In other words: The bottom line is that having tests is an overall time and cost saver.

I don't think it makes much sense to 'stop and think' here. There's not really a point to make an educated decision, it would take time (and nerves) to think about every little piece of code, and there necessarily would be a considerable percentage of wrong decisions anyway.

You need a default way of doing things, a way that you can blindly (well, almost) rely on and that certainly yields a high quality result in any case. That's why I do all my coding with TDD, and I don't think about it.

- Thomas

John Wilson wrote re: Unit tests are overrated
on 03-01-2011 7:35 AM

Absolutly agree, and I'm ahead of you.  Less than 1% of my code is covered by unit tests because they are more costly to write and maintain than module tests and don't provide the conclusive assurances we get from module tests.

Mike Woodhouse wrote re: Unit tests are overrated
on 03-01-2011 8:05 AM

"unit tests are overrated as a way of testing everything"

I don't think there's anything in the literature that suggests otherwise (unless you have some references, of course?) Nothing by Beck, Jeffries, Martin (Uncle Bob), (Martin) Fowler, Feathers, Osherove, at least; that's all I have within arm's reach just now.

So I take the message to be: "if you rely exclusively on unit tests then you're probably doing it wrong". With which I have no problem in agreeing.

multi wrote re: Unit tests are overrated
on 03-01-2011 8:43 AM

Me and my team are regularly building large multi user systems for the corporate world and there are no unit-tests to be seen at all in the code we're producing.

What we do rely on is simulations, automated robots taking actions as specified by test cases.

When a system passes the test cases the product is complete according to specification.

Then we ship it and that's it.

Dave wrote re: Unit tests are overrated
on 03-01-2011 8:53 AM

The message that unit tests should not be your only test methodology I completely agree with.  But you are giving a completely wrong impression of the value of unit testing in this article.  Unit tests have a particular role, they test a particular code sections functionallity in isolation.  They are particularly terrible at testing UI code, but that is not what they are for.  I have recently taken a dev position on a government  software team with a company that just won the maintenance contract from a different company.  The previous company didn't write a single unit test for the nearly million lines of code in the project.  This is a pain in ^&%%%^ to figure out what the code is doing in many places and every change comes with ulcers wondering what might be breaking.  You don't write unit tests for you right now, you right unit tests for the poor SOB who has to maintain your crappy code later.

Paul wrote re: Unit tests are overrated
on 03-01-2011 12:01 PM

I agree with the basic premise of your article. For us, it usually comes down to cost/benefit tradeoffs, and writing reams of unit tests doesn't pay off, especially given tight budgets and time constraints.

I hope you'll now consider writing an article: "TDD is overrated", or better yet, "The OO religion is overrated.  ;)

shan wrote re: Unit tests are overrated
on 03-01-2011 12:01 PM

The title is absolutely misleading!!! I was curious to see which idiot thinks unit tests are overrated. But after reading the article the author meant something else..something we all developers know. Of course unit tests is not everything! You need integration tests and functional tests in addition. And then you need the stress/load testing that we developers usually ignore or forget. Many times I've seen in my company we find problems that QA and developers failed to catch.

multi wrote re: Unit tests are overrated
on 03-01-2011 12:16 PM

It's worth to note that impeccable, flawless code, can cause failures in completely unrelated code. After all, the computer resources are limited, yet shared by all code. Cache, cpu and misc has to be used wisely and in symphony by all parts in a moderately complex system.

This is however no longer the programmers responsibility. Given the popularity of high-level languages, clever compilers and complex operating systems, it must be assumed the programmer counts on things to be sorted out to the best by the OS/Compilers and friends. Of course this never happens in the real world and there are countless of large multi million dollar systems that failed spectacularly, ready to attest to this fact.

Case of point, the Mars Pathfinder nearly tanked due to priority inversion.

I am tempted to say that unit testing can be dangerous, as they promote the idea that progress is in completing individual units and not in meeting real world requirements.

Tom wrote re: Unit tests are overrated
on 03-01-2011 12:19 PM

Interaction between modules are not tested by unit test, it should be done by integration test.   Changing the interface of a module should be done carefully with proper versioning providing backward compatibility (new module should take input with both old and new interface and write with new). Having too many unit tests can slow down refactoring of the code so it is essential to keep it in a reasonable amount.  It is important to make different layers of tests, not just unit tests. In our case doing stress tests (running servers at high load for some days) brought up bugs that were several years in the software - from the first version of the code, but remained silent. Static code analysing tools can give sometimes good hints on potential problems.

Despite any efforts some bugs can survive.

Hajo wrote re: Unit tests are overrated
on 03-02-2011 2:39 AM

A modern company with brains has/uses professional testers.

They find the glitches that unit tests will never cover. Plus they look at it much more (mostly) functionally and not technically as a developer always will.

Jason wrote re: Unit tests are overrated
on 03-05-2011 11:45 AM

If only TDD was called something like "Intention Driven Development" then maybe this sort of confusion wouldn't be so common.

I don't see much point in writing unit tests unless you're using TDD, because as you say there are better ways to test your software.

However, TDD when applied correctly is a highly disciplined way of writing software (Red -> Green -> Refactor; repeat). Using it also helps improve the design of your code by forcing you to actually interact with it at the unit level, thereby rewarding loosely coupled, highly cohesive and simple code with easy to write tests.

So I agree that if you view unit tests as a form of testing then there are better ways to spend your testing dollar. But if you view unit tests as a development tool (i.e. TDD), then I don't think you beat them as a tool to help you write high quality code.

user wrote re: Unit tests are overrated
on 03-06-2011 5:23 PM

Misleading titles sucks!

Krzysztof Koźmic wrote re: Unit tests are overrated
on 03-08-2011 11:16 PM

@user

yeah I agree. Good thing I never use them.

Matt wrote re: Unit tests are overrated
on 03-12-2011 5:48 AM

Its easy to rely on integration or acceptance level tests when you don't need to setup test data, or interact with external sources. We have 500 such tests that take over 30 mins to run, its a nightmare when you want to know if you've broken anything. I wish the people that had written them to begin with had stuck interfaces in front of everything and unit tested.

The point of unit tests is fast feedback of your logic. I 100% agree that you need also need to test integration points and system level testing, but not at the cost of your sanity...

Jason Barile wrote re: Unit tests are overrated
on 03-15-2011 11:09 AM

For me, it's about understanding the code being tested and picking the right tool for the job.  If I see a particularly complex unit of code, I'll throw unit tests at it.  If on the other hand the system is composed of lots of moving parts, I'll consider an integration test.  I say consider because depending on the system, integration tests may not be practical to run with every code change.  

Honestly though, my life as a QA dude would be much easier if developers just consistently tried their own code by hand first before handing it to the QA team.  That would catch most of the stupid bugs they allow through check-in, and would make QA more efficient because they wouldn't be tripping over obvious and blocking issues on their way to finding more interesting bugs.

Great article though!

Ken Baltrinic wrote re: Unit tests are overrated
on 04-19-2011 9:09 AM

I blogged on something similar a while back myself.  (blog.baltrinic.com/.../test-driven-development-without-unit-tests).  My focus was on writing test that survive and have value for refactoring and don't need to change as a result.

Though not stated in the article, the rule of thumb that I have come to rely on and would propose to other is "Tests should be written at one level of abstraction above the level where change is likely to occur."

About The CodeBetter.Com Blog Network
CodeBetter.Com FAQ

Our Mission

Advertisers should contact Brendan

Subscribe
Google Reader or Homepage

del.icio.us CodeBetter.com Latest Items
Add to My Yahoo!
Subscribe with Bloglines
Subscribe in NewsGator Online
Subscribe with myFeedster
Add to My AOL
Furl CodeBetter.com Latest Items
Subscribe in Rojo

Member Projects
DimeCasts.Net - Derik Whittaker

Friends of Devlicio.us
Red-Gate Tools For SQL and .NET

NDepend

SlickEdit
 
SmartInspect .NET Logging
NGEDIT: ViEmu and Codekana
LiteAccounting.Com
DevExpress
Fixx
NHibernate Profiler
Unfuddle
Balsamiq Mockups
Scrumy
JetBrains - ReSharper
Umbraco
NServiceBus
RavenDb
Web Sequence Diagrams
Ducksboard<-- NEW Friend!

 



Site Copyright © 2007 CodeBetter.Com
Content Copyright Individual Bloggers

 

Community Server (Commercial Edition)