Your point is not to stick to unit tests alone, which is quite different from overrating unit tests, therefore makes the title misleading. Just saying :)

More and more, I find myself writing less tests, to cover more code (most of the time 100% coverage) BUT my code is full of code contracts that fails when violated at testing time.

Sooner or later, the community will realize that having tests in unit test code or having code contracts in code, is pretty much the same thing, except that contracts are cheaper to write. Contracts are also good code embedded documentation.

codebetter.com/.../code-contracts-and-automatic-testing-are-pretty-much-the-same-thing

Your story was featured in ! Here is the link to vote it up and promote it: www.readof.com/.../1482

Krzysztof, I indeed still used much more Debug.Assert than the new MS contract API. (not sure abou that, I thing the MS contract API+tooling is supported on .NET 3.5, isn't it?)

Once I wrote about making your assertions fail while running tests, it is unfortunately not trivial:

codebetter.com/.../unit-test-vs-debug-assert

A thousand times yes. A vast majority of my bugs aren't bad code per se. Instead they are integration issues. It's just too easy to code for, and write tests for, the happy path. Given a choice between integration tests and unit tests, I'd choose integration tests any day.

+4.  Some of us have been saying this for *years*.

Some of the rules of unit tests say that they must not require any external dependencies and they should be pretty fast, because otherwise nobody will run them. Additionally, this make possible to safely run tests during the build process on a build server. Integration tests in turn may require you to deploy a database, configure IIS, create some registry entries, etc. You wouldn't want to do all these things on your build server, right? So while I agree with what you're saying, I still don't see how to make this process really smooth.

It's true what you say: Tests are quite often not the best solution in terms of 'getting the most bang for your buck'. But more often than not they are. The problems that you avoid by having tests are usually much bigger than the few minutes that you occasionally can save. And what's even more important: You cannot reliably know that in advance, so there will be a certain amount of wrong decisions - which in turn then will cost you additional time and effort. In other words: The bottom line is that having tests is an overall time and cost saver.

I don't think it makes much sense to 'stop and think' here. There's not really a point to make an educated decision, it would take time (and nerves) to think about every little piece of code, and there necessarily would be a considerable percentage of wrong decisions anyway.

You need a default way of doing things, a way that you can blindly (well, almost) rely on and that certainly yields a high quality result in any case. That's why I do all my coding with TDD, and I don't think about it.

- Thomas

Absolutly agree, and I'm ahead of you.  Less than 1% of my code is covered by unit tests because they are more costly to write and maintain than module tests and don't provide the conclusive assurances we get from module tests.

"unit tests are overrated as a way of testing everything"

I don't think there's anything in the literature that suggests otherwise (unless you have some references, of course?) Nothing by Beck, Jeffries, Martin (Uncle Bob), (Martin) Fowler, Feathers, Osherove, at least; that's all I have within arm's reach just now.

So I take the message to be: "if you rely exclusively on unit tests then you're probably doing it wrong". With which I have no problem in agreeing.

Me and my team are regularly building large multi user systems for the corporate world and there are no unit-tests to be seen at all in the code we're producing.

What we do rely on is simulations, automated robots taking actions as specified by test cases.

When a system passes the test cases the product is complete according to specification.

Then we ship it and that's it.

The message that unit tests should not be your only test methodology I completely agree with.  But you are giving a completely wrong impression of the value of unit testing in this article.  Unit tests have a particular role, they test a particular code sections functionallity in isolation.  They are particularly terrible at testing UI code, but that is not what they are for.  I have recently taken a dev position on a government  software team with a company that just won the maintenance contract from a different company.  The previous company didn't write a single unit test for the nearly million lines of code in the project.  This is a pain in ^&%%%^ to figure out what the code is doing in many places and every change comes with ulcers wondering what might be breaking.  You don't write unit tests for you right now, you right unit tests for the poor SOB who has to maintain your crappy code later.

I agree with the basic premise of your article. For us, it usually comes down to cost/benefit tradeoffs, and writing reams of unit tests doesn't pay off, especially given tight budgets and time constraints.

I hope you'll now consider writing an article: "TDD is overrated", or better yet, "The OO religion is overrated.  ;)

The title is absolutely misleading!!! I was curious to see which idiot thinks unit tests are overrated. But after reading the article the author meant something else..something we all developers know. Of course unit tests is not everything! You need integration tests and functional tests in addition. And then you need the stress/load testing that we developers usually ignore or forget. Many times I've seen in my company we find problems that QA and developers failed to catch.

It's worth to note that impeccable, flawless code, can cause failures in completely unrelated code. After all, the computer resources are limited, yet shared by all code. Cache, cpu and misc has to be used wisely and in symphony by all parts in a moderately complex system.

This is however no longer the programmers responsibility. Given the popularity of high-level languages, clever compilers and complex operating systems, it must be assumed the programmer counts on things to be sorted out to the best by the OS/Compilers and friends. Of course this never happens in the real world and there are countless of large multi million dollar systems that failed spectacularly, ready to attest to this fact.

Case of point, the Mars Pathfinder nearly tanked due to priority inversion.

I am tempted to say that unit testing can be dangerous, as they promote the idea that progress is in completing individual units and not in meeting real world requirements.

Interaction between modules are not tested by unit test, it should be done by integration test.   Changing the interface of a module should be done carefully with proper versioning providing backward compatibility (new module should take input with both old and new interface and write with new). Having too many unit tests can slow down refactoring of the code so it is essential to keep it in a reasonable amount.  It is important to make different layers of tests, not just unit tests. In our case doing stress tests (running servers at high load for some days) brought up bugs that were several years in the software - from the first version of the code, but remained silent. Static code analysing tools can give sometimes good hints on potential problems.

Despite any efforts some bugs can survive.

A modern company with brains has/uses professional testers.

They find the glitches that unit tests will never cover. Plus they look at it much more (mostly) functionally and not technically as a developer always will.

If only TDD was called something like "Intention Driven Development" then maybe this sort of confusion wouldn't be so common.

I don't see much point in writing unit tests unless you're using TDD, because as you say there are better ways to test your software.

However, TDD when applied correctly is a highly disciplined way of writing software (Red -> Green -> Refactor; repeat). Using it also helps improve the design of your code by forcing you to actually interact with it at the unit level, thereby rewarding loosely coupled, highly cohesive and simple code with easy to write tests.

So I agree that if you view unit tests as a form of testing then there are better ways to spend your testing dollar. But if you view unit tests as a development tool (i.e. TDD), then I don't think you beat them as a tool to help you write high quality code.

Misleading titles sucks!

Its easy to rely on integration or acceptance level tests when you don't need to setup test data, or interact with external sources. We have 500 such tests that take over 30 mins to run, its a nightmare when you want to know if you've broken anything. I wish the people that had written them to begin with had stuck interfaces in front of everything and unit tested.

The point of unit tests is fast feedback of your logic. I 100% agree that you need also need to test integration points and system level testing, but not at the cost of your sanity...

For me, it's about understanding the code being tested and picking the right tool for the job.  If I see a particularly complex unit of code, I'll throw unit tests at it.  If on the other hand the system is composed of lots of moving parts, I'll consider an integration test.  I say consider because depending on the system, integration tests may not be practical to run with every code change.  

Honestly though, my life as a QA dude would be much easier if developers just consistently tried their own code by hand first before handing it to the QA team.  That would catch most of the stupid bugs they allow through check-in, and would make QA more efficient because they wouldn't be tripping over obvious and blocking issues on their way to finding more interesting bugs.

Great article though!

I blogged on something similar a while back myself.  (blog.baltrinic.com/.../test-driven-development-without-unit-tests).  My focus was on writing test that survive and have value for refactoring and don't need to change as a result.

Though not stated in the article, the rule of thumb that I have come to rely on and would propose to other is "Tests should be written at one level of abstraction above the level where change is likely to occur."