Casey posted last year about statistics that lie. In the post Casey gives an overview of Cyclomatic Complexity and some guidelines to aim for when reviewing the cyclomatic complexity of your methods. There's a gem in the comments post by Davy Brion:
Yeah, code coverage is such a terrible indicator... the only time it really means something is when it's low... at least you can be sure that's not good. If it's high, it doesn't mean anything.
To illustrate why 100% code coverage should not be your ultimate goal, I have provided the simple illustration below.
The System Under Test (SUT)
I have a very simple method, which returns the quotient of two numbers:
1: public int SampleMethod(int dividend, int divisor)
2: {
3: return dividend / divisor;
4: }
The Test
1: [Test]
2: public void Sample()
3: {
4: // Act
5: var result = SampleMethod(8, 2);
6:
7: // Assert
8: Console.WriteLine(result);
9: Assert.That(result, Is.EqualTo(4));
10: }
The Warm Fuzzy
Well would you look at that, I'm covered, 100%. Time to ship this bad boy.
The Hammer Drops
1: [Test]
2: public void Sample_zero_divisor()
3: {
4: // Act
5: var result = SampleMethod(8, 0); // will throw an exception
6:
7: // Assert
8: Console.WriteLine(result);
9: Assert.That(result, Is.EqualTo(???));
10: }
Produces an error:
TestCase 'Samples.CodeCoverage.Tests.Test.Sample_zero_divisor'
failed: System.DivideByZeroException : Attempted to divide by zero.
Conclusion
Yes this is all a bit contrived, but it illustrates the point, 100% code coverage does not mean 100% tested. As demonstrated here, 100% coverage simply means all code paths have been visited. As such we have to both code defensively and test defensively.
Posted
03-31-2009 12:15 PM
by
Tim Barcz