Code Coverage - 100% Coverage and Still Failing

Casey posted last year about statistics that lie.  In the post Casey gives an overview of Cyclomatic Complexity and some guidelines to aim for when reviewing the cyclomatic complexity of your methods.  There's a gem in the comments post by Davy Brion:

Yeah, code coverage is such a terrible indicator... the only time it really means something is when it's low... at least you can be sure that's not good.  If it's high, it doesn't mean anything. 

To illustrate why 100% code coverage should not be your ultimate goal, I have provided the simple illustration below.

The System Under Test (SUT)

I have a very simple method, which returns the quotient of two numbers:

   1: public int SampleMethod(int dividend, int divisor)
   2: {
   3:     return dividend / divisor;
   4: }

The Test

   1: [Test]
   2: public void Sample()
   3: {
   4:     // Act
   5:     var result = SampleMethod(8, 2);
   6:  
   7:     // Assert
   8:     Console.WriteLine(result);
   9:     Assert.That(result, Is.EqualTo(4));
  10: }

The Warm Fuzzy

Well would you look at that, I'm covered, 100%.  Time to ship this bad boy.

clip_image002

The Hammer Drops

   1: [Test]
   2: public void Sample_zero_divisor()
   3: {
   4:     // Act
   5:     var result = SampleMethod(8, 0); // will throw an exception
   6:  
   7:     // Assert
   8:     Console.WriteLine(result);
   9:     Assert.That(result, Is.EqualTo(???));
  10: }

Produces an error:

TestCase 'Samples.CodeCoverage.Tests.Test.Sample_zero_divisor'
failed: System.DivideByZeroException : Attempted to divide by zero.

Conclusion

Yes this is all a bit contrived, but it illustrates the point, 100% code coverage does not mean 100% tested. As demonstrated here, 100% coverage simply means all code paths have been visited.  As such we have to both code defensively and test defensively.


Posted 03-31-2009 12:15 PM by Tim Barcz
Filed under:

[Advertisement]

Comments

Paco wrote re: Code Coverage - 100% Coverage and Still Failing
on 03-31-2009 2:52 PM

100% code coverage does not mean 100% tested, but less than 100% code coverage means not 100% tested.

Some code coverage tools can catch the divide by zero by the way.

DotNetShoutout wrote Code Coverage - 100% Coverage and Still Failing - Tim Barcz - Devlicio.us
on 03-31-2009 3:03 PM

Thank you for submitting this cool story - Trackback from DotNetShoutout

WillH wrote re: Code Coverage - 100% Coverage and Still Failing
on 04-01-2009 2:47 AM

  1: public int SampleMethod(int dividend, int divisor)

  2: {

  3:     return divident / divisor;

  4: }

Your code wouldn't even compile. Did you test it? ;-)

Krzysztof Koźmic wrote re: Code Coverage - 100% Coverage and Still Failing
on 04-01-2009 4:14 AM

Yeah,

that's why I believe you should in your build report show skip methods with coverage > X alogether.

Same with unit tests. I wish TeamCity allowed you to see only new tests (like tests from within last week) and show old tests only when they're failing.

But I digress

Xtoff ( ;) )

Jack wrote re: Code Coverage - 100% Coverage and Still Failing
on 04-01-2009 5:34 AM

100% means a little, it can not cover all the business rules and so on.

Ed wrote re: Code Coverage - 100% Coverage and Still Failing
on 04-01-2009 6:02 AM

Nice example, thanks

Tim Barcz wrote re: Code Coverage - 100% Coverage and Still Failing
on 04-01-2009 7:41 AM

@WillH

Fixed.  (I changed the variable names while posting and didn't had a typo)

JP wrote re: Code Coverage - 100% Coverage and Still Failing
on 04-01-2009 9:15 AM

Tim, great post.

Freddy wrote re: Code Coverage - 100% Coverage and Still Failing
on 04-01-2009 6:58 PM

I agree, but it isn't the only sample about it.

Also there is the case with code that is missing some logic, code coverage won't tell you are missing tests (and the related logic) :)

Adam Tybor wrote re: Code Coverage - 100% Coverage and Still Failing
on 04-05-2009 10:00 AM

A perfect simple example of why having code coverage thresholds on a build server provides little value and a lot of wasted effort.

Simply beautiful example Tim!

Robz wrote re: Code Coverage - 100% Coverage and Still Failing
on 04-26-2009 9:03 PM

I believe this is in the book of testing, chapter 3, verses 1-3.

Thou shalt test all errors. Thou shalt not trust the number of the test coverage, as it is surely wrong. Thou shalt inspect areas where coverage is very low as this will surely lead to the most evil of all bugs.

Ben Hyrman wrote re: Code Coverage - 100% Coverage and Still Failing
on 05-21-2009 11:22 AM

At my last employer, I should my boss a bunch of metrics we could look at if we wanted and what they were useful for (cyclomatic complexity, code coverage, afferent and efferent couplings, etc)

He decided he liked code coverage and wanted to make sure we were increasing it every month. Once it became tied to our goals, those getters and setters were exercised religiously.

devlicio.us wrote re: Code Coverage - 100% Coverage and Still Failing
on 04-05-2011 11:34 AM

Code coverage 100 coverage and still failing.. Peachy :)

devlicio.us wrote re: Code Coverage - 100% Coverage and Still Failing
on 04-22-2011 9:33 AM

Code coverage 100 coverage and still failing.. Nice :)

devlicio.us wrote re: Code Coverage - 100% Coverage and Still Failing
on 04-30-2011 9:13 PM

Code coverage 100 coverage and still failing.. Huh, really? :)

devlicio.us wrote re: Code Coverage - 100% Coverage and Still Failing
on 06-02-2011 8:23 PM

Code coverage 100 coverage and still failing.. Ho-o-o-o-t :)

devlicio.us wrote re: Code Coverage - 100% Coverage and Still Failing
on 06-03-2011 1:24 AM

Code coverage 100 coverage and still failing.. Bang-up :)

devlicio.us wrote re: Code Coverage - 100% Coverage and Still Failing
on 06-04-2011 8:44 PM

Code coverage 100 coverage and still failing.. May I repost it? :)

devlicio.us wrote re: Code Coverage - 100% Coverage and Still Failing
on 06-05-2011 1:43 AM

Code coverage 100 coverage and still failing.. Awful :)

devlicio.us wrote re: Code Coverage - 100% Coverage and Still Failing
on 06-13-2011 11:45 PM

Code coverage 100 coverage and still failing.. Great idea :)

b3hk porn 92a3 wrote re: Code Coverage - 100% Coverage and Still Failing
on 07-02-2011 2:28 AM

Code coverage 100 coverage and still failing.. Nice :)

l783 sex b5rn wrote re: Code Coverage - 100% Coverage and Still Failing
on 07-02-2011 7:30 AM

Code coverage 100 coverage and still failing.. Awful :)

qiqc sex video t2w3 wrote re: Code Coverage - 100% Coverage and Still Failing
on 07-02-2011 7:27 PM

Code coverage 100 coverage and still failing.. WTF? :)

cheap backlinks wrote re: Code Coverage - 100% Coverage and Still Failing
on 01-17-2013 1:16 PM

DBNEQ9 Wow, great blog article.Much thanks again. Fantastic.

loss weight pills wrote re: Code Coverage - 100% Coverage and Still Failing
on 01-31-2013 8:11 PM

lwbxvF Fantastic blog post.Really thank you!

buy imitrex online wrote re: Code Coverage - 100% Coverage and Still Failing
on 02-15-2013 1:34 PM

4L6A59 Very good blog.Thanks Again. Want more.

http://clomidnoprescription.beep.com/ wrote re: Code Coverage - 100% Coverage and Still Failing
on 02-27-2013 6:19 PM

XVWACW I value the blog.Thanks Again. Really Great.

buy clomid no prescription wrote re: Code Coverage - 100% Coverage and Still Failing
on 02-28-2013 10:30 PM

XEh43w Thanks again for the article post. Cool.

bookmarks wrote re: Code Coverage - 100% Coverage and Still Failing
on 03-13-2013 1:38 PM

JEvRyS Thanks for sharing, this is a fantastic blog article.Really thank you! Fantastic.

bookmarks wrote re: Code Coverage - 100% Coverage and Still Failing
on 03-14-2013 10:38 AM

jx66Bh Really appreciate you sharing this blog. Will read on...

social bookmarks wrote re: Code Coverage - 100% Coverage and Still Failing
on 03-14-2013 6:15 PM

u4qJEA Very informative blog article.Much thanks again. Much obliged.

buy social bookmarks wrote re: Code Coverage - 100% Coverage and Still Failing
on 03-22-2013 9:21 PM

N2Q8xc Very informative blog article.Really looking forward to read more. Awesome.

buy social bookmarks wrote re: Code Coverage - 100% Coverage and Still Failing
on 03-23-2013 11:08 AM

yWkcCO A round of applause for your article post.Much thanks again. Cool.

Social bookmarks wrote re: Code Coverage - 100% Coverage and Still Failing
on 04-12-2013 11:29 PM

6uaP9L Very good article post.Really thank you! Really Great.

Social bookmarks wrote re: Code Coverage - 100% Coverage and Still Failing
on 04-19-2013 8:07 PM

IpGm8h Major thankies for the blog post.Really thank you! Much obliged.

social bookmarking service wrote re: Code Coverage - 100% Coverage and Still Failing
on 04-28-2013 12:51 AM

47ZXU8 I really like and appreciate your article post. Fantastic.

slr lenses wrote re: Code Coverage - 100% Coverage and Still Failing
on 05-14-2013 6:16 AM

vn3lv2 Appreciate you sharing, great blog article.Much thanks again. Want more.

best social bookmarks wrote re: Code Coverage - 100% Coverage and Still Failing
on 06-19-2013 7:02 AM

Iyxxkm I really like and appreciate your post.Really looking forward to read more. Really Great.

ndlesaenro wrote re: Code Coverage - 100% Coverage and Still Failing
on 07-10-2013 5:11 PM
xgslbqfazx wrote re: Code Coverage - 100% Coverage and Still Failing
on 07-10-2013 9:15 PM
msdhfogwfv wrote re: Code Coverage - 100% Coverage and Still Failing
on 07-11-2013 1:17 AM
uwjqfhmhhl wrote re: Code Coverage - 100% Coverage and Still Failing
on 07-11-2013 5:19 AM
aquugdwuom wrote re: Code Coverage - 100% Coverage and Still Failing
on 07-11-2013 6:45 AM

bmmtdefwmjdjp, http://www.cisldkfglu.com qwwjkqomgu

gfxdaptlxb wrote re: Code Coverage - 100% Coverage and Still Failing
on 07-11-2013 10:46 AM

ffbxjefwmjdjp, http://www.vrceppxksl.com kkevbvhqkm

chiilywidy wrote re: Code Coverage - 100% Coverage and Still Failing
on 07-12-2013 7:28 PM

stpyaefwmjdjp, [url=http://www.hxapdxlkma.com]isdtekwerq[/url]

rfgxtdnisr wrote re: Code Coverage - 100% Coverage and Still Failing
on 07-12-2013 11:28 PM

avzzwefwmjdjp, [url=http://www.efdpjpyths.com]xintmmvsin[/url]

fmipdhknhk wrote re: Code Coverage - 100% Coverage and Still Failing
on 07-13-2013 3:29 AM

lszxlefwmjdjp, [url=http://www.xwymxbhpxp.com]gntpfrffxg[/url]

cdckankqwx wrote re: Code Coverage - 100% Coverage and Still Failing
on 07-13-2013 7:32 AM

vrftdefwmjdjp, [url=http://www.gcexqijiol.com]ixlgxbizfq[/url]

cwtdvbsgcr wrote re: Code Coverage - 100% Coverage and Still Failing
on 07-13-2013 11:34 AM

ohqvdefwmjdjp, [url=http://www.eytecrqifx.com]zcgimgxiyh[/url]

whszeewoau wrote re: Code Coverage - 100% Coverage and Still Failing
on 07-14-2013 5:34 PM
jkhwgdxvpl wrote re: Code Coverage - 100% Coverage and Still Failing
on 07-14-2013 9:41 PM
best news on earth wrote re: Code Coverage - 100% Coverage and Still Failing
on 07-26-2013 7:54 AM

BtYK7H This is one awesome blog article.Really looking forward to read more. Fantastic.

amazing news wrote re: Code Coverage - 100% Coverage and Still Failing
on 08-02-2013 1:08 PM

Im thankful for the blog post. Great.

greatest news wrote re: Code Coverage - 100% Coverage and Still Failing
on 08-02-2013 6:50 PM

vPwIjI I am so grateful for your article post.Really thank you! Will read on...

super news that are fun wrote re: Code Coverage - 100% Coverage and Still Failing
on 08-05-2013 12:13 AM

jUE7jt Great article post.Much thanks again. Fantastic.

great link buildng wrote re: Code Coverage - 100% Coverage and Still Failing
on 08-19-2013 12:47 PM

yIWYv9 wow, awesome article.Really thank you! Will read on...

bfndjlhtks wrote re: Code Coverage - 100% Coverage and Still Failing
on 08-23-2013 6:19 PM

exmaiefwmjdjp, <a href="www.gizvmuesni.com/.../a>

ankmcbyega wrote re: Code Coverage - 100% Coverage and Still Failing
on 08-23-2013 9:59 PM

kwthvefwmjdjp, <a href="www.saknmbltxm.com/.../a>

zdonwnsiwc wrote re: Code Coverage - 100% Coverage and Still Failing
on 08-24-2013 1:45 AM

frpfgefwmjdjp, <a href="www.fmtlukklqa.com/.../a>

yjiqgcvhdc wrote re: Code Coverage - 100% Coverage and Still Failing
on 08-24-2013 9:27 AM

sbfpdefwmjdjp, <a href="www.lzsnaaerpf.com/.../a>

qkorqdchbk wrote re: Code Coverage - 100% Coverage and Still Failing
on 08-24-2013 12:37 PM

qdfvcefwmjdjp, http://www.xwcmqubbos.com/ wjlhcabdgb

rrujvjihmq wrote re: Code Coverage - 100% Coverage and Still Failing
on 08-24-2013 4:14 PM

adfwrefwmjdjp, http://www.minowveyja.com/ kvojtrapcq

great seo service wrote re: Code Coverage - 100% Coverage and Still Failing
on 09-03-2013 9:29 PM

wC67gI I cannot thank you enough for the blog post.Thanks Again. Fantastic.

pro link building wrote re: Code Coverage - 100% Coverage and Still Failing
on 09-24-2013 9:47 AM

1BSe2B I value the blog.Really thank you! Keep writing.

best linkbuilding wrote re: Code Coverage - 100% Coverage and Still Failing
on 10-01-2013 10:42 AM

Yz0Low Thanks a lot for the article post.Really looking forward to read more. Really Cool.

link building team wrote re: Code Coverage - 100% Coverage and Still Failing
on 10-16-2013 10:32 AM

93A8fY Really enjoyed this article.Really thank you! Fantastic.

link building wrote re: Code Coverage - 100% Coverage and Still Failing
on 10-31-2013 8:41 PM

f5HUgZ I really enjoy the article. Really Great.

Add a Comment

(required)  
(optional)
(required)  
Remember Me?

About The CodeBetter.Com Blog Network
CodeBetter.Com FAQ

Our Mission

Advertisers should contact Brendan

Subscribe
Google Reader or Homepage

del.icio.us CodeBetter.com Latest Items
Add to My Yahoo!
Subscribe with Bloglines
Subscribe in NewsGator Online
Subscribe with myFeedster
Add to My AOL
Furl CodeBetter.com Latest Items
Subscribe in Rojo

Member Projects
DimeCasts.Net - Derik Whittaker

Friends of Devlicio.us
Red-Gate Tools For SQL and .NET

NDepend

SlickEdit
 
SmartInspect .NET Logging
NGEDIT: ViEmu and Codekana
LiteAccounting.Com
DevExpress
Fixx
NHibernate Profiler
Unfuddle
Balsamiq Mockups
Scrumy
JetBrains - ReSharper
Umbraco
NServiceBus
RavenDb
Web Sequence Diagrams
Ducksboard<-- NEW Friend!

 



Site Copyright © 2007 CodeBetter.Com
Content Copyright Individual Bloggers

 

Community Server (Commercial Edition)