Derik Whittaker

Syndication

News


Images in this post missing? We recently lost them in a site migration. We're working to restore these as you read this. Should you need an image in an emergency, please contact us at imagehelp@codebetter.com
Green tests != proven code

Let me just say this slowly.... Simply because your tests 'pass' does not mean your code has been proven correct or that it works.

I know from time to time the notion that 'because I test, my code is solid' floats around the blog-a-sphere.  Well, I am here to say, that is a 100% lie.

Here is a classic example.  Today I was writing a few tests to ensure that our data transaction management was working as expected.  In my test sweet I was going to have 2 tests:

  1. Transaction_EnsureRollbackWorks()
  2. Transaction_EnsureCommitWorks()

When I setout to write the test for the rollback all was going good.  My test passed and I was happy (kid in a candy store kinda happy :) ).  It was not till I was working on my test for the commit that I was a sad (like a kid who was told he could not eat candy in that candy store :( ).

When I started to test my commit logic I was not able to pull the newly committed data from the database.  I was perplexed.... what was the issue.  I decided to take a look at the rollback code (I had hijacked most of the logic from that test).  When I started to step through the code I noticed that something was missing.  My call to SubmitChanges() (we are using Linq). 

Here is the code for the test:

[Test]
public void Transaction_EnsureRollbackWorks( )
{
    RepositoryOverride repository = new RepositoryOverride();
    Domain.Entities.AccessKey accessKey1 = new Domain.Entities.AccessKey { AccessKeyActionTypeID = 1, Key = "123456789", PIN = "ertyui", ProgramID = 1, RecruitingID = 2, UserID = 3 };
    Domain.Entities.AccessKey accessKey2 = new Domain.Entities.AccessKey { AccessKeyActionTypeID = 1, Key = "56789", PIN = "ertyui", ProgramID = 1, RecruitingID = 2, UserID = 3 };
    
    repository.BeginTransaction();

    repository.DBContextInstance().AccessKey.InsertOnSubmit( accessKey1 );
    repository.DBContextInstance().AccessKey.InsertOnSubmit( accessKey2 );
    // code was not in the original test repository.DBContextInstance().SubmitChanges();

    repository.RollbackTransaction();

    // Check to make sure the data is NOT in the DB
    var results = from ak in repository.DBContextInstance().AccessKey
                  where ak.Key == "56789"
                  select ak;

    Assert.That( results.Count(), Is.EqualTo( 0 ) );
}

If you notice before the repository.RollbackTransaction(); line there is NO call to SubmitChanges().  Because of this my roll back appear to work and then when I attempted to re-query the data it came up empty, which was the expectation.

The goal to any test is to clearly put your code though its paces, to prove it correct.  However, if you test is bad, or has logic issues you are not proving your code.

You have been forewarned Green Test != Proven Code

Till next time,


Posted 03-26-2008 12:49 PM by Derik Whittaker
Filed under:

[Advertisement]

Comments

Jason Bock wrote re: Green tests != proven code
on 03-26-2008 2:20 PM

Saying that it's a 100% lie is a little too much. Yes, having green tests doesn't mean that you're "finished" or everything is perfect. But it's an extra check to see that all is well in the code base. Other things need to be run - code analysis, metrics, etc.

DotNetKicks.com wrote Green tests != proven code
on 03-26-2008 2:32 PM

You've been kicked (a good thing) - Trackback from DotNetKicks.com

Richard Lowe wrote re: Green tests != proven code
on 03-26-2008 2:43 PM

This is exactly the same problem that has made science difficult from the beginning.   Well, actually, 2 problems:

1. The test can be well designed or flawed or something on a spectrum inbetween and it's sometimes not obvious. and

2. You can never 100% prove a 'positive' empirically, you can only gather evidence that something works OR prove that it doesn't under certain (or all) circumstances.

IMO, Testing is often hard and should never be thought of as a guarantee, but rather, as another tool of risk management.

Christopher Bennage wrote re: Green tests != proven code
on 03-26-2008 3:08 PM

Did you start off with a failing test? I mean a real failing test, not a trivial one.

Derik Whittaker wrote re: Green tests != proven code
on 03-26-2008 3:14 PM

@Chris,

For this test, no.  

During the development of actual transaction code i did have failing tests, but it was not a 'good' test and tested too much stuff.  So i split them out with the new separate tests.

Scott Bateman wrote re: Green tests != proven code
on 03-26-2008 5:55 PM

I concur - if you are not sure how your tests should be crafted then a green light is only marginally better than a clean compile.

Markus Zywitza wrote re: Green tests != proven code
on 03-27-2008 3:47 AM

Look at any book on TDD or unit testing in general. All of them say (close to the beginning) that unit tests prove that your code works a intended, BUT NOT that your intentions are correct.

Finds of the Week - March 30, 2008 » Chinh Do wrote Finds of the Week - March 30, 2008 » Chinh Do
on 04-01-2008 1:00 AM

Pingback from  Finds of the Week - March 30, 2008 » Chinh Do

About The CodeBetter.Com Blog Network
CodeBetter.Com FAQ

Our Mission

Advertisers should contact Brendan

Subscribe
Google Reader or Homepage

del.icio.us CodeBetter.com Latest Items
Add to My Yahoo!
Subscribe with Bloglines
Subscribe in NewsGator Online
Subscribe with myFeedster
Add to My AOL
Furl CodeBetter.com Latest Items
Subscribe in Rojo

Member Projects
DimeCasts.Net - Derik Whittaker

Friends of Devlicio.us
Red-Gate Tools For SQL and .NET

NDepend

SlickEdit
 
SmartInspect .NET Logging
NGEDIT: ViEmu and Codekana
LiteAccounting.Com
DevExpress
Fixx
NHibernate Profiler
Unfuddle
Balsamiq Mockups
Scrumy
JetBrains - ReSharper
Umbraco
NServiceBus
RavenDb
Web Sequence Diagrams
Ducksboard<-- NEW Friend!

 



Site Copyright © 2007 CodeBetter.Com
Content Copyright Individual Bloggers

 

Community Server (Commercial Edition)