Jak Charlton - Insane World

Around the World in 90 Days

Jak Charlton — Tue, 02 Mar 2010 06:38:00 GMT

Around 3 months ago, I packed up in the UK, and headed to the other side of the world, to Australia.

Last Autumn I was chatting to Richard Banks over Twitter, and he mentioned that Readify were looking to hire good .Net developers in Australia. I joked that if they wanted to pay my air fare over I would certainly do an interview.

Sadly Readify don't pay for the air fare, but after a bit more joking I took Richard seriously and dropped them a copy of my CV. A few months later, after a fairly thorough interview (part of which took place over Skype and Live Meeting... sadly the reason for no free holiday), I took the decision to join Readify as a Senior Consultant.

For the last 18 years I have been a self employed consultant / contractor in the UK, and have made a fairly good living at it. In fact this will be the first time I have been an employee since I was aged 22. So this was a fairly big decision.

What swung it for me was the general positive attitude of the Readify people I spoke to, and their focus on technical excellence and good development practices. This is a company that is rapidly growing, and despite the general world economic conditions, Australia managed to avoid the vast majority of the financial crisis.

As part of their expansion, Readify are still recruiting, and due to the shortage of good developers in Australia, they are willing to hire from overseas, and deal with all of the immigration and visa issues needed to get you into the country. So far at least one person I have chatted to over Twitter has applied, and if you look through the list of Readify employees you are likely to see a number of people who are highly respected in the .Net world - it's a great bunch of people.

Apparently only 1% of applicants to Readify get through the interview process, so if you are looking for a new career, are open to moving, and are pretty darn hot around .Net and good development practices, maybe you should think of applying. http://readify.net/about-readify/careers/

ASP.NET MVC in the Wild

Jak Charlton — Tue, 09 Feb 2010 11:02:00 GMT

Just before I emigrated to Australia, I took on a small contract to build a website for a UK company who wanted to start up a new kind of UK recruitment site, one where employers could advertise directly, and more specifically one where recruitment agencies couldn’t.

The result was Empty Lemon (www.emptylemon.co.uk) – I leave you to figure out why it is called Empty Lemon, took me a fair while to be honest!

This site was built in around 4 weeks, using ASP.NET MVC2, NHibernate, Fluent NHibernate, Windsor, JQuery, Automapper, and a host of other great tools and libraries.

The site is actually, despite the occasional bit of code I’m not too delighted with, a great example of how fast a site can be developed with ASP.NET MVC and the chosen tooling. In just four weeks we went from more or less nothing, to a full domain model (all be it a fairly simple one) running over a SQL Server 2008 database, using a combination of NH, Fulltext search and the Spatial services from SQL Server 2008.

I also want to thank James Gregory, Steve Strong and Hadi Hariri for (sometimes unknowingly) helping me write the site, with useful snippets of code, and even in Steve’s case a lot of work on getting NH to play fair with the spatial services.

So, if you are looking for a job in the UK, or if you are looking to employ someone, I strongly suggest you take a look at www.emptylemon.co.uk. It’s a little empty right now, but some large employers are coming on board soon, and other companies are signing up day by day – these things are always chicken and egg, so if you like the site, feel free to tell others about it – after all, who really likes recruitment agents anyway ;)

Solving Complexity with Message Based Architectures

Jak Charlton — Thu, 04 Feb 2010 07:55:00 GMT

Ultimately most complexity in software comes not from the requirements, the business logic, or even the underlying systems. Most complexity comes out of a poorly considered and managed architecture, and this is commonly seen in tightly coupled systems that rapidly degrade into Big Balls of Mud.

The key to simplifying architectures is to decouple their component parts, making each more specialised and less dependent on the others. This allows every component to focus on a specific task without risk of those tasks becoming compromised by, or interfering with, others.

The basic premise of most architectures that succeed in this way is that they are message based, rather than being glorified CRUD systems pulling data to and from databases. Operations are central to the system, data is ultimately just the storage for the results of those operations.

This separation of concerns allows development teams to concentrate on component parts with little impact upon other teams, and no more reliance upon them than an agreement on the messages they will listen for and those they will publish. While messages share some characteristics of service interfaces, they differ mostly in their granularity and lack of dependence upon one another. Messages allow true decoupling, where traditional service layers only promise this and usually end up creating even more tightly coupled systems.

"Message based" does not inherently mean SOA, and in many ways differs substantially. Although both can be considered message based, SOA brings significant overhead and generalisation, along with an organisation wide remit. Simple message based systems operate on the basis of message within the application rather than across an enterprise.

Poor software architecture ends up impacting more than just the software system under development, and frequently the impact can be felt across an organisation; in the way development teams approach problems, in the way they respond to new requirements, and throughout the SDLC.

More often than not, the SDLC is a reflection of the weaknesses in software architecture, and is frequently trying to play 'catch up' to attempt to deal with the issues weak architecture brings. These compromises and aspects of the SDLC then start to propagate and impact other areas of the development and support process. Often decisions about software architecture become the underlying drivers for the entire IT department, and the SDLC turns into a point of conflict as it becomes more about trying to restrain and control a big ball of mud than it does about providing a framework for providing business value.

What Is The Difference Between an IoC Container and MEF?

Jak Charlton — Fri, 18 Dec 2009 06:50:00 GMT

Today a conversation sparked off on Twitter, started by Jimmy Bogard and Matt Hinze, and then carried on by myself and Glenn Block. The basic starting point was what the difference was between using an IoC container like Windsor or StructureMap and using MEF (the Managed Extensibility Framework)

I started off by answering a question from Jimmy, he wanted the one paragraph sales pitch for MEF that didn’t include technical terminology … I guess he was asking why MEF is a better option than one of the IoC containers we all love and hopefully all use religiously.

My key answers were:

MEF allows you to write your application as a series of much simpler mini applications, and have those brought together to create a much great whole

IoC container = service level components, MEF = Autonomous Components. A MEF "part" will probably use an IoC container

This sparked some comments from Glenn where he wanted to clarify that sometimes an IoC container is the right choice, sometimes MEF is, and sometimes both are. Luckily we are both pragmatic, so not much to disagree with there. He pointed to two of his blogs on the subject, I recommend reading them highly, they will do a much better job than I could at explaining the detail of the decision points, Should I use MEF for my general IoC needs? and Should I use MEF with an IoC container?

What’s Wrong With IoC Containers?

What an IoC container is very good at is low level control of your code, your dependency graphs, your lifestyle management, and gives you great low level ways to deal with facilities, factories, decorators and AOP. In just about any code base that goes past spike stage, I use an IoC container of some sort, it is just a fools errand to try and write and application without one these days. The keyword “new” is in my code smells book.

However, what an IoC container is pretty poor at is enforcing boundaries. Sure you could use an IoC container to write much larger applications, but as Glenn rightly pointed out, the central configuration becomes a headache. In fact, the central registration can rapidly become the equivalent of using global variables, you never really know what is using what as everything has access to everything else.

Most of the containers give you the capability to have child containers to try and deal with this situation, and while they do limit the global access issue, they are pretty hard to manage in their own right, and you can without some proper thought end up with a spaghetti mess of containers replacing your spaghetti mess of components.

An Architectural Solution To The Problem

To me, MEF is an architectural solution, while an IoC container is a code level solution. They both have their places in an application, but what MEF is exceptionally good at is creating process boundaries.

One of the core underlying principles of good use of an IoC container is that you only ever reference a container at a process boundary – in most applications that means it is probably referenced in your bootstrapping code inside your web application (maybe Global.ascx), or inside an HttpModule around your web services, but from that point on, your code should never reference the container again – the magic of an IoC container comes from relinquishing all object construction and dependency graph resolution to it.

But this means in a larger application, you may end up with massive containers with hundreds of registered components, and in reality most operations will only use a small handful at a time.

In most larger systems you would naturally have parts of the system you want to devolve into discrete components, or Autonomous Components. Doing this with an IoC container starts to lead you down an architectural design geared around the limitations of the IoC container, and not necessarily driven by good architectural principles.

What MEF allows you to do is to break the system apart easily, or as Glenn referred to it, MEF allows you to easily create composite applications. These composite parts can operate as Autonomous Components, in almost total isolation from the core application, and therefore they provide a natural boundary without having to use things like web of WCF service layers just for that purpose. Oddly the MEF terminology that Jimmy wanted to avoid in his original question was “part” and yet it fits naturally in this context.

Russian Dolls

As I started off by saying, I would almost always end up using an IoC container in any application I was to write, even a fairly small scale one. If we look at these MEF parts as now being Autonomous Components, and therefore operating as mini applications, it now becomes apparent that they will themselves need a way to create service components and dependency graphs, and while MEF could in theory do a lot of that, it’s just not the best fit for the job.

So it’s pretty certain that inside your MEF part you will be using yet another IoC container for object resolution, but now you don’t have a massive central registry, but one specifically geared to the MEF part in question.

And this pattern can recurse if needed, each time providing a clean boundary within your application.

A Real World Example

The last “large scale” system I worked on where I had the opportunity to put some of these ideas into practice was an insurance portal. This portal was responsible for providing insurance quotations, but across a massively diverse and inconsistent set of products.

The solution we came to in the end was a combination of all of the above principles – we had a web application that held a Castle Windsor container in Global.ascx, that container was responsible for instantiating everything from controllers backwards, and all of the service components they used.

But some of those controllers needed to call our product plugins, and in this case we actually used Castle Windsor to use MEF to instantiate the plugins (MEF parts). Within each product plugin we had another set of bootstrapping code that had it’s own instance of a Castle Windsor container, and it’s own registry specific to that plugin.

The plugins were actually used by a number of points in the application, some in process, and some out of process, either on other servers or just behind service or service bus boundaries – what MEF allowed us to do was create an independently deployable package that was self contained and self sufficient, without the need for managing masses of registration information – trying to use an IoC container for this would have been a mini-nightmare, but with MEF it was almost a pleasure.

Combining NHibernate, NH Spatial and SQL Server 2008 Fulltext Queries

Jak Charlton — Sun, 22 Nov 2009 22:42:00 GMT

Following Kevin Pang’s request on my last post regarding my problems getting NHibernate, NH Spatial and SQL Server Fulltext queries to work, I am finally getting around to giving some tips and pointers, and the code that Steve Strong of iMeta knocked up for me.

I needed to get these things working together as I was writing a recruitment website, and they had two basic requirements, the ability to search in candidates CVs and job descriptions for keywords, and the ability to search for those things within a certain distance – so an obvious search would be “find me all the C# developers within 10 miles of Central London”

Step One – NHibernate

Firstly, I’m not dumb, and at the beginning of the project I had decided to use NHibernate for all data access. The site didn’t have much of a domain model, but what it did have was ideally suited to a relational DB and simple persistence and querying by an ORM like NHibernate. What it also had going in it’s favour was the 2nd level caching capabilities of NHibernate, which would significantly reduce the workload on a site that was expected to grow quickly (or at least the owners hoped it would)

Win for NHibernate there – the icing on the cake was the excellent NHProfiler which would prove to make development a breeze over other options, and which also proved to make my querying a lot smarter and more bug free than my first horrid attempts.

Step Two – Spatial Searching

MS SQL Server 2008 introduced Spatial searching to it’s list of features – in fact it’s not just for searching, and not just by distance either, it can do a host of clever geographical type calculations and querying.

My first tests were done with “old fashioned” T-SQL and within a short while I had a reasonable distance search working, roughly speaking it looked like:

SET @from = geography::STGeomFromText('POINT(-0.094693 51.520223)', 4326); -- EC2M (London

SELECT * FROM JobSlot j
WHERE j.Location.STDistance(@from)<=(@distance * 1609.344)

Step Three – Fulltext Searching

Long ago, back in the age before the dawn of time, MS SQL Server had Fulltext searching, via the FREETEXT and CONTAINS syntax. Give it a keyword, or a list of them, and it will dig inside your text to find just what you want.

To use it on the site I was writing, we needed to index the content of CVs (resumes to your foreigners!) and some of those were in docx format – so my first step was to download the Microsoft Office 2007 iFilters. SQL Server uses the (now) age old technology of iFilters to extract the text from documents, and this filter pack allows it to look inside Office 2007 documents – if you want to index any other weird document formats, or even your own document format, there is probably an iFilter available, or you can write your own. With the iFilters installed, SQL Server could now index all of my content just fine.

SELECT *
FROM Candidate
WHERE FREETEXT(*, N'c#')

Combining It All

So far, so good. SQL Server can do pretty much all I need it to do, and combining those two queries is simplicity in TSQL, but as I was using NHibernate for all my data access, I wanted to try an get it all working there too.

NHibernate has support for the Spatial features of SQL Server 2008, along with other spatial databases, via the NHSpatial stuff in the NH Contrib project at http://nhcontrib.svn.sourceforge.net/svnroot/nhcontrib/

A download and compile of this will have you making HQL queries to do similar things to my TSQL spatial query above, in fact, the HQL query is remarkably similar to the TSQL:

select j from JobSlot j where NHSP.Distance(j.Loc, :point) <= (:distance * 1609.344)

Then I needed to get the Fulltext part working, after some chatting with Steve Strong, he came up with the following code:

            WKTReader wtk = new WKTReader();
            IGeometry point = wtk.Read("POINT( -0.123356 51.524142 )");
            point.SRID = 4326;

            string query =
                @"select j from JobSlot j
                  from freetexttable('JobSlot', :keywords) ftt
                  where NHSP.Distance(j.Loc, :point) <= (:distance * 1609.344)";

            var results = session.CreateQuery(query)
                .SetParameter("point", point, SpatialDialect.GeometryTypeOf(session))
                .SetParameter("distance", 1)
                .SetParameter("keywords", "hello world")
                .SetFirstResult(5)
                .SetMaxResults(10)
                .List();

And to get this to work we need 2 more classes, though the FreeText class is more an example of how to create an ISqlFunction for NHibernate:

	public class MyDialect : MsSql2008GeographyDialect
	{
		public MyDialect()
		{
			RegisterFunction("freetext", new FreeText());
			RegisterFunction("freetexttable", new StandardSQLFunction("freetexttable", null));
			RegisterFunction("contains", new StandardSQLFunction("contains", null));
		}
	}

	public class FreeText : ISQLFunction
	{
		public bool HasArguments
		{
			get { return true; }
		}

		public bool HasParenthesesIfNoArguments
		{
			get { return true; }
		}

		public virtual SqlString Render(IList args, ISessionFactoryImplementor factory)
		{
			SqlStringBuilder builder = new SqlStringBuilder();
			builder.Add("freetext(");

			if (args.Count == 1)
			{
				builder.Add("*, ");
				builder.AddObject(args[0]);
			}
			else
			{
				builder.AddObject(args[0]);
				builder.Add(", ");
				builder.AddObject(args[1]);
			}

			builder.Add(")");

			return builder.ToSqlString();
		}

		public virtual IType ReturnType(IType columnType, IMapping mapping)
		{
			return null;
		}
	}

Why I’m Not Using The Above Code

Yep, in my usual fashion I go through explaining all of that, just to say, I’m not doing it that way.

Eventually, I went back to another approach, that of using TSQL dynamically constructed in my code. I did this for a number of reasons, but primarily as it was easier for me to tune the query and add more options and variants in old fashioned SQL than in HQL by adding more extensions on to NHibernate. It also left the code base in a much easier to follow and extend state for the developers who would be maintaining the project – with the amount of investigation it took me to get this far, I figured extending it could be a nightmare for my successors.

The benefits I got from NHibernate I still retained, as this is a search function it was returning DTOs anyway, not entities, and these were just as efficient to read with a DataReader as with NHibernate (in fact a lot easier). When one of the results from the search is selected, the item is retrieved by NHibernate and we are back in ORM land.

It was also little effort to cache my SQL query results, so I also had all that NH was offering me in that respect too – a simple hash of the query parameters allowed me to cache the results object.

So there ya go, how to use Spatial and Fulltext with NHibernate, and why I didn’t do it! :)

When NHibernate Won't Do What You Think It Should

Jak Charlton — Mon, 02 Nov 2009 13:57:00 GMT

There are times, when you just think something should work, and one way or another it just doesn't. I've been having one of those weeks with NHibernate. I have been using MS SQL Server 2008 spatial searching, along with SQL Server Full text search, and using NHibernate for almost all my data access.

The only problem is that NHibernate out of the box doesn't like to play nicely with Spatial or Fulltext. After head scratching, head bashing, and general teeth grinding I figured better to cut my losses and ask some people that actually knew what they were doing.

Firstly I fired a twitter off in the direction of Ayende to get a feel for whether I was doing something wrong - with CreateSqlQuery NHibernate didn't want me to use the Spatial stuff - NHibernate uses a colon : to identify parameters, but in SQL 2008 Microsoft decided to use a double colon :: in it's spatial bits, so for example to declare a point you would use:

geography::STGeomFromText('POINT(51.520223 -0.094693 )', 4326)

That promptly causes NH to stop you from carrying on with a missing parameter. Ayende did indeed confirm it's not right, and after a few helpful suggestions he ran out of quick wins and suggested I might want to take advantage of his commercial support for NHibernate. I'm not adverse to such a solution, but felt this one might get complex, and Ayende's payment structure didn't fit my needs - this is a small project for a small client, and I could work around NH's limitations by using stored procedures (yes, I know, heresy).

After a bit more head scratching I remebered that iMeta also provide NH commecial support now, and having taken a look at their pages, I figured it would work much better for my limited budget.

After a quick chat with Hadi Hariri, he directed me on to Steve Strong, the man who wrote a great deal of the Linq to NHibernate stuff, and a bit of an expert on the parser.

Steve ripped through my queries with lighning pace, and had me a viable solution that afternoon - now that's what I call customer service. Today Steve has continued to sort out niggles with the problem, and it looks like we may have identified a few bugs in NHibernate Spatial and possibly the parser, which Steve will now push back into the NH trunk for future geenrations to benefit from.

So, althought NHibernate is open source software, and in theory you can fix this stuff yourself, this is just a post to point out that sometimes the best use of your time and effort is in things you can deal with quickly, and sometimes it is worth paying some real specialists to help out - the cost saving could be immense. If you have problems with NHibernate, I encourage you to take a look at the offerings available, especially the iMeta one :)

Just because NHibernate is free - doesn't mean it isn't worth paying for!

Welcome to Hadi Hariri

Jak Charlton — Wed, 14 Oct 2009 08:40:00 GMT

I am pleased to say that after a bit of gentle persuasion, one of my favourite Twitterers, general top notch blogger and all around good egg Hadi Hariri is joining us as a new blogger on devlicio.us

He has a wealth of experience, but I kept forgetting to visit his blog - and felt that devlicio.us could really benefit from his thoughts and advice - so I am really pleased he agreed to join the crew.

Welcome Hadi!

Good Developers Write Good Code, Instinctively

Jak Charlton — Tue, 06 Oct 2009 08:09:00 GMT

So my last post ruffled some feathers, but Steve hit the nail right on the head in his commments, and it was sort of the subtext of what I was saying:

All this kerfuffle about "software craftsmen" and "code quality" and "best practice" misses the ultimate fundamental truth :

Good Developers Write Good Code, Instinctively

Let start with a funny statistic ... it certainly applies to drivers, I'm sure it applies to developers too (and almost any group of people) ...

80% of developers think they are above average!

Our logical brains can't help but admit that at least 30% of developers are therefore deluded. Our mathematical brains can't help but admit that 50% of developers must be totally average or below average. And yet, most of us think we are above average developers. I would boldly go further, and put myself in the top 20% of developers - I like to think I at least have a damn good idea who the 20% above me are and why they are better.

If you are a "good developer" you will write "good" code. It will probably use a dozen standard patterns, maybe without you even knowing what they are called, it will have low coupling, high cohesion, be easy to test, easy to extend, hard to modify, avoid duplication and abstract code behind well thought out interfaces. And yet despite all that - you may well have never heard of software patterns, OCP, ISP, DIP, unit Testing, DRY, YAGNI, etc etc etc

Before The Renaissance

There was a time, which sadly I am old enough to remember, when we didn't have all these fancy names, and yet we pretty much did what all the enlightend are doing today.

We may not have had "unit testing", but we had "test harnesses", and we may not have had automated tests running after every build, but we had QA people using our test harnesses to verify our code.

We may not have had software patterns, but we all used factories, observers and singletons.

We may not have has SOLID, but we wrote good decoupled code, well abstracted, easy to test and easy to maintain.

None of These Things Are New

While many like to go on almost holy crusades about these wonderful new inventions, some of us know that these are just evolutions of things good developers have been doing since programming began. Sure they are now formalised, they have great names, they have been refined to be more automated and more efficient - but at the end of the day - good developers already did this stuff.

What concerns me more is how "new" these things seem to some of the holy crusaders. I remember code I wrote in 6502 assembler that had these principles applied (you try writing games in 3.5kb of RAM if you don't apply DRY), I remember code I wrote in COBOL on a PDP/11 that had "factories".

I remember working on a RAD team in an investment bank, long before RAD had meaning. We delivered software fast, direct to the customers, with rapid feedback and weekly releases. We sat with the traders, we were "Agile" and "Lean" long before anyone thought of proper words.

These great software practices were not invented, they were formalisations of things we were already doing.

All Of These Things Are Good

Before anyone decides to rip my head off and banish me to the pits of hell, let me state - all of these practices are good. More over, these practices are (probably) the pinnacle of our current understanding of development practices garnered over the last 30 or 40 years - and we should strive to write code with these things in mind. In 5 years I have no doubt all these things will have evolved, and there will be new best practices. Half of these things only apply to Java and C# anyway - in the world of F#, Erlang, Ruby and Python there are new principles and best practices.

Formalisation of all these practices, principles and methodologies is good - it helps promote understanding and gives us a way of communicating.

The thing is, the real "above average" developers were probably already doing this - and all that code they wrote for the past 30 years is probably still running your bank accounts, processing your taxes and helping keep your aircraft in the sky, precisely because they already did.

Below Average Developers Will Always Write Below Average Code

No way to avoid this I'm afraid - we can talk about the need to raise the bar, to raise the average, to help those below average developers write better code - but fundamentally, below average developers will always write below average code. The above average developers need to recognise this, and understand that they will always feel they are being held back by the lower 50%, and they are always fixing the code of the lower 50%, and they are always doing more than their share.

And one last question... are you sure YOU are above average? I'm not sure myself, in fact my most common comment in interviews is that I am not the best C# developer in the world by a long way, I am rarely the best C# developer in my team for that matter. Am I in the top 20%? Probably by my own standards and measures - that doesn't mean I am by everyone else's measures.

Developer Myopia

Jak Charlton — Mon, 05 Oct 2009 12:07:00 GMT

While some may argue that TDD, BDD, SRP, ahderence to SOLID, use of Patterns, and many other development practices and principles are critical to achieving great software, these all seem to miss some core truths.

Most Software Has Not Been Developed This Way

Yep, sorry to disappoint, but the vast majority of software in production in the world today has had none of these principles and practices applied. Some of the software may have had some of these practices applied - but it is (I would guesstimate) an incredibly small percentage.

And despite that (feel free to tell me I'm wrong) fact, most software works and continues to work. It processes your taxes correctly, makes sure your airplane doesn't blow up on take off, scans your body for cancer, lets you bill your customers, tells you what you are doing tomorrow, and lets you keep in touch with friends and family.

Most Software "Mistakes" Are Not Development Practices

Steve McConnell, he of Code Complete and Rapid Development fame (books everyone in development or management should read) has just published his "Software's Classic Mistakes" list for 2008, and none of these things we developer's like to refer to as "best practice" or "the right way" feature on the list - not in their specific forms, mnor even in their general forms.

In fact - all of the classic mistakes (which cause software projects to be delayed or fail) are process mistakes, they are failures of people, not of coding constructs and practices.

Unrealistic Expectations
Overly Optimistic Schedules
Shortchanged Quality Assurance
Wishful Thinking
Confusing Estimates With Targets
Excessive Multi Tasking
Feature Creep
Noisy Crowded Offices
Abandoning Planning Under Pressure
Insufficient Risk management

We can certainly look at this list and see how Agile, Lean and similar processes can help alleviate many of these, but where do you see "TDD", "BDD", "SOLID" or even "Automated Testing" resolving any of them? The closest we will come is to see "Shortchanged Quality Assurance" as being ticked off the list, but this still doesn't remove the need for real QA on a project, and real QA people.

Developer Myopia

While as developer's we see great value in many of these practices, we need to admit that we are actually a very small part of the equation, and sometimes developer myopia sneaks in - we think that what we do at a code level matters, substantially more than it does in the grand scheme of things.

By all means, apply and practice these things that create "quality" software, but bear in mind, there are much bigger and more important problems to resolve too.

Or to paraphrase (and bastardize) the Agile manifesto: While we see value in software patterns and practices, there is more value in solving process problems first

Ship it, or Ship Out

Jak Charlton — Fri, 25 Sep 2009 08:38:00 GMT

Joel, in his inimitable way, posted the flame bait of all flame bait posts yesterday, explaining the role of the Duct Tape Programmer.

To my surprise, the Twitterverse started to reverberate with commentary, but weirdly, almost all of it was very negative about the post, many claiming that Joel was just creating straw men and knocking them down, or was characterising developers who care about quality as being poor developers.

I was surprised, because I read the article totally the other way around - and had said so earlier in the day to someone on Twitter, saying I largely agreed with it. It seems that there were two ways of reading the article, and mine differed substantially to most people I follow on Twitter.

I was quite pleased to see Michael Neel post last night in a semi-defence of Joel's standpoint, and I thought the 140 char limit of Twitter wasn't enough for me to post my opinions.

Background

Joel has a bit of a record on this, he isn't best liked by a large proportion of the developer community who see their focus and remit as producing high quality, well tested and highly maintainable software. His original tirades in a podcast about his deep distrust of Test Driven Development and of Unit Testing in general brought uproar. And on that matter he was so far off the mark, it was understandable why he had riled so many people - he really just didn't understand or "get" TDD or testing in general.

But, what he had done, in conjunction with Jeff Atwood, was to deliver a website, stackoverflow.com, that was proving hugely successful, and was being used extensively by all those same developers who railed against his podcasts and blogs.

That website, as Joel and Jeff explained, was not done with TDD, was not well tested (via Unit Tests), was badly hacked together in places, and used "old" technologies like TSQL over "modern" solutions like ORMs

Whether Joel "got" TDD wasn't important, what he and Jeff had done was to ship a product, get it to market, and build upon that release to make stackoverflow the massive success it is today. It's so successful, that soon after stackoverflow was released, they used the same code and model to release other Q&A sites, and they continue to do so.

The Duct Tape Programmer

So with all of that in mind, it was sort of obviously flame bait for Joel to make a post that basically boiled down to "shipping it way more important than fancy patterns and great tools" - he certainly has a way of putting such a simple message that can antagonise the meekest of individuals (and some not so meek).

It was a terrible choice of label, but I guess "Pragmatic Programmer" had already been taken.

Time, Quality, Functionality

I read the article slight differently - I read it as an urge to maintain your highest priority as shipping the product your business wants you to ship.

There are three factors involved in a project (if you exclude Resourcing, which falls under Brooks Law):

Time
Quality
Functionality (or Scope)

If you try and maintain all three factors as fixed, your project is doomed to fail. Something will, unless you are really lucky, break. And then one of those factors will suffer, and it will be luck as to which it is. At best you can fix two of those factors, and then the third has to be allowed to slip when the unexpected happens (as it always does).

But, the important part here, is that which of those factors is fixed, and which is allowed to slip is NOT a developer decision - it is a business decision.

Much of the outrage about Joel's article stated that he isn't a developer - and they are correct - he is a business owner, and prior to that a project manager. His focus is therefore on shipping his product, and inevitably from that perspective, the instinct is to fix Time and Functionality (ship as soon as possible, with as much functionality as possible) and to let Quality suffer to achieve that objective.

This is a TOTALLY VALID business decision - it is just one that developers don't like, it offends their sense of code quality and high standards.

Why Do Developers Care So Much About Quality?

I'll start with the cynical view: development is only fun while you are trying new things, trying to perfect a tricky algorithm, or make some great new OSS component do some magic. By nature, the developers who care deeply about quality are also perfectionists, they like to make sure they have done a good job, and have written the best code they can.

Go on, admit it, if developers just did routine coding all day long, our jobs would be boring. We like new toys, new language features, we like to play.

And we know that better code will be easier to maintain - and developers HATE bug fixing duty, so we want to ensure we don't have to do much of it.

To a developer, the most important factor in any project is Quality, followed by Time (because we know our bosses will be mad if we miss our deadlines), but we are happy to slip Functionality because it matters little to us if a new feature isn't included in the release.

Fundamentally We Are Not Aligned

Developers and business owners are not aligned in their goals - we prioritise the three factors very differently, and for different reasons.

Only two things can happen here, your developers need to align with their business objectives, or the business needs to align with the developers - or they can both compromise.

Agile is all about the compromise, making both sides aware of the issues involved, and then coming to an agreement, but fundamentally - it is STILL A BUSINESS DECISION.

While it is the responsibility of a professional developer to explain all the problems they see with low quality code, and to make their boss aware of all the potential future issues this may cause, it is also their professional responsibility to go with the decision their boss makes.

They have one other option - resign and go somewhere that is more closely aligned with their values.

Ship It or Ship Out

Whatever your opinions of Joel and Jeff's understandings of TDD, Unit testing, maintainable code or just code in general (and I'm sure yours aren't too far from mine), you have to admit they delivered a product to market (many in fact), and have turned it into a huge success. They may have to pay for that speed to market later, but that was a business decision they were willing to take.

Never underestimate the value to a business of being fast to market - there is a risk the product will be so bugged it will backfire - but there is a chance they could steal the march on their competitors, and gain critical traction. Business is all about assessing risk, that is what a business owner has to do, and if they do it right their business flourishes.

If your boss wants a product shipped by a particular date, and that is their highest priority, you have a responsibility to ship it. If they won't sacrifice functionality to achieve that goal, then code quality will have to suffer - in that case do the best you can do with the restraints you are under. And if all that fails, quit, go somewhere that appreciates your values and dedication to your art - you do not have a right to hold your employer to ransom.

So, Ship It or Ship Out.

Make It Easy To Refactor

Jak Charlton — Mon, 21 Sep 2009 15:17:00 GMT

I'm currently looking at a code base that could do with some refactoring, overall it's a pretty good code base, and the test coverage is pretty good too. But looking through it gave me some ideas on what makes a code base tricky to safely refactor. Here are some tips to make sure your code is in a good state to be refactored with safety and confidence.

Provide Good Test Names

Test methods should have good, meaningful names; this is of course exactly the same as all code - naming is very important to give context and meaning. I am looking at a lot of tests that make sense to the original developer, and within reason make sense to me, but they do not properly describe what the test tries to do and what it expects.

A test named "ValidHardware()" is less than ideal for telling me the intent of the test. If the method had been correctly named it would have been called:

ensure_that_hardware_detector_returns_true_when_valid_hardware_is_present()

Now you tell me, which is easier to understand? And which is going to give me more confidence in refactoring that hardware detector?

Make Sure Variables Are Well Named

Probably blindingly obvious given the above section, but if you use poor variable names, I am probably going to waste a lot of time trying to figure out exactly which thing is which, and worse still, I am probably going to miss subtle bugs.

Cover The Right Things

Beware of high coverage that does not actually test either all the aspects of a component, or that makes a lot of assumptions.

Getting 100% coverage is likely to be impossible, and probably your return on investment over 75% is going to be low, but don't achieve the 75% by testing the trivial and easy to test things only - it's going to be those tricky to test components that will cause most bugs when refactoring.

If something is tricky to test, perhaps you have the wrong level of abstraction applied, or perhaps the test is trying to tell you something. Skipping it and going for some simpler tests may pay off in the short term, but in the long term will hurt you.

Avoid Fragile Tests

There is a lot of debate between the "classicist and mockist schools" about what to test, and personally I veer towards state based testing for most components - but regardless of where your personal impulses lie, avoid creating fragile tests. Any test that is likely to break with trivial refactoring could be considered fragile, and those tests are going to cost you a lot of time and cause many potential bugs when refactoring.

One of the main causes of fragile tests are an overreliance on mocks, and specifically using mocks over stubs. By all means mock where necessary, but try to limit mock usage to very specific test cases, and try to favour stubs over mocks and favour "Setup.ResultFor" over "Expect.Call" type semantics.

Avoid [Setup] and [Teardown]

Yes, I know they make it easier to create lots of similar tests, but they make your tests fragile and interdependent.

James Newkirk (who was on the original NUnit team) made a good blog post on this when he moved to writing xUnit, where Setup and Teardown were removed.

They make it hard to read your tests (you have to read code in two or three locations to understand what the test does) and can cause Setup/Teardown getting too many responsibilities, leading to fragility or code bloat.

Either switch to setup within the individual test, or favour a Context based testing approach such as with BDD

Conclusion

Tests are code, and code has to be maintained too. Make sure your tests are of a similar or higher standard than the rest of your codebase, make sure they test the right things, in the right way, and you will make your life much easier, and that of the guy who comes after you much easier too

The Largest Wicked Problem in Software is Politics and Management

Jak Charlton — Tue, 25 Aug 2009 11:40:00 GMT

I previously wrote a post stating that Software is a Wicked Problem, and indeed it is.

There is a bigger Wicked Problem that affects software projects though, that of Politics and Management.

While there are at least technical solutions to most software problems, and even wicked problems can be cracked eventually, it is a much harder and more time consuming issue to deal with political and managerial problems, and they largely turn out to be wicked problems in of themselves.

And, as with all wicked problems, it is faster to fix the problem than it is to estimate how long it will take to fix - it also happens that with politics and managerial issues, it is often nearly impossible to estimate, or to actually fix the problems, so software developers spend most of their time trying to work around these problems rather than addressing them.

http://en.wikipedia.org/wiki/Wicked_problem

http://www.poppendieck.com/wicked.htm

Software is a Wicked Problem

Jak Charlton — Tue, 04 Aug 2009 13:06:00 GMT

A Wicked Problem, roughly speaking, has a number of properties:

The problem is not understood until after the formulation of a solution.
Wicked problems have no stopping rule.
Solutions to wicked problems are not right or wrong.
Every wicked problem is essentially novel and unique.
Every solution to a wicked problem is a 'one shot operation'
Wicked problems have no given alternative solutions.

Software development is largely a Wicked Problem, or in headline terms:

It is faster to create a solution to the problem than to estimate how long it will take to create that solution with any degree of accuracy, and the estimate will still be wrong.

http://en.wikipedia.org/wiki/Wicked_problem

http://www.poppendieck.com/wicked.htm

We Are Not Doing DDD – Part Two - CQS

Jak Charlton — Mon, 22 Jun 2009 16:32:58 GMT

From the beginning of my current project we have been working under some horrible constraints, many imposed by legacy systems, many by late decisions that would have speeded things immensely if made earlier, and many imposed by decisions that are outside of my control.

This lead us early on to make decisions on architecture that eliminated any possibility of using DDD heavily, and as I mentioned in my original post about this project it is also essentially a glorified CRUD system - we read data from databases, we modify it a bit, and we put it back again.

What We Are Doing – Command Query Separation

The first architectural choice that I made, and in hindsight got accepted with relative ease, was to employ CQS to simplify the system.

Earlier projects had suffered quite badly from being abstractions of legacy systems with new functionality bolted on. As

far as possible I wanted to eliminate this problem from our project, as I was all too aware that this was an easy trap for the team to fall into, and would burn a lot of development time very fast.

To make the system simpler I chose to separate our query mechanisms from the "domain" type stuff, our commands and data writing/updating code.

CQS is a pretty simple concept. One path through your system is responsible for querying of anything much more than "Get By ID" type calls. This side of our system is responsible for reading data from legacy systems, reading data from legacy web services, and for reading data from our application database.

Querying

The Query side deals largely in DataTables, DataSets and a very very limited number of DTOs. This data is largely and essentially used for display purposes, it is the contents of search results, the results of postcode lookups, the information to populate dropdown lists and tables.

The Query side of CQS does not need strongly typed entities, nor does it require strongly typed DTOs - as it is largely ad-hoc data maintaining these entities and DTOs would consume a disproportionate amount of development time for something a DataTable can deal with more than adequately.

Our Query side actually uses the Query Object pattern, most of which encapsulate a call to an Oracle DB, some of which wrap simple NHibernate criteria and a few of which sit around web services. These are fronted by a WCF facade, which allows us to put these queries out of process, and to cache them aggressively should be need.

What we do not have in the system anywhere, and especially in the Query side, is any big entity model – we have two very small entity models (5 entities in total) used for very specific functions within our application.

Command

The Command side of the CQS equation is a little different. Firstly, it deals with anything that is essentially a request to "go do something". In our case these are commands like "SaveDraft" and "CancelPolicy"

The Command side can write data to the databases, the Query side is not allowed to.

If we were doing DDD, this is where our Domain would sit - and in fact we have been referring to this as "the domain" for most of this project. But, we aren’t using DDD and we don't really have a Domain Model. It is however where most stuff we could describe as "logic" lives. This is primarily a bunch of application services, most of which operate in response to messages fired over MSMQ using NServiceBus.

This side of the architecture has a number of listeners that are responsible for pulling messages from various MSMQs and for dealing with the specific requests, for example pushing data back to our legacy systems, for logging, and eventually for notifications.

Benefits

Early on, the use of CQS paid for itself easily. By removing the complications of maintaining read and write models together, we quickly managed to get working prototype up and running, with no need for a "domain" or entity model up front.

As the project has progressed, this decision has helped simplify decisions that otherwise would have involved structural or fundamental changes to the architecture – but more importantly, it has helped all of the developers think in terms of a command driven system, rather than viewing the system as CRUD over a database.

Commercial Suicide - Integration at the Service Level

Jak Charlton — Tue, 19 May 2009 07:03:00 GMT

In my previous post I described the problems with trying to integrate your organisation at the database level, and the fallacies surrounding the idea of the One True Authority Database. I also alluded to this being a problem with services too.

When you try to create a monolithic and authoritarian database that knows all, sees all, and brings everything into a single coherrent whole, you are heading down a slippery slope. Different applications have different requirements, and even within the same job role, the meaning of Customer can have a totally different meaning in two different contexts. I also argued that data has no value without context, and the context is the application that is meeting the business requirements, be these user interaction, system interaction, or reporting.

SOA to the Rescue!

I'm not arguing without ample evidence behind me here - the idea of integration at a database level was one that was well discredited many years ago. I grant you many organisations haven't got the message yet, and that many vested interests still keep trying to make the magic solution work, but fundamentally a long time back we in IT recognised that the One True Authority Database (OTADB) was a "bad thing" - and this lead us to the promised land - initially Web Services and then on to Service Oriented Architectures.

SOA is an attempt to resolve the OTADB issue - take all those database calls, and even calls to different databases, and unify them through a set of coherrent and cohesive services - SOA can provide us with the abstraction over out data stores that is required - and now we can have the One True Authority Service Layer.

With SOA we can define services to wrap our databases, services to update data, service to read and query data, and even services to unify those "different" Customers into a single coherrent and canonical data model.

And that's where it all goes wrong again.

Anti-Pattern: The Canonical Data Model

Back when our database guys were trying to integrate all this information, what they were trying to achieve, but maybe didn't realise, was a canonical data model - this is the "holy grail" of SOA. Martin Fowler describes it thus:

How can you minimize dependencies when integrating applications that use different data formats? …Design a Canonical Data Model that is independent from any specific application. Require each application to produce and consume messages in this common format

Sounds perfect! We now have a way of abstracting our systems from each other, and our canonical data model will pull this information all together at the end.

So, I'm going out on a limb here, disagreeing with Martin Fowler is rarely a smart move.

Except I'm not disagreeing with Martin Fowler so much as I am disagreeing with those that try to implement his ideas and principles. I don't recollect anywhere that Martin says that all data can be squeezed into this model, or that this model is the "one true view". In fact, later evolutions of Martin's work have included things like Domain Driven Design, which is almost the opposite of a canoncial data model - DDD says that you have a different model for every domain or context, each specifically engineered to solve the problem at hand. Given the choice, I am certainly in the DDD camp here - after all data without context is worthless, and it is the domain (and DDD) that provides the context.

The key and important part of a canonical data model is it provides a mapping between inconsistent formats and differing views of data.

So as an ideal, the canonical data model is fine - in practice it is often a major anti-pattern.

There is still no One True Authority, and SOA really never promised this, nor did the canonical data model - what they promised was enough abstraction to allow applcaitions to work independently, and to fulfill their specific requirements independently. And then to leave it to the service layers to unify this data.

So Service Oriented Architectures HAVE Come to the Rescue?

Not quite - again my argument here isn't so much about SOA, or any of the things around it, but more the people who try to implement it, often without understanding what they are trying to achieve, the problems that SOA or a service layer solves, or even the amount of effort that is required to get a canonical data model right.

The key part here is that a canonical data model is a massive excercise to do right. Essentially it means analysing every applciation, every service and every database within your organisation and extracting their individual requirements and current functionality, and trying to distill that down to a set of common definitions.

While this may be eventually achievable, in practice it is almost never the case. This "holy grail" of SOA is much like the "holy grail" of integration at the database level, eventually it boils down to "the problem is too complex, too fast moving and too indeterminate to define". In development we figured out long ago that we needed Agile to help us move quickly as requirements shifted and changed, but the OTADB and the SOA canonical data model do not allow this to happen, they seek to solidify the design early.

SOA done "right" may well make this problem smaller, but SOA is rarely done right.

Can Domain Driven Design Help?

DDD was largely an attempt to address these problems - by identifying that individual applications and contexts required different data, DDD allowed us to recognise that square pegs and round holes do not match. Eventually with a big enough hammer you may be able to force it through, but there is going to be a lot of effort and a lot of collaterel damage.

Domain Driven Design includes terms like "Domain", "Bounded Context" and "Anti Corruption Layer" specifically to deal with the issues around trying to use a "one solution fits all" hammer to crack what is quite a simple nut.

DDD is largely unconcerned with what other things Domains talk to, or even how, all that matters is that within a Domain there is one logical view of the world, tailored to the requirements of that specific Domain. Beyond that Domain, there may be a large central database, there may be a perfect canonical data model, but at application level, we have the Domain, Bounded Contexts and Anot Corruption Layers to protect us from all of that.

DDD gives us a loosely coupled way of thinking.

So SOA is Evil and the One True Authority Database is Evil?

Well hardly, but unfortunately we live in an imperfect world, and the people implementing these noble ideas are often far from perfect. They do not have mystical powers of future vision, and they do make mistakes.

By creating properly decoupled systems, and in this I certainly do not mean "systems that use web services", we can isolate our applications from external change and influence, and ensure we meet the requirements we have today, and can react quickly to future requirements as they appear.

Big Design Up Front is still BDUF, whether it is done at a database level, a service level, or in a canonical data model.