Jak Charlton - Insane World : Rants

Avoiding Prescriptive Requirements

Jak Charlton — Thu, 09 Feb 2012 03:50:00 GMT

Core Requirements

Requirements should be given as the Core Requirement, and avoid the common pitfall of providing Prescriptive Requirements

As an example, a core requirement in an insurance system may be:

As a user of the Broker system, I want to create a Settlement Batch of Eligible Documents across Insurance Policies

This is eminently testable, is clear and unambiguous, and has real business value, and it meets the general INVEST principle of good user stories (which is also a good maxim for any form of requirements):

It is Independent of other stories
It is easily Negotiated as to priority and value
It has real business Value
It is small enough to easily Estimate with some degree of certainty
It is of an appropriate Size for easy estimation, planning and prioritisation
It is able to be Tested both manually and in an automated fashion with clear and unambiguous pass or fail criteria, you can either do it or not

The stated requirement is not prescriptive, it does not specify how the user triggers this action, what data they need to provide, how they select the documents, nor does it specify what a "user of the Broker System" is within this context, nor what a Settlement Batch is, nor what an Eligible Document is. This means the requirement can be documented very early on, with little analysis work needed, allowing very quick comparison with other requirements, and very quick verification from the business as to the validity of the story.

Later this placeholder can be expanded in discussions between the technical teams and the business to include the actual definitions and scenarios that define its acceptance criteria:

The user must have the “Batch Creation” permission
Eligible Documents are those which are awaiting Settlement
A Settlement Batch is a group of Documents that will be dealt with together by the Claims Settlement system, so as to save managing on an individual policy level

This would lead to a requirement with maybe a dozen test scenarios giving all the variations of user and statuses that would allow or not allow the story to pass the test. All of these stories and scenarios are now easily tested with automated and repeatable testing.

At this point it would probably be obvious where within the UI this functionality should surface, and the design of the UI interaction can be worked out. This UI work is actually a very specific development skill within its own right - there is significant complexity in designing UI interactions that mirror both the technical system and the user’s mental model.

Through all of these steps, the original core requirement has not been changed, though its implementation and business logic may have evolved, changed and adapted to line up with other functionality.

Good requirements are merely a placeholder for a later conversation. Their level of completeness or accuracy only dictates your certainty in implementing them accurately and efficiently, but does not preclude moving forwards, and does not hold development up waiting on “complete requirements”.

As granularity increases, so does complexity required to understand, verify and implement, but oddly so does instability. By starting with core stories, you maintain a fairly stable set of requirements, and only deal with complexity when you need to, and suffer the instability only on individual scenarios.

Which brings us neatly to development

It is fairly well established now that development must be iterative within short cycles to increase its chance of being successful. Early feedback and revision saves massive amounts of time and effort correcting incorrect assumptions and poor translation.

The sooner a story enters development, the sooner a basic version of it can be demonstrated to business users, to verify against their assumptions, the faster it can be revised and the more accurately it will reflect the actual requirements. In addition it will expose underlying technical constraints and limitations earlier on.

What is Domain Driven Design?

Jak Charlton — Mon, 16 May 2011 02:15:00 GMT

"... the key to expert performance in many fields is domain knowledge rather than intelligence."
Don Reinertsen

Domain Driven Design is a software development methodology, intended to achieve a software system closely modelled on and aligned with real business processes.

Traditionally development tends to be a technically led process, where requirements are passed from the business to the development teams, and then the development teams go off and produce their best guesstimate at what the requirements said.

In a waterfall approach this leads to large requirements documents that are diligently collated, analysed, reviewed and approved. These documents are then given to development teams to turn into working software.

Agile approaches can also accept the requirements documents that waterfall tends to produce, but for them to actually progress forwards they must be broken to small tasks and stories which can then be queued ready for development.

Domain Driven Design to a large degree steps back from these two distinct ends of the spectrum, and looks at how the requirements are gathered in the first place - if you like, it bridges the gap between doing everything up front and everything at the last minute.

DDD understands that requirements are never 'done', but exist as a living document. More importantly, the living document in question is actually the software itself - all other documents are an artefact and reflection of the code.

As the software system develops and grows, deeper understanding of the problem grows too - DDD is all about the discovery of the solution through deeper understanding of the problem.

Where DDD really differs though is that it sees the software system as a reflection of the business process, an enabler rather than the driver. DDD is deeply concerned about the business processes, business terminology and practices. Technical concerns are largely secondary and a means to an end.

The Ubiquitous Language is at the centre of DDD - this is a shared and growing language. A negotiated language derived from the business terminology and enriched by the development teams. If a business person doesn't understand a term in the UL, then the chances are the UL needs an evolution. If a technical person doesn't understand a term in the UL then the chances are they need a conversation with a Domain Expert.

The Domain Experts are the second essential component of DDD - people who deeply understand the business domain, and the business itself. These people are integral to the development process. They may not be required "full time" like the traditional Product Owners of some Agile methodologies, but they must be accessible continually, and be prepared to invest time in the process. Domain Experts are not to be considered outsiders, but as the core of the DDD process - they are as much a part of the development team as any developer or tester.

DDD doesn't begin and end - it is a constant process of re-evaluation, refactoring, remodelling and redesign - every conversation should bring you to a closer understanding of the problem. At no point is DDD 'done' - it is always there; the Ubiquitous Language grows and develops, the Domain Models change as understanding changes, code is restructured and refactored to better represent that understanding.

Artefacts will come and go, but only one really matters - the code base. It is the only expression of the solution that is free of abstraction. While documents may try to explain or describe the system, only the code does so without losing information. This means that in DDD the code must remain of high quality, be clear and expressive, be free of technical abbreviations or jargon, and as far as possible should make at least some sense to a Domain Expert when explained to them.

Clever code doesn't work in DDD, nor do magical processes or "you don't need to know" components. Domain Experts shouldn't need to become developers to understand what the key parts of the software system is doing for them. They also shouldn't need to think in terms of databases or batch jobs or other technical concerns.

DDD is the ultimate expression of Agile - it is about dealing with a constantly shifting and evolving set of requirements - because as anyone who has ever been involved with a software project knows - it is very rare for a requirement to remain intact from the beginning to the end of a project, and almost impossible for it to do so as the business grows and changes.

Through constant conversations, DDD will guide you to an ever more accurate software representation of your business process.

Trust, Honesty, Openness, Cooperation and Friction-free Tooling

Jak Charlton — Tue, 03 May 2011 22:59:00 GMT

There is an inherent danger in using electronic tools to manage a process like software development, and one that almost always comes true - the tool defines your process, it doesn't enable it.

When people encounter limitations in the software tool, they begin to accept that these are limitations of the process and these become ingrained in your methodology. If the tool doesn't support the "thing you want to do" then it is far easier to change the "thing" rather than rewrite or change the software.

Were any of the electronic tools to have got the process "right" this might not be such a bad thing, but it would be reasonable to assume that in a field as adaptive and evolutionary as software development there is no such thing as the "right" process for an individual team, let alone for such a widely disparate ecosystem as development.

And this conflict between our tools and our actual practices and processes is what we refer to as "friction", and as physics defines it;

Friction is the "evil" of all motion. No matter which direction something moves in, friction pulls it the other way. Move something left, friction pulls right. Move something up, friction pulls down. It appears as if nature has given us friction to stop us from moving anything.

We aim for "friction-free" tools to allow us to better get on with moving things, specifically developing software, which is a constantly moving and adaptive process.

Some tools aim to allow endless configuration or programmability, in their objective of being a universal solution. These tools tend to actually make the problem worse, by either leaving you bogged down in configuration or customisation, or leaving the tool so generic that it solves no problem at all.

Of course, some tools are better than others. Lightweight tools are obviously going to create less friction as they have less moving parts, and less touch points with your actual process. But these are the ones that some see as "lacking features", or as being "too simplistic" - but it is these things specifically that enable the process of development to continue.

And ultimately, these tools expose a greater problem - there is usually a disjoint between project management and the development teams here. A lack of trust and visibility leads managers to seek to control or monitor the process more tightly, and these tools promise to allow that to happen. Surely if we can tie everything into a database or a spreadsheet or something similar, we can then see exactly what is happening, who is doing what, when things will be done, and how long they will take?

But it is not the tool that enables this to happen - it is serving here merely to try and enforce control upon people - which as any good manager knows, enforcement is a far blunter instrument than cooperation.

What enables proper management of the software development process is trust, honesty, openness and cooperation. Tools should reflect and enable those objectives, rather than try to enforce them. And most importantly, they should stay out of the way of the development teams to allow them to get on with their real job - developing software.

CQRS - The Cult of Shiny Things

Jak Charlton — Fri, 29 Oct 2010 00:12:00 GMT

CQRS has become another casualty of the buzzword culture and the cult of 'shiny things'

What started as a collection of some reasonably good principles has now turned into an almost religious mantra for some, with more and more outrageous claims, and almost no basis in fact or experience.

Scalability

Claims like "almost infinitely scalable systems" associated to CQRS are ludicrous, especially as none of the currently "near infinitely scalable systems" use CQRS or have probably even heard of it. The largest systems on the planet have no need of CQRS, they use many tricks and techniques that have been associated with CQRS - but they don't use it, nor are they ever likely to use it, because when you have really large systems, things like CQRS become totally irrelevant.

The scaling benefits of CQRS come from understanding eventual consistency and messaging - concepts that were around long before CQRS, and that will be around long after it too.

Task Based UIs

Well, yes CQRS may require a task based UI, but task based UIs were again around long before CQRS, and will again outlast it. This is just a different way of thinking about how people interact with computers - this is a UX issue, not a technical one.

In fact, if anything, CQRS can be a major cause of issues here - when you really have a CRUD style system, CQRS is going to cause you all sorts of issues you will need to resolve. And much as I dislike CRUD based UIs, the reality is we have abused users to work this way in many cases - so sometimes that is just how we have to make our next UI work.

Side Effect Free

Another aspect of CQS (and by extrapolation CQRS) is that functions employing CQS are side effect free - you don't accidentally introduce subtle bugs by writing and reading in one operation.

And while this is nice in theory, the scale of system where this preventative measure would have benefit is the one where this problem is just as easily solved by many other simpler means, like just writing smart code. Larger systems have so many points in them where read and write are already being mixed, CQRS here just lends an illusion of safety.

Denormalised Read Models

A valuable benefit of CQRS is the concept that the views have specific read models that are denormalised from the (presumably) complex domain model.

This has a number of benefits, though these are largely boiled down to improved efficiency of querying (and action that tends to happen frequently). These models can also be extrapolated out to provide reporting and MI models too.

But again, we have been doing this for many years already, and usually to account for overloaded database servers that we don't wish to query against, particularly with MI.

And querying these days, even with horribly complex joins on highly complex models is becoming faster and faster. Push querying to a slave server, and you gain all the benefits of a rich model, and fast queries. And for even more speed you can look to NoSQL solutions - which are effectively based on the principle that you denormalised your data anyway.

Performance Considerations

Part of the theoretical benefit of CQRS is that your queries will be faster as your denormalised datastores are updated at write time, so you don't have to do horrible joins on queries (which in theory on most systems happen far more frequently than writes)

These days, the performance of relational databases and especially NoSQL databases is advancing rapidly. The new breed of databases takes things like sharding and replication very seriously, and as first class features, not afterthoughts like the databases of old did.

Replication in MySQL takes roughly 0.0002 of a second (see http://nr-content.s3.amazonaws.com/railslab/videos/17-ScalingRails-Scaling-Your-Database-Part-1.mp4 at 6 minutes in) - compare that with how long a message based approach like CQRS will take to push that to your secondary datastores and you are talking orders of magnitude faster for the dumb replication approach. Apparently it operates at this speed for up to 10 slave databases before degrading.

Now, think about this bit, the 37 Signal systems (a huge system) operates on a single MySQL instance running with 105Gb of memory caching. And 37 Signal could be considered the archetype of an Active Record approach, after all it is the reason Ruby on Rails exists. Are you writing a system that big?

Maybe you are so let's examine a larger system - Wordnik stores over 12 billion documents, 3Tb of data per node, 500k requests per hour and 4 times that at peak load, and yet their fetch time is around 60ms. (http://blog.wordnik.com/12-months-with-mongodb)

Let's not even begin to discuss Twitter, Ebay, Facebook or other similar "mega systems" - they still respond in milliseconds with tens or hundreds of millions of users - think that's down to CQRS?

Dealing with Complexity

Yep - CQRS deals with complexity to one degree - it forces you to explicitly think about what each call is doing - "Is this writing data or reading data?" - and that thought process is a good thing.

But it introduces another layer of complexity that is more subtle and harder to deal with - the replication to secondary datastores is complex in itself - it effectively needs to do that horrible join you are trying to avoid in your queries so that it can update your denormalised tables. That in itself isn't simple.

But it also needs to be able to maintain those secondary datastores if, for example they are corrupted and need recreating or if they miss some messages. In normal replication this is trivial, drop the slave, add a new slave - wait a short while - someone already wrote all that for you on a database system - you are going to need to write it all yourself with CQRS.

So Why Have I Implemented CQRS Before?

Well, at the time I first implemented CQRS I had two reasons for doing so - and oddly neither had to do with any of the aforementioned technical aspects. In fact, one was a technical problem and one was a people problem.

Firstly this was nearly two years back, and the system that we were writing a layer over was a very old and crumbling legacy database system. It had no real normalised schema, and was open to abuse from just about every development team in the organisation without warning to others - it was also horribly prone to outages.

One project had just "failed" (though I'm sure some would say it was just a lesser success), primarily due to trying to put a domain model on top of the legacy database.

So, the problem we had was to create a disconnected system that could operate independently of the legacy system, but also use the legacy system as it's true authority for the information it used. Because of the horrible complexity in the legacy system, using CQRS allowed us to write what were frankly quite horrible SQL statements for reading data from legacy, while using a pseudo-document database for the daily operations. We then used horrible SQL statements to replicate this data back into legacy.

Of course, what we were really writing was a standard message based architecture, but the term CQRS encompassed that quite nicely.

We could have approached this without the CQRS badge - but it served as a useful political tool - a number of the existing team had heard of it and were quite enthusiastic, and management would be confounded enough by it to not get in the way too much. It also broke the mental model the development teams had with "put an ORM over the database".

So the secondary benefit of CQRS here was a people problem - it was a new buzzword - it got new enthusiasm from a team that was demotivated from the previous project - and more than anything - that motivation was going to be the key to the success or failure of the new project.

So Why Have I Presented On It Before?

Well two reasons really, the first being that I have always tried to dispel the mythology around CQRS and to explain the real benefits of a flexible and scalable architecture - it just so happens that people want the buzzword explaining - you have to admit it's a pretty vague and fuzzy term with a myriad of definitions and explanations.

The secondary reason being, as my last presentation at DDD Sydney explained - my objective was to make you think. There's a lot of good things hiding under the CQRS banner, and I really would like people to think about better ways of writing software.

Conclusion

In the few years since CQRS was "invented" it has got a bit of a cult buzzword status. It is talked about as though it fixes a multitude of problems.

But in that time, NoSQL has really taken off. RBDMS' have got far more powerful. And memory and processing power has increased significantly. Database caching is exceptionally impressive. REST gives us proxy caching for near enough free. Memcached and Velocity make light of massive data loads. We even have the nefarious "cloud" now, where processing power and memory is cheaper than the effort involved in maintain complex architectures.

Scaling systems is now a pretty commonplace business problem for most of the startups taking off -and they don't need CQRS - there is a very good chance you don't either. We are past that point in time where we had to prematurely optimise things like database queries.

If it's about thinking differently, then I can tell you all you need to know about CQRS in a simple paragraph:

Learn to think of systems as being inconsistent at all times, no matter what illusion they present of 'real time'. Learn to embrace messaging, REST, caching, replication, sharding, and a hundred other techniques to solve these problems. Embrace user centric task based UIs, embrace strong UX, embrace strong domain driven design, embrace business led functionality and delivering systems that match real requirements. Embrace thinking differently.

CQRS is no silver bullet, in fact it has almost now become a solution to a problem that doesn't exist any more. If you can give me a problem that CQRS claims to solve, I'm fairly sure I can solve it more easily than I could have done 2 years ago when I started using CQRS. I'm sure I'll still use a lot of the things that CQRS encapsulates, because they are after all very good principles - I'm just pretty much over playing buzzword bingo!

Ruby Is Scary

Jak Charlton — Sat, 31 Jul 2010 01:55:00 GMT

Why is Ruby on Rails all the rage at the moment, and why do a lot of .NET people seem so defensive?

Undoubtedly, there is a buzz in development right now, things are changing rapidly, possibly more rapidly than they have for a good number of years. New languages are sprouting up all over it seems, with PHP and Python gaining massive popularity, F#, Erlang and Haskell suddenly being talked about as mainstream, Javascript evolving beyond that thing that makes "onClick" work in your browser, and Ruby (and especially Rails) becoming "flavour of the month" for web development. Ruby is even starting to invade the .NET mainstream space with evolutions like Rake, Cuke4Nuke and Nu muddying the divide that there once was. We even have IronRuby to make Ruby run on .NET

And yet, many .NET people seem defensive, as though their safe world was being invaded by a parasite.

I find this quite odd. As a developer who has been around for far too long, I have seen more than my fair share of languages and platforms come and go, none of them was perfect, and I would venture to say none of the ones we have now are either.

It would appear that a large proportion of the uncertainty is in the C# development 'community', though that may be as I don't really frequent circles that use VB.NET (and I'm fairly sure they are still busy being defensive about C# and probably haven't noticed that's *so* last year :) Somehow the introduction of new ideas is rocking the C# type safe world.

Some people seem to be concerned about Ruby itself, some seem to have concerns about the lack of a statically typed safety net, and some seem to think that "the Enterprise" needs something more professional than toys like Ruby and Rails.

To The Static Typing Stalwarts

Yes, it's nice to have that compile time checking that static typing gives you, it gives us all a warm sense of fuzziness knowing that the compiler told us that this code would run just fine when we dropped it onto our web server.

Yes, we appreciate the great benefits that this has for things like Intellisense and for easily discoverable APIs, and even for automated documentation.

And yet, none of these things seem to bother the Ruby (and PHP/Python/Javascript) crowd too much. Why is that?

I would suggest that the static typing benefits a language like C# brings are illusory to a large degree.

The compiler isn't telling us that this code will run correctly at all, it's actually just telling us that the CLR won't throw our code out before the application even starts up - but correctness is so much more important than that. The safety net static typing provides us with is just one part of the equation, and possibly the least important part - we really should be enforcing correctness upon our code.

Correctness is verified by good automated testing, and static typed languages start to come back and bite us here - we trade that compile time security blanket for a much harder experience testing our code. Sure, tools like NUnit, xUnit, Rhino Mocks, Moq, SpecFlow, StorEvil et al ease away our pain, but still our application code is littered with constructs that are pretty much there to support SOLID (or SOLID is required due to the constraints of type safe OO languages), and they become most obvious when we start testing in languages like C# (I am *so* over Interfaces, IoC, Covariance, Contravariance and Generics !

The testing experience in Ruby is a world apart from in C# and .NET, it is just easy and simple ... so simple in fact that it is almost harder to not write automated tests in Ruby - and that's how it should be. In C# testing is often convoluted, complex, fragile and just plain hard work - I can't think of the last time a C# developer described automated testing as fun ... and yet, in Ruby, it just seems like FUN!

And yes, Ruby without automated testing leaves you open to all kinds of runtime errors that C# just wouldn't - so in Ruby, you just make sure you test, and not only test your code will run, but also test it will run *correctly* - there is no safety net, but the trade off is you are ensuring correctness of code, not just compilation of it.

Intellisense, sure it's nice, but as RubyMine and even Visual Studio (under Javascript) demonstrate, you can do some form of Intellisense in dynamic languages, you just can't always be certain what may or may not be available - but this is getting better. And honestly, is discovering which method you are meant to call though typing "." all that great an idea anyway? Shouldn't you as a competent developer know *in advance* what you wanted to call, rather than guessing it as you type?

To The Enterprise

Sure, Ruby doesn't have the track record that C# has, nor does it have the exposure, or market reach, or backing of a company like Microsoft.

But, is that what is really important?

I would bet that a large part of the appeal for statically typed languages in the "Enterprise" is actually a need to deal with a much lower skilled developer base overall. Compile time safety means that your developers don't have to be top notch, the compiler can make up for their weaknesses (and in the case of C# and .NET, Visual Studio 'drag and drop' tooling is trying to make up for a lot of the rest)

Enterprises tend to see dollars and cents, not people and value. So anything that lets them hire cheaper developers to churn out more product must be good.

Again, I would argue this is a false economy. Sure that team of 30 developers may give you a warmer fuzzier feeling inside than a team of 4. And sure, the quantity of code those 30 can type out will probably be much more.

But this misses some real issues around productivity and value - a 'good' developer is 10x as productive as an 'average' developer, and more importantly, the 'average' developer will never be able to create the same quality level as the 'good' developer, even given a near infinite amount of time. No matter how much time a hobbyist painter spends with oil and canvas, he will never create a Leonardo. And as a Jackson Pollock will amply demonstrate, sometimes less is more.

So are you measuring bodies or productivity? Surely it is the end result that matters, not the comfort factor of people looking busy.

And, when you get a good developer, and let them have the freedom of a dynamic language, they can spend less time fighting compile time safety and language constructs that exist almost solely for that type safety, and more time concentrating on business logic and the end product.

In fact, the best argument for why Ruby is not suited to the Enterprise is ... the Enterprise probably already has a large investment in one platform and one core language (or maybe two), and the cost of retraining all those developers, support staff, renegotiating supplier contracts, etc etc is just disproportionately expensive. Lethargy is rife in the Enterprise, and monoliths take a long time to move - sure it's easier for startups to hop on new technologies and run with them, and often it's why they quickly challenge the status quo. This isn't really a technical issue, this is a political and management one.

Maybe Ruby and Rails will gain traction here, maybe not. Things like functional programming are certainly starting to grow roots in financial services, and things like Javascript are prevalent in web apps in the Enterprise so maybe dynamic programming in things like Ruby isn't so far off. It just takes one small faction to show how well they solved a particular problem with a new language or platform to plant that thought.

So Should We All Switch To Ruby Now?

Of course not, and maybe this is what the defensive people in .NET miss. The Ruby people aren't saying "you guys have it all wrong, convert now", they are just saying "hey, I found a really cool way of solving this problem".

Ruby is about choice, it's another tool in your arsenal. As are Javascript, PHP, Python, Erlang, F#, Java, Groovy, C++ and a host of others.

There is no "one right path", so let's all stop pretending there is.

We don't need to be defensive about any one language or platform, unless we are so unsure of our abilities as developers that we cannot learn and adapt - and if that is the case, maybe a new career is a better idea?

As my recent presentation at DDDSydney said ... My Objective Today was to make you think ... "Maybe There Is A Better Way" - because it would just be depressing to think we are already doing things the best possible way.

My Objective Today is to Make You Think

Jak Charlton — Mon, 19 Jul 2010 00:08:00 GMT

"Maybe There is a Better Way"

I recently presented at DeveloperDeveloperDeveloper in Sydney, and although my talk was Stuff About CQRS, I opened with the slide

My Object Today Is to Make You Think ... 'Maybe There is a Better Way'

(slides here)

The real focus of this was around how normal people think, and how unlike normal people we developers really are. My role in development is all about enabling better communication, because fundamentally I believe the real value a developer brings to a project is not technical, but is in the way they interact with the team, and more importantly with the normal people they are actually creating software for.

Obviously some of this has roots in DDD, the Ubiquitous Language is obviously an attempt to traverse this chasm that seems to exist between us.

Then I touched on user interfaces, and how they are so rarely designed the way people think - normal people think about their objectives and goals, not in terms of data like we developers do. Inductive UIs are focused on tasks, unlike the traditional data driven UIs that we tend to throw at users - users and people don't think in grids and columns and rows, they think "I want to change my address", not "open my customer record, edit the three fields under address and save to the database" - only a sadist or a developer would think that way.

And finally I touched on things like NoSQL databases, which neatly solve a communication problem - they stop us thinking about How to store information and let us focus on What we are storing and Why.

And lastly I tried to show the link between CQRS and these business problems - how it made you focus on the language, on tasks and objectives and how it let you detach the How from the What and Why.

But most importantly, what I was trying to do in that presentation was to throw some non-mainstream ideas out into the audience, to spark discussion and debate, and to get people to think - Maybe There is a Better Way

If only a few of those in the audience went away and Googled some of the ideas I was talking about, it will be another step towards moving development away from it's heavy focus on technology and technological solutions - and towards a people and business driven focus, where technology is an artifact, not the deciding factor.

Agile is Performance, Feedback, Revision

Jak Charlton — Thu, 15 Apr 2010 02:45:08 GMT

I came across a great video link yesterday, entitled “Performance, Feedback, Revision”. It’s a Canadian rapper named Baba Brinkman covering the theory of evolution and the work of Charles Darwin, and he equates evolution with how he writes his lyrics, Performance, Feedback, Revision.

As I listened to it, it struck me that this is Agile … we develop, we run retrospectives, and we revise what we have.

Evolution may be more like waterfall, and take millions of years to evolve something as fundamentally pointless as the appendix, and in Agile we may look to have Performance, Feedback and Revision inside a week or two.

I can’t quite believe I just showed this video to a client to explain Agile, but it seemed to get the idea across pretty well. I’m considering using it more often, but I promise to draw the line at learning to rap.

But from now on, in big letters across the top of my white boards will go the words:

Performance, Feedback, Revision

Good Developers Write Good Code, Instinctively

Jak Charlton — Tue, 06 Oct 2009 08:09:00 GMT

So my last post ruffled some feathers, but Steve hit the nail right on the head in his commments, and it was sort of the subtext of what I was saying:

All this kerfuffle about "software craftsmen" and "code quality" and "best practice" misses the ultimate fundamental truth :

Good Developers Write Good Code, Instinctively

Let start with a funny statistic ... it certainly applies to drivers, I'm sure it applies to developers too (and almost any group of people) ...

80% of developers think they are above average!

Our logical brains can't help but admit that at least 30% of developers are therefore deluded. Our mathematical brains can't help but admit that 50% of developers must be totally average or below average. And yet, most of us think we are above average developers. I would boldly go further, and put myself in the top 20% of developers - I like to think I at least have a damn good idea who the 20% above me are and why they are better.

If you are a "good developer" you will write "good" code. It will probably use a dozen standard patterns, maybe without you even knowing what they are called, it will have low coupling, high cohesion, be easy to test, easy to extend, hard to modify, avoid duplication and abstract code behind well thought out interfaces. And yet despite all that - you may well have never heard of software patterns, OCP, ISP, DIP, unit Testing, DRY, YAGNI, etc etc etc

Before The Renaissance

There was a time, which sadly I am old enough to remember, when we didn't have all these fancy names, and yet we pretty much did what all the enlightend are doing today.

We may not have had "unit testing", but we had "test harnesses", and we may not have had automated tests running after every build, but we had QA people using our test harnesses to verify our code.

We may not have had software patterns, but we all used factories, observers and singletons.

We may not have has SOLID, but we wrote good decoupled code, well abstracted, easy to test and easy to maintain.

None of These Things Are New

While many like to go on almost holy crusades about these wonderful new inventions, some of us know that these are just evolutions of things good developers have been doing since programming began. Sure they are now formalised, they have great names, they have been refined to be more automated and more efficient - but at the end of the day - good developers already did this stuff.

What concerns me more is how "new" these things seem to some of the holy crusaders. I remember code I wrote in 6502 assembler that had these principles applied (you try writing games in 3.5kb of RAM if you don't apply DRY), I remember code I wrote in COBOL on a PDP/11 that had "factories".

I remember working on a RAD team in an investment bank, long before RAD had meaning. We delivered software fast, direct to the customers, with rapid feedback and weekly releases. We sat with the traders, we were "Agile" and "Lean" long before anyone thought of proper words.

These great software practices were not invented, they were formalisations of things we were already doing.

All Of These Things Are Good

Before anyone decides to rip my head off and banish me to the pits of hell, let me state - all of these practices are good. More over, these practices are (probably) the pinnacle of our current understanding of development practices garnered over the last 30 or 40 years - and we should strive to write code with these things in mind. In 5 years I have no doubt all these things will have evolved, and there will be new best practices. Half of these things only apply to Java and C# anyway - in the world of F#, Erlang, Ruby and Python there are new principles and best practices.

Formalisation of all these practices, principles and methodologies is good - it helps promote understanding and gives us a way of communicating.

The thing is, the real "above average" developers were probably already doing this - and all that code they wrote for the past 30 years is probably still running your bank accounts, processing your taxes and helping keep your aircraft in the sky, precisely because they already did.

Below Average Developers Will Always Write Below Average Code

No way to avoid this I'm afraid - we can talk about the need to raise the bar, to raise the average, to help those below average developers write better code - but fundamentally, below average developers will always write below average code. The above average developers need to recognise this, and understand that they will always feel they are being held back by the lower 50%, and they are always fixing the code of the lower 50%, and they are always doing more than their share.

And one last question... are you sure YOU are above average? I'm not sure myself, in fact my most common comment in interviews is that I am not the best C# developer in the world by a long way, I am rarely the best C# developer in my team for that matter. Am I in the top 20%? Probably by my own standards and measures - that doesn't mean I am by everyone else's measures.

Developer Myopia

Jak Charlton — Mon, 05 Oct 2009 12:07:00 GMT

While some may argue that TDD, BDD, SRP, ahderence to SOLID, use of Patterns, and many other development practices and principles are critical to achieving great software, these all seem to miss some core truths.

Most Software Has Not Been Developed This Way

Yep, sorry to disappoint, but the vast majority of software in production in the world today has had none of these principles and practices applied. Some of the software may have had some of these practices applied - but it is (I would guesstimate) an incredibly small percentage.

And despite that (feel free to tell me I'm wrong) fact, most software works and continues to work. It processes your taxes correctly, makes sure your airplane doesn't blow up on take off, scans your body for cancer, lets you bill your customers, tells you what you are doing tomorrow, and lets you keep in touch with friends and family.

Most Software "Mistakes" Are Not Development Practices

Steve McConnell, he of Code Complete and Rapid Development fame (books everyone in development or management should read) has just published his "Software's Classic Mistakes" list for 2008, and none of these things we developer's like to refer to as "best practice" or "the right way" feature on the list - not in their specific forms, mnor even in their general forms.

In fact - all of the classic mistakes (which cause software projects to be delayed or fail) are process mistakes, they are failures of people, not of coding constructs and practices.

Unrealistic Expectations
Overly Optimistic Schedules
Shortchanged Quality Assurance
Wishful Thinking
Confusing Estimates With Targets
Excessive Multi Tasking
Feature Creep
Noisy Crowded Offices
Abandoning Planning Under Pressure
Insufficient Risk management

We can certainly look at this list and see how Agile, Lean and similar processes can help alleviate many of these, but where do you see "TDD", "BDD", "SOLID" or even "Automated Testing" resolving any of them? The closest we will come is to see "Shortchanged Quality Assurance" as being ticked off the list, but this still doesn't remove the need for real QA on a project, and real QA people.

Developer Myopia

While as developer's we see great value in many of these practices, we need to admit that we are actually a very small part of the equation, and sometimes developer myopia sneaks in - we think that what we do at a code level matters, substantially more than it does in the grand scheme of things.

By all means, apply and practice these things that create "quality" software, but bear in mind, there are much bigger and more important problems to resolve too.

Or to paraphrase (and bastardize) the Agile manifesto: While we see value in software patterns and practices, there is more value in solving process problems first

Ship it, or Ship Out

Jak Charlton — Fri, 25 Sep 2009 08:38:00 GMT

Joel, in his inimitable way, posted the flame bait of all flame bait posts yesterday, explaining the role of the Duct Tape Programmer.

To my surprise, the Twitterverse started to reverberate with commentary, but weirdly, almost all of it was very negative about the post, many claiming that Joel was just creating straw men and knocking them down, or was characterising developers who care about quality as being poor developers.

I was surprised, because I read the article totally the other way around - and had said so earlier in the day to someone on Twitter, saying I largely agreed with it. It seems that there were two ways of reading the article, and mine differed substantially to most people I follow on Twitter.

I was quite pleased to see Michael Neel post last night in a semi-defence of Joel's standpoint, and I thought the 140 char limit of Twitter wasn't enough for me to post my opinions.

Background

Joel has a bit of a record on this, he isn't best liked by a large proportion of the developer community who see their focus and remit as producing high quality, well tested and highly maintainable software. His original tirades in a podcast about his deep distrust of Test Driven Development and of Unit Testing in general brought uproar. And on that matter he was so far off the mark, it was understandable why he had riled so many people - he really just didn't understand or "get" TDD or testing in general.

But, what he had done, in conjunction with Jeff Atwood, was to deliver a website, stackoverflow.com, that was proving hugely successful, and was being used extensively by all those same developers who railed against his podcasts and blogs.

That website, as Joel and Jeff explained, was not done with TDD, was not well tested (via Unit Tests), was badly hacked together in places, and used "old" technologies like TSQL over "modern" solutions like ORMs

Whether Joel "got" TDD wasn't important, what he and Jeff had done was to ship a product, get it to market, and build upon that release to make stackoverflow the massive success it is today. It's so successful, that soon after stackoverflow was released, they used the same code and model to release other Q&A sites, and they continue to do so.

The Duct Tape Programmer

So with all of that in mind, it was sort of obviously flame bait for Joel to make a post that basically boiled down to "shipping it way more important than fancy patterns and great tools" - he certainly has a way of putting such a simple message that can antagonise the meekest of individuals (and some not so meek).

It was a terrible choice of label, but I guess "Pragmatic Programmer" had already been taken.

Time, Quality, Functionality

I read the article slight differently - I read it as an urge to maintain your highest priority as shipping the product your business wants you to ship.

There are three factors involved in a project (if you exclude Resourcing, which falls under Brooks Law):

Time
Quality
Functionality (or Scope)

If you try and maintain all three factors as fixed, your project is doomed to fail. Something will, unless you are really lucky, break. And then one of those factors will suffer, and it will be luck as to which it is. At best you can fix two of those factors, and then the third has to be allowed to slip when the unexpected happens (as it always does).

But, the important part here, is that which of those factors is fixed, and which is allowed to slip is NOT a developer decision - it is a business decision.

Much of the outrage about Joel's article stated that he isn't a developer - and they are correct - he is a business owner, and prior to that a project manager. His focus is therefore on shipping his product, and inevitably from that perspective, the instinct is to fix Time and Functionality (ship as soon as possible, with as much functionality as possible) and to let Quality suffer to achieve that objective.

This is a TOTALLY VALID business decision - it is just one that developers don't like, it offends their sense of code quality and high standards.

Why Do Developers Care So Much About Quality?

I'll start with the cynical view: development is only fun while you are trying new things, trying to perfect a tricky algorithm, or make some great new OSS component do some magic. By nature, the developers who care deeply about quality are also perfectionists, they like to make sure they have done a good job, and have written the best code they can.

Go on, admit it, if developers just did routine coding all day long, our jobs would be boring. We like new toys, new language features, we like to play.

And we know that better code will be easier to maintain - and developers HATE bug fixing duty, so we want to ensure we don't have to do much of it.

To a developer, the most important factor in any project is Quality, followed by Time (because we know our bosses will be mad if we miss our deadlines), but we are happy to slip Functionality because it matters little to us if a new feature isn't included in the release.

Fundamentally We Are Not Aligned

Developers and business owners are not aligned in their goals - we prioritise the three factors very differently, and for different reasons.

Only two things can happen here, your developers need to align with their business objectives, or the business needs to align with the developers - or they can both compromise.

Agile is all about the compromise, making both sides aware of the issues involved, and then coming to an agreement, but fundamentally - it is STILL A BUSINESS DECISION.

While it is the responsibility of a professional developer to explain all the problems they see with low quality code, and to make their boss aware of all the potential future issues this may cause, it is also their professional responsibility to go with the decision their boss makes.

They have one other option - resign and go somewhere that is more closely aligned with their values.

Ship It or Ship Out

Whatever your opinions of Joel and Jeff's understandings of TDD, Unit testing, maintainable code or just code in general (and I'm sure yours aren't too far from mine), you have to admit they delivered a product to market (many in fact), and have turned it into a huge success. They may have to pay for that speed to market later, but that was a business decision they were willing to take.

Never underestimate the value to a business of being fast to market - there is a risk the product will be so bugged it will backfire - but there is a chance they could steal the march on their competitors, and gain critical traction. Business is all about assessing risk, that is what a business owner has to do, and if they do it right their business flourishes.

If your boss wants a product shipped by a particular date, and that is their highest priority, you have a responsibility to ship it. If they won't sacrifice functionality to achieve that goal, then code quality will have to suffer - in that case do the best you can do with the restraints you are under. And if all that fails, quit, go somewhere that appreciates your values and dedication to your art - you do not have a right to hold your employer to ransom.

So, Ship It or Ship Out.

The Largest Wicked Problem in Software is Politics and Management

Jak Charlton — Tue, 25 Aug 2009 11:40:00 GMT

I previously wrote a post stating that Software is a Wicked Problem, and indeed it is.

There is a bigger Wicked Problem that affects software projects though, that of Politics and Management.

While there are at least technical solutions to most software problems, and even wicked problems can be cracked eventually, it is a much harder and more time consuming issue to deal with political and managerial problems, and they largely turn out to be wicked problems in of themselves.

And, as with all wicked problems, it is faster to fix the problem than it is to estimate how long it will take to fix - it also happens that with politics and managerial issues, it is often nearly impossible to estimate, or to actually fix the problems, so software developers spend most of their time trying to work around these problems rather than addressing them.

http://en.wikipedia.org/wiki/Wicked_problem

http://www.poppendieck.com/wicked.htm

Software is a Wicked Problem

Jak Charlton — Tue, 04 Aug 2009 13:06:00 GMT

A Wicked Problem, roughly speaking, has a number of properties:

The problem is not understood until after the formulation of a solution.
Wicked problems have no stopping rule.
Solutions to wicked problems are not right or wrong.
Every wicked problem is essentially novel and unique.
Every solution to a wicked problem is a 'one shot operation'
Wicked problems have no given alternative solutions.

Software development is largely a Wicked Problem, or in headline terms:

It is faster to create a solution to the problem than to estimate how long it will take to create that solution with any degree of accuracy, and the estimate will still be wrong.

http://en.wikipedia.org/wiki/Wicked_problem

http://www.poppendieck.com/wicked.htm

Twitter Went And Stuffed Replies!

Jak Charlton — Wed, 13 May 2009 06:48:00 GMT

I'm not going to spend too much time describing the problem ... Kevin Pang has done so admirably here, however I do want to mention this "small" change as it directly affects a new site I am building right now to work over Twitter.

Twitter's latest blog post titled Small Settings Update states the following:

We've updated the Notices section of Settings to better reflect how folks are using Twitter regarding replies. Based on usage patterns and feedback, we've learned most people want to see when someone they follow replies to another person they follow-it's a good way to stay in the loop. However, receiving one-sided fragments via replies sent to folks you don't follow in your timeline is undesirable. Today's update removes this undesirable and confusing option.

What this essentially means is that the "viral" nature of Twitter has been largely disabled. No longer will you see interesting replies from people you follow to complete strangers and think "maybe that guy is worth following too, I'll check him out". If the person is only mentioned, then you will still see the message, bbut I would lay a large bet this is only a very small percentage of the Replies.

I have no idea if Twitter did this to save bandwidth, or for some other more obscure reason, but ultimately it will hurt Twitter by slowing growth, as growth is largely by "discovering" people from replies. I'm sure "new" applciations will appear to try and plug this gap, but it is so deep in the Twitter API that it cannot be countered easily by other tools.

From my own perspective, it seriously damages the growth potential of my new Twitter based site, so will leave us with other less palatable options like spamming people or creating follow bots to try and get people interested - neither of which is desirable or in my ethos.

Dear Twitter, change it back please while it's not too late!

Dear Twitter users, please express your feelings via #fixreplies ... and remember you can follow my mini-rants @JakCharlton if this one wasn't enough for you!

Job Satisfaction, And Making The World A Better Place

Jak Charlton — Sun, 29 Mar 2009 11:24:00 GMT

You know what we achieve with all our wonderful software patterns and practices? Not a lot on the whole.

We do fairly well financially out of it, and many of those who use our software do fairly well financially from it too. A select few of us write software that can have a small impact on the well being of others, but even then, it is in a totally detached way.

But how often can you say that your day at work has affected somebody else's life for the better, or made a really positive change to the world? How often can you look in the mirror and say my whole day was spent helping other people?

So, what the hell am I doing in the software game? Well, like I suspect many of us, I fell into it by accident. It was my hobby at school, I wrote games software and made a fairly comfortable (for a schoolboy) living from doing it.

Although my choice of O Levels (equivalent to GCSEs these days) were picked around becoming a doctor, and even though my mother had to argue with the school for weeks to get them to allow me to do three sciences, home and family circumstances at that time meant I never actually took most of my school exams. So with nothing in the way of real qualifications, I drifted into what I was naturally good at - technical support and writing computer software. My hobby turned into
my career.

Twenty odd years later, and I am more than comfortable with my career, I like to think I am pretty good at it. I also still often think, I could have been adding much more value to other people's lives if I had stuck with my original choice. Recently I met a lady who was a nurse, and I was totally inspired by her and how much satisfaction she clearly got from her work – she told me that I could still retrain even at this stage in life.

Now I expect you are waiting for some kind of development related conclusion here ... but unfortunately there isn't one. What there is, is a small realisation that I could probably be doing something more for the world.

So, I have started seriously investigating my options for a radical career change - and to retrain as a medical doctor. Now this isn't quite my swansong from development just yet. Not only do I have some major considerations to make, but I also have to put enough money aside to finance this shift, and after all that I have to succeed in an application to medical school, which in the UK is a very difficult task indeed, as places at our universities are very sought after, and the universities can afford to be very selective. I could even do one or two years getting the entry requirements, only to fail to get a place at medical school.

But, I have started sending off enquiries to various universities asking for their advice on the best route in, and perhaps it will pay off. It will be time consuming, financially very hard in the short
term, but it could be the most rewarding thing I could possibly do.

Ultimately, Job Satisfaction is About More Than Financial Reward

Whatever I eventually do, I have made a decision to get out and start giving back something to the world, maybe that will be with 6 years of medical training, followed by years of excessively long hours for very little pay ... or maybe it will be just finding ways to make people's lives better while continuing my career in software.

I encourage you all to find something positive you can give back to the world - the development community has some incredibly smart and insightful people - let's not waste all our efforts on arguing over whether we should invert our dependencies or not!

And if you want somewhere to start doing your small bit, can I suggest you take a read of The Girl Next Door by Bil Simser

Bloody hell that was DEEP!

Welcome to Tuna – a Very Smart Fish!

Jak Charlton — Wed, 11 Mar 2009 16:39:10 GMT

Devlcio.us is proud to have Tuna Toksoz join us as our newest blogger … he really is one smart fish, and if you have been following his blogging on http://tunatoksoz.com/ then, you’ll be only too aware of his in depth knowledge of things like NHibernate and his involvement in Alt.Net Turkiye. He is also one of the most helpful and nicest guys I have come across in the InternetoSphere!

Looking forward to many great posts from him soon at: http://devlicio.us/blogs/tuna_toksoz