Jak Charlton - Insane World : Agile, Rants

Trust, Honesty, Openness, Cooperation and Friction-free Tooling

Jak Charlton — Tue, 03 May 2011 22:59:00 GMT

There is an inherent danger in using electronic tools to manage a process like software development, and one that almost always comes true - the tool defines your process, it doesn't enable it.

When people encounter limitations in the software tool, they begin to accept that these are limitations of the process and these become ingrained in your methodology. If the tool doesn't support the "thing you want to do" then it is far easier to change the "thing" rather than rewrite or change the software.

Were any of the electronic tools to have got the process "right" this might not be such a bad thing, but it would be reasonable to assume that in a field as adaptive and evolutionary as software development there is no such thing as the "right" process for an individual team, let alone for such a widely disparate ecosystem as development.

And this conflict between our tools and our actual practices and processes is what we refer to as "friction", and as physics defines it;

Friction is the "evil" of all motion. No matter which direction something moves in, friction pulls it the other way. Move something left, friction pulls right. Move something up, friction pulls down. It appears as if nature has given us friction to stop us from moving anything.

We aim for "friction-free" tools to allow us to better get on with moving things, specifically developing software, which is a constantly moving and adaptive process.

Some tools aim to allow endless configuration or programmability, in their objective of being a universal solution. These tools tend to actually make the problem worse, by either leaving you bogged down in configuration or customisation, or leaving the tool so generic that it solves no problem at all.

Of course, some tools are better than others. Lightweight tools are obviously going to create less friction as they have less moving parts, and less touch points with your actual process. But these are the ones that some see as "lacking features", or as being "too simplistic" - but it is these things specifically that enable the process of development to continue.

And ultimately, these tools expose a greater problem - there is usually a disjoint between project management and the development teams here. A lack of trust and visibility leads managers to seek to control or monitor the process more tightly, and these tools promise to allow that to happen. Surely if we can tie everything into a database or a spreadsheet or something similar, we can then see exactly what is happening, who is doing what, when things will be done, and how long they will take?

But it is not the tool that enables this to happen - it is serving here merely to try and enforce control upon people - which as any good manager knows, enforcement is a far blunter instrument than cooperation.

What enables proper management of the software development process is trust, honesty, openness and cooperation. Tools should reflect and enable those objectives, rather than try to enforce them. And most importantly, they should stay out of the way of the development teams to allow them to get on with their real job - developing software.

Agile is Performance, Feedback, Revision

Jak Charlton — Thu, 15 Apr 2010 02:45:08 GMT

I came across a great video link yesterday, entitled “Performance, Feedback, Revision”. It’s a Canadian rapper named Baba Brinkman covering the theory of evolution and the work of Charles Darwin, and he equates evolution with how he writes his lyrics, Performance, Feedback, Revision.

As I listened to it, it struck me that this is Agile … we develop, we run retrospectives, and we revise what we have.

Evolution may be more like waterfall, and take millions of years to evolve something as fundamentally pointless as the appendix, and in Agile we may look to have Performance, Feedback and Revision inside a week or two.

I can’t quite believe I just showed this video to a client to explain Agile, but it seemed to get the idea across pretty well. I’m considering using it more often, but I promise to draw the line at learning to rap.

But from now on, in big letters across the top of my white boards will go the words:

Performance, Feedback, Revision

Ship it, or Ship Out

Jak Charlton — Fri, 25 Sep 2009 08:38:00 GMT

Joel, in his inimitable way, posted the flame bait of all flame bait posts yesterday, explaining the role of the Duct Tape Programmer.

To my surprise, the Twitterverse started to reverberate with commentary, but weirdly, almost all of it was very negative about the post, many claiming that Joel was just creating straw men and knocking them down, or was characterising developers who care about quality as being poor developers.

I was surprised, because I read the article totally the other way around - and had said so earlier in the day to someone on Twitter, saying I largely agreed with it. It seems that there were two ways of reading the article, and mine differed substantially to most people I follow on Twitter.

I was quite pleased to see Michael Neel post last night in a semi-defence of Joel's standpoint, and I thought the 140 char limit of Twitter wasn't enough for me to post my opinions.

Background

Joel has a bit of a record on this, he isn't best liked by a large proportion of the developer community who see their focus and remit as producing high quality, well tested and highly maintainable software. His original tirades in a podcast about his deep distrust of Test Driven Development and of Unit Testing in general brought uproar. And on that matter he was so far off the mark, it was understandable why he had riled so many people - he really just didn't understand or "get" TDD or testing in general.

But, what he had done, in conjunction with Jeff Atwood, was to deliver a website, stackoverflow.com, that was proving hugely successful, and was being used extensively by all those same developers who railed against his podcasts and blogs.

That website, as Joel and Jeff explained, was not done with TDD, was not well tested (via Unit Tests), was badly hacked together in places, and used "old" technologies like TSQL over "modern" solutions like ORMs

Whether Joel "got" TDD wasn't important, what he and Jeff had done was to ship a product, get it to market, and build upon that release to make stackoverflow the massive success it is today. It's so successful, that soon after stackoverflow was released, they used the same code and model to release other Q&A sites, and they continue to do so.

The Duct Tape Programmer

So with all of that in mind, it was sort of obviously flame bait for Joel to make a post that basically boiled down to "shipping it way more important than fancy patterns and great tools" - he certainly has a way of putting such a simple message that can antagonise the meekest of individuals (and some not so meek).

It was a terrible choice of label, but I guess "Pragmatic Programmer" had already been taken.

Time, Quality, Functionality

I read the article slight differently - I read it as an urge to maintain your highest priority as shipping the product your business wants you to ship.

There are three factors involved in a project (if you exclude Resourcing, which falls under Brooks Law):

Time
Quality
Functionality (or Scope)

If you try and maintain all three factors as fixed, your project is doomed to fail. Something will, unless you are really lucky, break. And then one of those factors will suffer, and it will be luck as to which it is. At best you can fix two of those factors, and then the third has to be allowed to slip when the unexpected happens (as it always does).

But, the important part here, is that which of those factors is fixed, and which is allowed to slip is NOT a developer decision - it is a business decision.

Much of the outrage about Joel's article stated that he isn't a developer - and they are correct - he is a business owner, and prior to that a project manager. His focus is therefore on shipping his product, and inevitably from that perspective, the instinct is to fix Time and Functionality (ship as soon as possible, with as much functionality as possible) and to let Quality suffer to achieve that objective.

This is a TOTALLY VALID business decision - it is just one that developers don't like, it offends their sense of code quality and high standards.

Why Do Developers Care So Much About Quality?

I'll start with the cynical view: development is only fun while you are trying new things, trying to perfect a tricky algorithm, or make some great new OSS component do some magic. By nature, the developers who care deeply about quality are also perfectionists, they like to make sure they have done a good job, and have written the best code they can.

Go on, admit it, if developers just did routine coding all day long, our jobs would be boring. We like new toys, new language features, we like to play.

And we know that better code will be easier to maintain - and developers HATE bug fixing duty, so we want to ensure we don't have to do much of it.

To a developer, the most important factor in any project is Quality, followed by Time (because we know our bosses will be mad if we miss our deadlines), but we are happy to slip Functionality because it matters little to us if a new feature isn't included in the release.

Fundamentally We Are Not Aligned

Developers and business owners are not aligned in their goals - we prioritise the three factors very differently, and for different reasons.

Only two things can happen here, your developers need to align with their business objectives, or the business needs to align with the developers - or they can both compromise.

Agile is all about the compromise, making both sides aware of the issues involved, and then coming to an agreement, but fundamentally - it is STILL A BUSINESS DECISION.

While it is the responsibility of a professional developer to explain all the problems they see with low quality code, and to make their boss aware of all the potential future issues this may cause, it is also their professional responsibility to go with the decision their boss makes.

They have one other option - resign and go somewhere that is more closely aligned with their values.

Ship It or Ship Out

Whatever your opinions of Joel and Jeff's understandings of TDD, Unit testing, maintainable code or just code in general (and I'm sure yours aren't too far from mine), you have to admit they delivered a product to market (many in fact), and have turned it into a huge success. They may have to pay for that speed to market later, but that was a business decision they were willing to take.

Never underestimate the value to a business of being fast to market - there is a risk the product will be so bugged it will backfire - but there is a chance they could steal the march on their competitors, and gain critical traction. Business is all about assessing risk, that is what a business owner has to do, and if they do it right their business flourishes.

If your boss wants a product shipped by a particular date, and that is their highest priority, you have a responsibility to ship it. If they won't sacrifice functionality to achieve that goal, then code quality will have to suffer - in that case do the best you can do with the restraints you are under. And if all that fails, quit, go somewhere that appreciates your values and dedication to your art - you do not have a right to hold your employer to ransom.

So, Ship It or Ship Out.

Software Development is a Creative Skill, Building Houses is a Technical One

Jak Charlton — Mon, 18 Aug 2008 06:26:00 GMT

Brad Wilson mentioned on the TDD mailing list that the waste and inefficiency within the software industry was akin to the house building industry. I'm sure in some respects he is right, but in a more fundamental way I disagree.

An average layman off the street hiring a builder can ask them to build a wall. The average layman can see if the wall isn't straight, the average layman can see when bricks are not aligned, they can see when a house is built to a high quality finish, where attention has been paid to detail. They may not be able to see if the builder has been deceitful by using substandard materials or hiding shortcuts, but a few hundred pounds to a surveyor will tell them all they need to know.

Even the most technical of software management cannot tell if a basic 5 line piece of code is well written or not, let alone an application comprising many millions of lines of code. The best of them cannot even tell a "good" user experience from a "bad" one. It is almost accepted wisdom now that all developers write bad code, that all software is buggy, and that software projects never deliver what they promise.

It is this problem that needs to be overcome ... it is easy to spend a lifetime in a development career, without ever having written a single line of even average quality code. There is very little visibility to non-developers.

However, I assert the problem is that development is largely a creative skill, not a technical one. And creative skills are nearly impossible to quantify - you know when you like a piece of artwork, but you cannot say why in a way that means anything to anyone else. I cannot prove I am worth my daily rate, other than by people trusting me.

With that kind of stumbling block, the best we can do is to rely upon automated tests, continuous integration, and Agile methodologies to at least make us more accountable, even if we are still unquantifiable.

Recovering a Project with ASP.NET MVC and Agile

Jak Charlton — Wed, 13 Aug 2008 06:35:00 GMT

I recently got brought on board to a new client where, true to form, a project was in a state of failure - everybody sort of knew it, but nobody would say it out loud.

What Was Going Wrong?

After my initial assessment, I made a quick decision that the major thing holding everyone back was an early decision to use MOSS 2007 to host an external website, and as the primary content management system. This wouldn't have been a totally bad decision, if the team had a lot of good MOSS 2007 experience on board, but unfortunately they were learning as they went. If there is one thing that really doesn't work well in MOSS it is a "lets just start and change it as we learn and go along" approach. MOSS takes a fair amount of careful up front analysis and planning to get right, and has enough permeutations and "gotchas" to require some good experience on the team before starting. Otherwise you can rapidly end up with a big ball of mud.

So we took the decision that using MOSS was a major stumbling block, and this had to be removed from the equation. Of course it *could* have all been done in MOSS, but it wasn't going to happen in the time left. There was also a small "jedi mind trick" involved too ... people had become so focused on the SharePoint "way" that they could no longer see the bigger picture.

The decision was made to separate the public application away from MOSS, and to write it as a bespoke system. MOSS would still be used internally for document collaboration (which is where the strength of MOSS lies), and later on for publishing that information to the web site.

The Software Stack

After some fast prototyping, some instinct based decisions, and a bit of faith, I finally decided on the following software stack to write the application:

ASP.NET MVC (preview 4)
NHibernate
Castle ActiveRecord
Castle Windsor
Rhino Commons (IRepository, ARRepository, UnitOfWork)
log4net

Some of these were fairly safe choices, I have used them before, or they are tried and tested in the field. But obviously the major exception to that rule was ASP.NET MVC

I very briefly considered Monorail, but it does not have the tooling support or documentation support of ASP.NET MVC, and going forwards Monorail is likely to dwindle whilst ASP.NET MVC will only grow.

I also considered writing a Webforms application, and using MVP to split the UI from the logic, but this approach required a fair ramp up from the developers, and had too many possible variations to try and coax people into the right appraoch.

ASP.NET MVC therefore would provide a degree of familiarity to the ASP.NET developers we have in the team, whilst providing a clean separation of responsibilities. The only question was stability, which after asking around seemed not to be an issue, despite being only at preview stage (something MS haven't done before but is very welcome here) nobody reported any problems.

By changing pretty much every element of the software stack we were using the team has been revitalised, and has approached the project with renewed vigor and enthusiasm. They get to learn some new things for their CV (pretty much none of them have used any of the new comnponents), and due to the amount of "magic" that things like Rhino and Windsor introduce they could get on with writing real functionality almost from day one.

I am very impressed by how fast the developers have picked up MVC, here a large amount of credit goes to MS for making a very usable, approachable, and simple framework that does exactly what it needs to, and very little more.

Making it Build With Continuous Integration

When I arrived there was no project in place. Nothing, nada. I could not pull the project, build it and see it work. As everything was being done in MOSS people were writing projects for each small bit of fucntionality, and manually copying assemblies into a single MOSS deployment. And the only source control that existed was Visual Source Safe. To say the least, this was scary.

So the first thing I did here was to install VisualSVN and TeamCity and get a build server running. After setting up a single solution with all the relevant projects included, we now have a build that operates on every check in, and it automatically displays on a web site for everyone internally to see the progress. On the dev machines we installed Tortoise SVN and Ankh SVN, Ankh now seems a lot more stable in version 2, but has some irritating quirks still - it is however much better than any MS source control plugin, and falling back to Tortoise almost always fixes the problem.

We also installed the trial version of ReSharper on the dev machines. None of the people here had used it before, but within only a few days we were hearing a lot of "oh wow", "that's amazing", and "that used to take me so long to do" ... in a week or two we will have to buy licences, but I think it has more than justified itself to the development team.

The URL of the build site was distributed to the business users, and now they take an active interest in telling me the "walking skeleton" is a "pile of bones" when someone changes the DB loging details last thing at night :)

Changing the Process

Also evident was that the approach that was being taken to the project was classic waterfall, along with a mixture of daily constantly changing and fluctuating requests and requirements. Communication was good with daily meetings, but the same things seemed to be repeated at each.

So another bold decision I took was to approach this as a far more Agile project - using some elements of XP.

I threw out the gant chart - much to the dismay of the project manager - and started writing up story cards, which promptly got placed on a commandeered wall. The project manager now has these story cards listed in an Excel spreadsheet so that he and the people he reports to can see exaclty where things are in a palatable form, but the wall remains the "one true master"

We changed the 8.30am sit down meeting to a 9.30am stand up meeting, and have been trying to enforce a fast "what I did yesterday, what I will do today, what is holding me up" approach to stop us getting dragged down into details.

We now have a lot more communication between developers as a result of this, and the business representatives are feeling very involved. There is still a degree of trepidation around what will be delivered, my protests of "you can see it as it goes along" and "it will be done when it will be done" are proving quite hard to swallow, but I think everyone understands there is no magic pill here and we just have to judge it step by step.

The Proof Will Be in the Pudding

So some good early steps have been taken. I cannot say the project is back on track, but everyone is now far more involved, has far more enthusiasm, and can see something starting to take shape. The perceived deadline of a few weeks from now is unlikely to be met with a fully functional site, but as "fully functional" is only vaguely defined, we are aiming to have something that is "good enough" by then, and then to make it better as people feed back into the process.

So far a major success on the approach, now to see if we can turn that into a successfully delivered solution!

What Determines High Quality Code?

Jak Charlton — Fri, 16 May 2008 13:59:00 GMT

Code quality is an abstract concept again, and can be defined may ways depending on how you perceive quality. A good discussion of the many aspects of code quality can be found on Wikipedia at http://en.wikipedia.org/wiki/Software_quality

Some general high level objectives for software quality could be considered to be:

Conformance to requirements
Scalability
Correctness
Completeness
Absence of Bugs
Fault tolerance
Extensibility
Maintainability
Documentation

At a code quality level, some general good guidelines would be:

Readability
- Code should be simple to read, even without comments
- Class, method, and variable names should be expressive and unambiguous
- Functional intent should be easily understood
Ease of Maintenance, Testing and Debugging
The degree of effort required to maintain, test and debug an application is a strong indication of the quality of the code and architecture. As the great majority of the cost of an application is actually composed of these three activities, these should be primary considerations
Low Complexity
Complex code is rarely complex because the business functionality it implements is complex. Most software complexity comes through poorly designed and implemented code, over analysis of the problem, or adding additional functionality for requirements that do not yet exist.
Low Resource Consumption
Poorly written code will not take account of resource usage, for example it will not cache information it could cache, or it will create new objects when it did not need to. These issues lead to hard to maintain applications when deployed and used.

To achieve these objectives, some basic principles of development and design can be applied. These have been referenced elsewhere, but are repeated here for clarity:

Single Responsibility Principle
There should never be more than one reason for a class to change
Liskov Substitution Principle
Methods that use references to base classes should be able to use derived classes without knowing they are doing so
Open Closed Principle
Software entities (classes, modules, methods) should be open for extension but closed for modification
Interface Segregation Principle
Clients should not be forced to depend upon interfaces they do not use
Dependency Inversion Principle
High level modules should not depend upon low level modules, they should both depend upon abstractions
Abstractions should not depend upon details, details should depend upon abstractions
Principle of Least Surprise (sometimes known as Principle of Least Astonishment)
The result of performing some operation should be obvious, consistent, and predictable, based upon the name of the operation and other clues
Separation of Concerns
The application should be broken into components that overlap in functionality as little as possible

An assessment of code against these core principles should give a developer a good basis for assessment of the quality of a code base.

Statistics and How They Lie

Jak Charlton — Fri, 16 May 2008 09:49:00 GMT

Industry experience suggests that the design of metrics will encourage certain kinds of behaviour from the people being measured. The common phrase applied is "you get what you measure" (or "be careful what you wish for").

A Brief Explanation of Cyclomatic Complexity and Code Coverage

Cyclomatic complexity is a measure of the number of possible paths through a piece of code. This is a relatively easy measurement to make, and code analysis tools can give you this figure directly. As cyclomatic complexity is directly analogous to the number of paths through the code, it is also a direct count of the number of unit tests required as a minimum to test all the code paths.

A cyclomatic complexity of 1-10 is a simple piece of code that is easily maintainable.
A cyclomatic complexity of 11-20 is a moderately complex piece of code, that is relatively easy to maintain.
A cyclomatic complexity of 21-50 would indicate either a highly complex or high risk piece of code. The functionality of the code observed indicates it is in no way complex, and therefore it must fall into the high risk category.
A cyclomatic complexity of 51 and above indicates un-testable code, and a very high risk to the project.

Code coverage measures the number of paths through your code that had at least one execution when unit tests were run. It does not however give any real quality measure against the value of those tests, or the actual quality of those tests.

It is actually easier to get better coverage results from writing poor code. The poorer the quality of your code and of your tests, the easier it is to achieve high coverage figures.

Lines of Code as a Measure of Progress

Jak Charlton — Fri, 16 May 2008 08:51:00 GMT

Some reports will highlight lines of code as a figure to attach some relevance to, and these become measures used to establish progress.

These are possibly the most misleading figures to use, in fact almost always within a well designed application and code base, the reverse is true.

Good code tends to be more concise than poor code, and therefore it is less lines of code.

Good code also tends to have high levels of code reuse; one of the primary refactoring exercises is removing code duplication. This obviously again leads to a lower number of lines of code. However, the concept of reuse can also be used to increase lines of code and reduce quality in the hands of inexperienced developers.

An example:

The following pieces of code are all functionally identical:

Version One (4 lines):

        public void MyMethod()
        {
            if (myObject == null) doSomething();
        }

Version Two (8 lines):

        public void MyMethod()
        {
            if (Assert(myObject)) doSomething();
        }

        public bool Assert(object theObject)
        {
            return theObject == null;
        }

Version Three (8 lines of code):

        public void MyMethod()
        {
            if (Assert(myObject))
            {
                doSomething();
            }
        }

        public bool Assert(object theObject)
        {
            bool theResult = (theObject==null);
            return theResult;
        }

Version Four (20 lines of code):

        public void MyMethod()
        {
            if (Assert(myObject))
            {
                doSomething();
            } 
        }

        public bool Assert(object theObject)
        {
            bool theResult;
            if (theObject==null)
            {
                theResult = true;
            }
            else
            {
                theResult = false;
            }
            return theResult;=
        }

As it turns out, the version of that code with the highest quality is Version One, coming in at 4 lines of code total, and actually only 1 line of code that does anything. Version Four has 5 times the number of lines of code as a base guideline, but has 8 times the lines of code if only actual statements are counted. Version Four is coincidentally the poorest quality version of the functionality.

So if lines of code are a measure of progress, a developer can, consciously or subconsciously, increase their visible progress, by writing lower quality and less maintainable code.

How About Measuring Decreasing Lines of Code as Success?

Depending on how you are measuring lines of code, even looking for trends in reduction of lines of code can be considered the wrong approach. For example, now if I concatenate multiple statements into a single terse, unreadable and un-maintainable statement, I can reduce my line count.

An example:

Version One (8 lines):

   public int MyMethod(string myString, int result)
   {
       private int const SmallIncrease = 1;
       private int const LargeIncrease = 2;

       if (string.IsNullOrEmpty(myString))
           result = result + SmallIncrease;
       else
           result = result + LargeIncrease;
      return result;
   }

Version Two (4 lines):

   public int MyMethod(string myString, int result)
   {
      return string.IsNullOrEmpty(myString) ? result+1 result+2;
   }

In this example, Version One of the code is simpler to read, but is twice as many lines of code. This is also a very simple example; a more complex example would be even more drastic in the difference between the numbers of lines of code in the two methods.

Within this simple example the first is much easier to read, understand, and therefore maintain. The second obfuscates the values of 1 and 2 in the calculation, and makes it harder to follow the execution path of the code.

Arguably Version Two could be considered acceptable with some modifications (like the addition of the constants defined in Version One), however both of these piece of code will actually compile to identical binary code.

In this case Version One is higher quality code as it is easier to understand its intention, and the ability to quickly read and understand intent is a major factor in determining the quality of code.

Measuring Progress

Jak Charlton — Thu, 15 May 2008 08:48:00 GMT

"Working software is the primary measure of progress"

Fundamentally, there is no more valid measure for progress, than the working software itself. This only leaves open to discussion, the definition of "working software".

Defining "Working Software"

The criterion for defining working software is obviously open to debate. A common definition is:

Software can be called "working software" when it meets a defined set of business requirements and can be demonstrated to do so through testing.

This is one reason why Agile processes put so much value on unit testing, these tests show very early on in the process that software is meeting business requirements, without the need to create a fully functional user testable application. These unit tests also allow demonstration at a fairly low level of granularity that code is meeting requirements, where a user facing application is dependent on too many factors to easily establish correctness.

So How Do We Measure Progress?

Based on this definition, our best way to measure progress and velocity on projects is to evaluate defined business requirements, against the code that is provided to meet those requirements.

Code that is written, but is not yet functional and passing tests cannot be considered progress until it has been completed to a level as defined above.

This then prioritises getting components of functionality completed early, rather than attempting to do all things simultaneously.

The traditional "waterfall" approach to software development is to spread large amounts of functional requirements out amongst large teams, for example assigning each piece of functionality to a team member with an expected delivery date measured in weeks or months. This leads to very long cycles for delivery of something correlating to "working software"

An Agile approach to the same problem is to focus the teams on only a small subset of that functionality, and to attempt to deliver it in a working and testable state in very short iterations. The degree of success with which they do this then becomes their "velocity". The velocity can then be used as a predictor of future success rates and therefore of future timescales. This also allows a "fail fast" mentality, where it is better to hit problems early on and resolve them, rather than delay all the problems for as long as possible down the development path.

Therefore, the best way of measuring success is to do one thing at a time, do it well, ensure it works, ensure it meets criteria, ensure it can be tested, and then to replicate the things that went right on the next piece of functionality, and eliminate the things that did not go so well.

The Only Certainty is Change

Jak Charlton — Wed, 14 May 2008 06:08:00 GMT

I got an email at the end of last week from a developer asking about Agile development. It highlights a few problems with development in general, and with Agile as a "badge of honour" that are worth exploring. It deserved a fairly detailed reply, so excerts of the email follow:

I just came into an Agile project with some difficulties, they have stopped doing Agile development and gone back to a Waterfall approach. Coming into the project late, I saw numerous problems with the approach that are not helping.

It strikes me that developers rarely come onto a project and think how wonderful it is and how well it is running. This could be because they just have different ideas to everyone else, we all know developers are pretty opinionated, and believe they have the "right" answer to all the questions.

There is however a more likely reason, when you are close to a project, it is sometimes hard to see what is glaringly obvious to anyone with fresh eyes.

Agile can be daunting, you are fully exposed, you need to have Courage, you need to resist the urge to falling back to lines like "well we don't have written requirements", "it will be fixed in testing", "it isn't my code", and all the other Waterfall phrases.

Developers would be working on stories but things were not available to them as they had not been created yet - those items were in other stories

This sounds like a bad planning excercise. If you are using user stories, then it is the job of the developers to be involved in the planning of these, and to be able to assess up front what dependencies these stories have upon each other, and to properly break these down to more manageable, and more appropriate, stories.

I have seen similar problems, and it was generally caused by project managers deciding upon the priorities for things, and not listening to developer feedback (or not even including them). A good project manager is vital to a project to act as an "enabler", someone who can make things happen, can remove stumbling blocks, and can generally "grease the wheels". A bad project manager is the worst thing that can happen to a project.

Requirements changes by the business could cause certain functionality to have to be reworked significantly, causing a lot of problems, if the stakeholders changed their mind every time they saw something, more changes would be introduced. The more they saw, the more they wanted to change.

Well, Agile dictates (in so far as Agile dictates anything) that the only certainty in a project is that the requirements will change. We embrace change. Agile methodologies are all designed to allow change to happen as easily as possible, and to encourage change to be a core part of the driving process.

There is a point at which this can easily go wrong though, and again it comes down to either the wrong people in the planning excercise, or someone not listening.

Understanding change happens is only the first step.

You have to also explain to the business that change is inevitable, but it is certainly not free. The decision rests with them, but it is the job of the development team to properly evaluate requests and user stories to see what impact they will have on the project now, and into the future. Business users presented with an option to change, but no consequences of doing so, will of course choose change. When asked whether they want to make the change they think is needed, or do the other three user stories that were scheduled for this week instead, the choice becomes rather different. Agile encourages everyone in the process to become a stakeholder, but everyone also has to bear the cost of change.

Then you have to be able to think ahead and to design your system in such a way as to allow for change. This is a technical challenge, largely only obtained via experience, both of the development language and tools you are using, but also of the business team you are dealing with - how often do they change their minds, how often do they express things badly, how often do they get the wrong person to make the decision. In the sales game, you always want to be talking to the decision maker, becasue only they can give you the answer you need - make sure as a development team you are talking to the decision maker too.

A well designed architecture is fundamental to a successful project, and a key quality of a well designed architecture is the ability to make changes with relatively little impact upon other system components. Adding or modifying behaviour should not be a traumatic process - it may not be trivial either, but it should never be traumatic. I covered some of the aspects of this previously, and although that post was geared towards why Inversion of Control can benefit your code, it also deals with Robert C Martin's observations on a Rotting Design. Make no mistake, it is perfectly possible to start a greenfield project and have a rotting design from day one. It is also a large risk to any project that the once elegant code base rapidly slips into being a rotting design when time pressures are on, and people start taking shortcuts.

It is hard to know when it will be finished when things are always open to change

Agile methodologies do not say we have no delivery date, in fact almost to the contrary. Generally under an Agile project you should be able to deliver a working version of the software at any point in the development.

Admittedly the functionality present within that software will be largely restricted the earlier on in the process you decide to deliver, but 75% of functionality with a high quality code base is a lot better than 100% functionality which it is heavily bugged and fragile.

I often draw a triangle for clients, with Time, Quality, Functionality as the three points (Resources is often included here, but I ignore it for this purpose, Brooks Law usually applies by the point this matters). I tell them they can pick any two of those aspects and fix them, but that means the last one will have to vary. They cannot fix all three.

If Time is a critical deliverable, then you can choose to sacrifice Quality or Functionality (or both). The perfectionist in me says sacrifice Functionality, Quality is too precious to be the one to slip.

If Quality is a critical deliverable, then choose to slip Time or Functionality. The choice here only comes down to whether you want it fast or complete, or what combination of the two you want.

If Functionality is the key deliverable, then you can slip Time or Quality, and again I would say that Time has to slip, as Quality is too precious.

My personal opinion is therefore that Quality comes first, Functionality and Time are the things you can alter - but not only does this vary by project, but also by business circumstances. Sometimes it really is better to get to market with a low quality product, in the lowest possible time, with limited functionality (Twitter immediately comes to mind), just to steal a march on the opposition.

But what is really important, is that you express these aspects of development explicitly. If you are sacrificing quality because the business has asked for something unachievable, then let them know you are doing this, and make it an express design decision. If later on the business tells you the software is bugged to hell, you can point out when they took that decision to satisfy another need. If you need to take two more months because the business just changed the business model, then fine, but again make it an expressed and documented design decision.

Too many of these things get left as "assumptions" ... if you don't say otherwise, the business will assume a really high quality product, with massive functionality, in a very short timescale ... it isn't possible, so let them make the choice.

Project Failures - Blame the Process, or the People?

Jak Charlton — Fri, 09 May 2008 07:15:00 GMT

Prompted partially by some comments yesterday on my post on How to Make Late Software, Even Later, and partially by a discussion on the altdotnet Yahoo list, I wrote this long email. As it became an epic in its own right, I thought it deserved a blog.

We Are Doing Agile!

I can claim to be doing Agile, and following with some degree of conformance, a known and understood Agile methodology. If my project fails, we must blame the Agile process, and our practices in some way for this

I can claim to be doing Agile, but make up my own variation of a known methodology, in which case a failure of the project is largely encumbant upon me for deciding to sacrifice the practices we chose to ignore, and the new ones we added.

I can claim to be doing Agile while just coding hell for leather and following no methodology, just cherry picking a couple of practices here and there, and picking the principles I like - in which case any failure is mine to bear.

All of these approaches should be highlighted in a risk analysis to management to make the decisions *before* you decide to pursue any of these paths. It is the decision of management to weigh risk vs reward and to make that decision. If I didn't write that report, or I wrote it badly, then again the fault is mine to bear. If I wrote it clearly expressing the dangers of a hack and see approach, and management chose that path, then it is their cross to bear.

You Cannot Blame A Process For Producing Bad Results

If all participants in a waterfall methodology like RUP followed the practice rigourously, then I have little doubt the project would have a high chance of success (as defined by the process, a close match to the intial requirements)

If all participants in an Agile process like XP followed the practice rigourously, then I have little doubt the project would have a high chance of success (as defined by by the process, a close match to the evolving requirements)

Bad people on any project can stuff any methodology, and bad application of good practices will always result in failure. Agile just gives you much earlier feedback that you are not getting it right - how you choose to interpret that feedback, or how management choose to, is the fault of those communicating or interpreting the information - not of the process.

How To Fail Under Agile

The worst, and I mean absolute worst project failure I ever worked on was done using Agile, using Extreme Programming, and the practices were fairly closely followed. It should have been a success. And for a few small but critical mistakes it would have been.

The first big mistake was not evolving the architecture first. The application had been developed under XP for over 9 months when I was brought in the provide the "Enterprise component" of the application. It became rapidly apparent that the application had been written as a desktop applciation, and was too tightly coupled to allow any extension points for an evolution of the functionality into a different UI and service layer.

The second big mistake was a large amount of code that had been written to allow the client applciation to operate on domain objects, using an ORM, persisted across a web service layer, relying on massive amounts of code generation. This was a technical mistake that should have been picked up by prototyping and getting the architecture right first.

The third big mistake was a very poor application of the XP process. The bug backlog for each week was continual, and continally increasing. It was seen as acceptable that stuff didn;t work, and in each stand up it was just refered to as a bug and would go on the backlog. Nobody ever stopped to take a grip of this, and realise that this was a warning sign. As complexity was increaing, the number of interactions in the software was going up rapidly, and people spent more time bug fixing. Out of around 5000 unit tests (afair) that existed when I got there, around 2000+ were failing - this was explained as changes to the code and hence they couldn't get them working. This invalidated pretty much the entire test suite.

These were not problems with the process (XP), but clear problems with no up front design or prototyping (point one), no technical understanding of the challenges involved (point two), and people badly applying the process (point three).

Was Agile or Extreme Programming to Blame?

Yeah, we know loads of it is in flux, and it is a bit 'out there' but this is Agile man, we live by the seat of our pants, we ride fast and hard, this is EXTREME!!!!!

In the respect that it allowed developers to go to senior management and say this, hell yes Agile and XP were to blame. It gave the excuse to developers that they could excuse stuff being chaotic, as being all part of the way this stuff works.

The reality was the project failed due to a manager that for all his ability, had never done Agile or XP before, and had taken to quoting Martin Fowler PoEAA so often when we tried to raise problems, that his copy was starting to fall apart. And the business owner took the Agile/XP no requirements thing to the Nth degree and would change his mind on features on a daily basis, so the team could never get a handle on the problem. And lastly the team, that had gone into a terminal march towards inevitable failure, but had never spoken up loudly enough.

Oddly enough, that was the single team I have worked on that had by far the highest technical ability of any team as a whole I have ever worked with. They had some absolutely superb developers - but sometimes even that isn't enough.