Jak Charlton - Insane World

Avoiding Prescriptive Requirements

Jak Charlton — Thu, 09 Feb 2012 03:50:00 GMT

Core Requirements

Requirements should be given as the Core Requirement, and avoid the common pitfall of providing Prescriptive Requirements

As an example, a core requirement in an insurance system may be:

As a user of the Broker system, I want to create a Settlement Batch of Eligible Documents across Insurance Policies

This is eminently testable, is clear and unambiguous, and has real business value, and it meets the general INVEST principle of good user stories (which is also a good maxim for any form of requirements):

It is Independent of other stories
It is easily Negotiated as to priority and value
It has real business Value
It is small enough to easily Estimate with some degree of certainty
It is of an appropriate Size for easy estimation, planning and prioritisation
It is able to be Tested both manually and in an automated fashion with clear and unambiguous pass or fail criteria, you can either do it or not

The stated requirement is not prescriptive, it does not specify how the user triggers this action, what data they need to provide, how they select the documents, nor does it specify what a "user of the Broker System" is within this context, nor what a Settlement Batch is, nor what an Eligible Document is. This means the requirement can be documented very early on, with little analysis work needed, allowing very quick comparison with other requirements, and very quick verification from the business as to the validity of the story.

Later this placeholder can be expanded in discussions between the technical teams and the business to include the actual definitions and scenarios that define its acceptance criteria:

The user must have the “Batch Creation” permission
Eligible Documents are those which are awaiting Settlement
A Settlement Batch is a group of Documents that will be dealt with together by the Claims Settlement system, so as to save managing on an individual policy level

This would lead to a requirement with maybe a dozen test scenarios giving all the variations of user and statuses that would allow or not allow the story to pass the test. All of these stories and scenarios are now easily tested with automated and repeatable testing.

At this point it would probably be obvious where within the UI this functionality should surface, and the design of the UI interaction can be worked out. This UI work is actually a very specific development skill within its own right - there is significant complexity in designing UI interactions that mirror both the technical system and the user’s mental model.

Through all of these steps, the original core requirement has not been changed, though its implementation and business logic may have evolved, changed and adapted to line up with other functionality.

Good requirements are merely a placeholder for a later conversation. Their level of completeness or accuracy only dictates your certainty in implementing them accurately and efficiently, but does not preclude moving forwards, and does not hold development up waiting on “complete requirements”.

As granularity increases, so does complexity required to understand, verify and implement, but oddly so does instability. By starting with core stories, you maintain a fairly stable set of requirements, and only deal with complexity when you need to, and suffer the instability only on individual scenarios.

Which brings us neatly to development

It is fairly well established now that development must be iterative within short cycles to increase its chance of being successful. Early feedback and revision saves massive amounts of time and effort correcting incorrect assumptions and poor translation.

The sooner a story enters development, the sooner a basic version of it can be demonstrated to business users, to verify against their assumptions, the faster it can be revised and the more accurately it will reflect the actual requirements. In addition it will expose underlying technical constraints and limitations earlier on.

Single Action per Controller in ASP.NET MVC

Jak Charlton — Mon, 11 Jul 2011 01:08:00 GMT

As I keep doing this on MVC projects, thought I may as well blog it, so the next time I need it I can find it myself :)

To keep ASP.NET MVC controllers clean and organised, I find it much easier to split them so they handle only a single action each. This helps significantly with maintaining some semblance of the Single Responsibility Principle, allowing each controller to deal with a single action, and not have concerns like Logon, Logoff and ChangePassword all start globbing together under a single controller called "Account"

With a little bit of magic in the ControllerFactory you can tell ASP.NET MVC3 to resolve controllers from your own directory and class structure, meaning we can have controllers that look like:

public class IndexController : Controller
{
   [Authorize]
   public ActionResult Index()
   {
      return View();
   }
}

https://gist.github.com/1075169

The Controller Factory

I tend to use Castle Windsor to do this magic for me, though any IoC container would do, and you could even do it without a container if you were really masochistic!

In Windsor the ControllerFactory looks like:

  public class WindsorControllerFactory : DefaultControllerFactory
    {
        private readonly IKernel kernel;

        public WindsorControllerFactory(IKernel kernel)
        {
            this.kernel = kernel;
        }

        public override void ReleaseController(IController controller)
        {
            kernel.ReleaseComponent(controller);
        }

        protected override IController GetControllerInstance(RequestContext context, Type controllerType)
        {
            var baseNs = typeof(SecureController).Namespace;

            var ns = context.RouteData.GetRequiredString("controller").ToLower();
            var controllerName = context.RouteData.GetRequiredString("action").ToLower() + "controller";
            try
            {
                var controller = kernel.Resolve<IController>(baseNs.ToLower() + "." + ns + "." + controllerName);
                return controller;
            }
            catch (ComponentNotFoundException ex)
            {
                throw new HttpException(404, string.Format("The controller for path '{0}' could not be found.", context.HttpContext.Request.Path));
            }
        }
    }

https://gist.github.com/1075171

The Controller Installer

So that Windsor has some way of resolving our controllers, you need to register them in the container. This installer will scan for all classes implementing IController and add them into the container against a key matching their namespace:

public void Install(IWindsorContainer container, IConfigurationStore store)
{
     foreach (var type in Assembly.GetExecutingAssembly().GetTypes())
     {
      if (typeof(IController).IsAssignableFrom(type) && !type.IsAbstract)
      container.Register(Component.For(type).ImplementedBy(type).Named(type.FullName.ToLower()).LifeStyle.Transient);
     }
}

https://gist.github.com/1075176

Which leaves your controller folder structure looking like:

GET and POST

Where an action has a Get and a Post action, for example where you display a form, then take some actions based on the form, you just put the two actions in the same Controller, you aren't breaking SRP here as both actions are inherently linked anyway:

   public class ChangePasswordController : Controller
    {
        private readonly IUsersDB users;

        public ChangePasswordController(IUsersDB users)
        {
            this.users = users;
        }

        [Authorize]
        public ActionResult ChangePassword()
        {
            return View();
        }

        [Authorize]
        [HttpPost]
        public ActionResult ChangePassword(ChangePasswordViewModel model)
        {
            if (ModelState.IsValid)
            {
                try
                {
                    users.ChangePassword(User.Identity.Name, model.OldPassword, model.NewPassword);
                    return RedirectToAction("ChangePasswordSuccess");
                }
                catch (DataAccessException ex)
                {
                    ModelState.AddModelError("", "The current password is incorrect or the new password is invalid.");
                }
            }

            return View(model);
        }
    }

What is Domain Driven Design?

Jak Charlton — Mon, 16 May 2011 02:15:00 GMT

"... the key to expert performance in many fields is domain knowledge rather than intelligence."
Don Reinertsen

Domain Driven Design is a software development methodology, intended to achieve a software system closely modelled on and aligned with real business processes.

Traditionally development tends to be a technically led process, where requirements are passed from the business to the development teams, and then the development teams go off and produce their best guesstimate at what the requirements said.

In a waterfall approach this leads to large requirements documents that are diligently collated, analysed, reviewed and approved. These documents are then given to development teams to turn into working software.

Agile approaches can also accept the requirements documents that waterfall tends to produce, but for them to actually progress forwards they must be broken to small tasks and stories which can then be queued ready for development.

Domain Driven Design to a large degree steps back from these two distinct ends of the spectrum, and looks at how the requirements are gathered in the first place - if you like, it bridges the gap between doing everything up front and everything at the last minute.

DDD understands that requirements are never 'done', but exist as a living document. More importantly, the living document in question is actually the software itself - all other documents are an artefact and reflection of the code.

As the software system develops and grows, deeper understanding of the problem grows too - DDD is all about the discovery of the solution through deeper understanding of the problem.

Where DDD really differs though is that it sees the software system as a reflection of the business process, an enabler rather than the driver. DDD is deeply concerned about the business processes, business terminology and practices. Technical concerns are largely secondary and a means to an end.

The Ubiquitous Language is at the centre of DDD - this is a shared and growing language. A negotiated language derived from the business terminology and enriched by the development teams. If a business person doesn't understand a term in the UL, then the chances are the UL needs an evolution. If a technical person doesn't understand a term in the UL then the chances are they need a conversation with a Domain Expert.

The Domain Experts are the second essential component of DDD - people who deeply understand the business domain, and the business itself. These people are integral to the development process. They may not be required "full time" like the traditional Product Owners of some Agile methodologies, but they must be accessible continually, and be prepared to invest time in the process. Domain Experts are not to be considered outsiders, but as the core of the DDD process - they are as much a part of the development team as any developer or tester.

DDD doesn't begin and end - it is a constant process of re-evaluation, refactoring, remodelling and redesign - every conversation should bring you to a closer understanding of the problem. At no point is DDD 'done' - it is always there; the Ubiquitous Language grows and develops, the Domain Models change as understanding changes, code is restructured and refactored to better represent that understanding.

Artefacts will come and go, but only one really matters - the code base. It is the only expression of the solution that is free of abstraction. While documents may try to explain or describe the system, only the code does so without losing information. This means that in DDD the code must remain of high quality, be clear and expressive, be free of technical abbreviations or jargon, and as far as possible should make at least some sense to a Domain Expert when explained to them.

Clever code doesn't work in DDD, nor do magical processes or "you don't need to know" components. Domain Experts shouldn't need to become developers to understand what the key parts of the software system is doing for them. They also shouldn't need to think in terms of databases or batch jobs or other technical concerns.

DDD is the ultimate expression of Agile - it is about dealing with a constantly shifting and evolving set of requirements - because as anyone who has ever been involved with a software project knows - it is very rare for a requirement to remain intact from the beginning to the end of a project, and almost impossible for it to do so as the business grows and changes.

Through constant conversations, DDD will guide you to an ever more accurate software representation of your business process.

Trust, Honesty, Openness, Cooperation and Friction-free Tooling

Jak Charlton — Tue, 03 May 2011 22:59:00 GMT

There is an inherent danger in using electronic tools to manage a process like software development, and one that almost always comes true - the tool defines your process, it doesn't enable it.

When people encounter limitations in the software tool, they begin to accept that these are limitations of the process and these become ingrained in your methodology. If the tool doesn't support the "thing you want to do" then it is far easier to change the "thing" rather than rewrite or change the software.

Were any of the electronic tools to have got the process "right" this might not be such a bad thing, but it would be reasonable to assume that in a field as adaptive and evolutionary as software development there is no such thing as the "right" process for an individual team, let alone for such a widely disparate ecosystem as development.

And this conflict between our tools and our actual practices and processes is what we refer to as "friction", and as physics defines it;

Friction is the "evil" of all motion. No matter which direction something moves in, friction pulls it the other way. Move something left, friction pulls right. Move something up, friction pulls down. It appears as if nature has given us friction to stop us from moving anything.

We aim for "friction-free" tools to allow us to better get on with moving things, specifically developing software, which is a constantly moving and adaptive process.

Some tools aim to allow endless configuration or programmability, in their objective of being a universal solution. These tools tend to actually make the problem worse, by either leaving you bogged down in configuration or customisation, or leaving the tool so generic that it solves no problem at all.

Of course, some tools are better than others. Lightweight tools are obviously going to create less friction as they have less moving parts, and less touch points with your actual process. But these are the ones that some see as "lacking features", or as being "too simplistic" - but it is these things specifically that enable the process of development to continue.

And ultimately, these tools expose a greater problem - there is usually a disjoint between project management and the development teams here. A lack of trust and visibility leads managers to seek to control or monitor the process more tightly, and these tools promise to allow that to happen. Surely if we can tie everything into a database or a spreadsheet or something similar, we can then see exactly what is happening, who is doing what, when things will be done, and how long they will take?

But it is not the tool that enables this to happen - it is serving here merely to try and enforce control upon people - which as any good manager knows, enforcement is a far blunter instrument than cooperation.

What enables proper management of the software development process is trust, honesty, openness and cooperation. Tools should reflect and enable those objectives, rather than try to enforce them. And most importantly, they should stay out of the way of the development teams to allow them to get on with their real job - developing software.

CQRS - The Cult of Shiny Things

Jak Charlton — Fri, 29 Oct 2010 00:12:00 GMT

CQRS has become another casualty of the buzzword culture and the cult of 'shiny things'

What started as a collection of some reasonably good principles has now turned into an almost religious mantra for some, with more and more outrageous claims, and almost no basis in fact or experience.

Scalability

Claims like "almost infinitely scalable systems" associated to CQRS are ludicrous, especially as none of the currently "near infinitely scalable systems" use CQRS or have probably even heard of it. The largest systems on the planet have no need of CQRS, they use many tricks and techniques that have been associated with CQRS - but they don't use it, nor are they ever likely to use it, because when you have really large systems, things like CQRS become totally irrelevant.

The scaling benefits of CQRS come from understanding eventual consistency and messaging - concepts that were around long before CQRS, and that will be around long after it too.

Task Based UIs

Well, yes CQRS may require a task based UI, but task based UIs were again around long before CQRS, and will again outlast it. This is just a different way of thinking about how people interact with computers - this is a UX issue, not a technical one.

In fact, if anything, CQRS can be a major cause of issues here - when you really have a CRUD style system, CQRS is going to cause you all sorts of issues you will need to resolve. And much as I dislike CRUD based UIs, the reality is we have abused users to work this way in many cases - so sometimes that is just how we have to make our next UI work.

Side Effect Free

Another aspect of CQS (and by extrapolation CQRS) is that functions employing CQS are side effect free - you don't accidentally introduce subtle bugs by writing and reading in one operation.

And while this is nice in theory, the scale of system where this preventative measure would have benefit is the one where this problem is just as easily solved by many other simpler means, like just writing smart code. Larger systems have so many points in them where read and write are already being mixed, CQRS here just lends an illusion of safety.

Denormalised Read Models

A valuable benefit of CQRS is the concept that the views have specific read models that are denormalised from the (presumably) complex domain model.

This has a number of benefits, though these are largely boiled down to improved efficiency of querying (and action that tends to happen frequently). These models can also be extrapolated out to provide reporting and MI models too.

But again, we have been doing this for many years already, and usually to account for overloaded database servers that we don't wish to query against, particularly with MI.

And querying these days, even with horribly complex joins on highly complex models is becoming faster and faster. Push querying to a slave server, and you gain all the benefits of a rich model, and fast queries. And for even more speed you can look to NoSQL solutions - which are effectively based on the principle that you denormalised your data anyway.

Performance Considerations

Part of the theoretical benefit of CQRS is that your queries will be faster as your denormalised datastores are updated at write time, so you don't have to do horrible joins on queries (which in theory on most systems happen far more frequently than writes)

These days, the performance of relational databases and especially NoSQL databases is advancing rapidly. The new breed of databases takes things like sharding and replication very seriously, and as first class features, not afterthoughts like the databases of old did.

Replication in MySQL takes roughly 0.0002 of a second (see http://nr-content.s3.amazonaws.com/railslab/videos/17-ScalingRails-Scaling-Your-Database-Part-1.mp4 at 6 minutes in) - compare that with how long a message based approach like CQRS will take to push that to your secondary datastores and you are talking orders of magnitude faster for the dumb replication approach. Apparently it operates at this speed for up to 10 slave databases before degrading.

Now, think about this bit, the 37 Signal systems (a huge system) operates on a single MySQL instance running with 105Gb of memory caching. And 37 Signal could be considered the archetype of an Active Record approach, after all it is the reason Ruby on Rails exists. Are you writing a system that big?

Maybe you are so let's examine a larger system - Wordnik stores over 12 billion documents, 3Tb of data per node, 500k requests per hour and 4 times that at peak load, and yet their fetch time is around 60ms. (http://blog.wordnik.com/12-months-with-mongodb)

Let's not even begin to discuss Twitter, Ebay, Facebook or other similar "mega systems" - they still respond in milliseconds with tens or hundreds of millions of users - think that's down to CQRS?

Dealing with Complexity

Yep - CQRS deals with complexity to one degree - it forces you to explicitly think about what each call is doing - "Is this writing data or reading data?" - and that thought process is a good thing.

But it introduces another layer of complexity that is more subtle and harder to deal with - the replication to secondary datastores is complex in itself - it effectively needs to do that horrible join you are trying to avoid in your queries so that it can update your denormalised tables. That in itself isn't simple.

But it also needs to be able to maintain those secondary datastores if, for example they are corrupted and need recreating or if they miss some messages. In normal replication this is trivial, drop the slave, add a new slave - wait a short while - someone already wrote all that for you on a database system - you are going to need to write it all yourself with CQRS.

So Why Have I Implemented CQRS Before?

Well, at the time I first implemented CQRS I had two reasons for doing so - and oddly neither had to do with any of the aforementioned technical aspects. In fact, one was a technical problem and one was a people problem.

Firstly this was nearly two years back, and the system that we were writing a layer over was a very old and crumbling legacy database system. It had no real normalised schema, and was open to abuse from just about every development team in the organisation without warning to others - it was also horribly prone to outages.

One project had just "failed" (though I'm sure some would say it was just a lesser success), primarily due to trying to put a domain model on top of the legacy database.

So, the problem we had was to create a disconnected system that could operate independently of the legacy system, but also use the legacy system as it's true authority for the information it used. Because of the horrible complexity in the legacy system, using CQRS allowed us to write what were frankly quite horrible SQL statements for reading data from legacy, while using a pseudo-document database for the daily operations. We then used horrible SQL statements to replicate this data back into legacy.

Of course, what we were really writing was a standard message based architecture, but the term CQRS encompassed that quite nicely.

We could have approached this without the CQRS badge - but it served as a useful political tool - a number of the existing team had heard of it and were quite enthusiastic, and management would be confounded enough by it to not get in the way too much. It also broke the mental model the development teams had with "put an ORM over the database".

So the secondary benefit of CQRS here was a people problem - it was a new buzzword - it got new enthusiasm from a team that was demotivated from the previous project - and more than anything - that motivation was going to be the key to the success or failure of the new project.

So Why Have I Presented On It Before?

Well two reasons really, the first being that I have always tried to dispel the mythology around CQRS and to explain the real benefits of a flexible and scalable architecture - it just so happens that people want the buzzword explaining - you have to admit it's a pretty vague and fuzzy term with a myriad of definitions and explanations.

The secondary reason being, as my last presentation at DDD Sydney explained - my objective was to make you think. There's a lot of good things hiding under the CQRS banner, and I really would like people to think about better ways of writing software.

Conclusion

In the few years since CQRS was "invented" it has got a bit of a cult buzzword status. It is talked about as though it fixes a multitude of problems.

But in that time, NoSQL has really taken off. RBDMS' have got far more powerful. And memory and processing power has increased significantly. Database caching is exceptionally impressive. REST gives us proxy caching for near enough free. Memcached and Velocity make light of massive data loads. We even have the nefarious "cloud" now, where processing power and memory is cheaper than the effort involved in maintain complex architectures.

Scaling systems is now a pretty commonplace business problem for most of the startups taking off -and they don't need CQRS - there is a very good chance you don't either. We are past that point in time where we had to prematurely optimise things like database queries.

If it's about thinking differently, then I can tell you all you need to know about CQRS in a simple paragraph:

Learn to think of systems as being inconsistent at all times, no matter what illusion they present of 'real time'. Learn to embrace messaging, REST, caching, replication, sharding, and a hundred other techniques to solve these problems. Embrace user centric task based UIs, embrace strong UX, embrace strong domain driven design, embrace business led functionality and delivering systems that match real requirements. Embrace thinking differently.

CQRS is no silver bullet, in fact it has almost now become a solution to a problem that doesn't exist any more. If you can give me a problem that CQRS claims to solve, I'm fairly sure I can solve it more easily than I could have done 2 years ago when I started using CQRS. I'm sure I'll still use a lot of the things that CQRS encapsulates, because they are after all very good principles - I'm just pretty much over playing buzzword bingo!

Ruby Is Scary

Jak Charlton — Sat, 31 Jul 2010 01:55:00 GMT

Why is Ruby on Rails all the rage at the moment, and why do a lot of .NET people seem so defensive?

Undoubtedly, there is a buzz in development right now, things are changing rapidly, possibly more rapidly than they have for a good number of years. New languages are sprouting up all over it seems, with PHP and Python gaining massive popularity, F#, Erlang and Haskell suddenly being talked about as mainstream, Javascript evolving beyond that thing that makes "onClick" work in your browser, and Ruby (and especially Rails) becoming "flavour of the month" for web development. Ruby is even starting to invade the .NET mainstream space with evolutions like Rake, Cuke4Nuke and Nu muddying the divide that there once was. We even have IronRuby to make Ruby run on .NET

And yet, many .NET people seem defensive, as though their safe world was being invaded by a parasite.

I find this quite odd. As a developer who has been around for far too long, I have seen more than my fair share of languages and platforms come and go, none of them was perfect, and I would venture to say none of the ones we have now are either.

It would appear that a large proportion of the uncertainty is in the C# development 'community', though that may be as I don't really frequent circles that use VB.NET (and I'm fairly sure they are still busy being defensive about C# and probably haven't noticed that's *so* last year :) Somehow the introduction of new ideas is rocking the C# type safe world.

Some people seem to be concerned about Ruby itself, some seem to have concerns about the lack of a statically typed safety net, and some seem to think that "the Enterprise" needs something more professional than toys like Ruby and Rails.

To The Static Typing Stalwarts

Yes, it's nice to have that compile time checking that static typing gives you, it gives us all a warm sense of fuzziness knowing that the compiler told us that this code would run just fine when we dropped it onto our web server.

Yes, we appreciate the great benefits that this has for things like Intellisense and for easily discoverable APIs, and even for automated documentation.

And yet, none of these things seem to bother the Ruby (and PHP/Python/Javascript) crowd too much. Why is that?

I would suggest that the static typing benefits a language like C# brings are illusory to a large degree.

The compiler isn't telling us that this code will run correctly at all, it's actually just telling us that the CLR won't throw our code out before the application even starts up - but correctness is so much more important than that. The safety net static typing provides us with is just one part of the equation, and possibly the least important part - we really should be enforcing correctness upon our code.

Correctness is verified by good automated testing, and static typed languages start to come back and bite us here - we trade that compile time security blanket for a much harder experience testing our code. Sure, tools like NUnit, xUnit, Rhino Mocks, Moq, SpecFlow, StorEvil et al ease away our pain, but still our application code is littered with constructs that are pretty much there to support SOLID (or SOLID is required due to the constraints of type safe OO languages), and they become most obvious when we start testing in languages like C# (I am *so* over Interfaces, IoC, Covariance, Contravariance and Generics !

The testing experience in Ruby is a world apart from in C# and .NET, it is just easy and simple ... so simple in fact that it is almost harder to not write automated tests in Ruby - and that's how it should be. In C# testing is often convoluted, complex, fragile and just plain hard work - I can't think of the last time a C# developer described automated testing as fun ... and yet, in Ruby, it just seems like FUN!

And yes, Ruby without automated testing leaves you open to all kinds of runtime errors that C# just wouldn't - so in Ruby, you just make sure you test, and not only test your code will run, but also test it will run *correctly* - there is no safety net, but the trade off is you are ensuring correctness of code, not just compilation of it.

Intellisense, sure it's nice, but as RubyMine and even Visual Studio (under Javascript) demonstrate, you can do some form of Intellisense in dynamic languages, you just can't always be certain what may or may not be available - but this is getting better. And honestly, is discovering which method you are meant to call though typing "." all that great an idea anyway? Shouldn't you as a competent developer know *in advance* what you wanted to call, rather than guessing it as you type?

To The Enterprise

Sure, Ruby doesn't have the track record that C# has, nor does it have the exposure, or market reach, or backing of a company like Microsoft.

But, is that what is really important?

I would bet that a large part of the appeal for statically typed languages in the "Enterprise" is actually a need to deal with a much lower skilled developer base overall. Compile time safety means that your developers don't have to be top notch, the compiler can make up for their weaknesses (and in the case of C# and .NET, Visual Studio 'drag and drop' tooling is trying to make up for a lot of the rest)

Enterprises tend to see dollars and cents, not people and value. So anything that lets them hire cheaper developers to churn out more product must be good.

Again, I would argue this is a false economy. Sure that team of 30 developers may give you a warmer fuzzier feeling inside than a team of 4. And sure, the quantity of code those 30 can type out will probably be much more.

But this misses some real issues around productivity and value - a 'good' developer is 10x as productive as an 'average' developer, and more importantly, the 'average' developer will never be able to create the same quality level as the 'good' developer, even given a near infinite amount of time. No matter how much time a hobbyist painter spends with oil and canvas, he will never create a Leonardo. And as a Jackson Pollock will amply demonstrate, sometimes less is more.

So are you measuring bodies or productivity? Surely it is the end result that matters, not the comfort factor of people looking busy.

And, when you get a good developer, and let them have the freedom of a dynamic language, they can spend less time fighting compile time safety and language constructs that exist almost solely for that type safety, and more time concentrating on business logic and the end product.

In fact, the best argument for why Ruby is not suited to the Enterprise is ... the Enterprise probably already has a large investment in one platform and one core language (or maybe two), and the cost of retraining all those developers, support staff, renegotiating supplier contracts, etc etc is just disproportionately expensive. Lethargy is rife in the Enterprise, and monoliths take a long time to move - sure it's easier for startups to hop on new technologies and run with them, and often it's why they quickly challenge the status quo. This isn't really a technical issue, this is a political and management one.

Maybe Ruby and Rails will gain traction here, maybe not. Things like functional programming are certainly starting to grow roots in financial services, and things like Javascript are prevalent in web apps in the Enterprise so maybe dynamic programming in things like Ruby isn't so far off. It just takes one small faction to show how well they solved a particular problem with a new language or platform to plant that thought.

So Should We All Switch To Ruby Now?

Of course not, and maybe this is what the defensive people in .NET miss. The Ruby people aren't saying "you guys have it all wrong, convert now", they are just saying "hey, I found a really cool way of solving this problem".

Ruby is about choice, it's another tool in your arsenal. As are Javascript, PHP, Python, Erlang, F#, Java, Groovy, C++ and a host of others.

There is no "one right path", so let's all stop pretending there is.

We don't need to be defensive about any one language or platform, unless we are so unsure of our abilities as developers that we cannot learn and adapt - and if that is the case, maybe a new career is a better idea?

As my recent presentation at DDDSydney said ... My Objective Today was to make you think ... "Maybe There Is A Better Way" - because it would just be depressing to think we are already doing things the best possible way.

My Objective Today is to Make You Think

Jak Charlton — Mon, 19 Jul 2010 00:08:00 GMT

"Maybe There is a Better Way"

I recently presented at DeveloperDeveloperDeveloper in Sydney, and although my talk was Stuff About CQRS, I opened with the slide

My Object Today Is to Make You Think ... 'Maybe There is a Better Way'

(slides here)

The real focus of this was around how normal people think, and how unlike normal people we developers really are. My role in development is all about enabling better communication, because fundamentally I believe the real value a developer brings to a project is not technical, but is in the way they interact with the team, and more importantly with the normal people they are actually creating software for.

Obviously some of this has roots in DDD, the Ubiquitous Language is obviously an attempt to traverse this chasm that seems to exist between us.

Then I touched on user interfaces, and how they are so rarely designed the way people think - normal people think about their objectives and goals, not in terms of data like we developers do. Inductive UIs are focused on tasks, unlike the traditional data driven UIs that we tend to throw at users - users and people don't think in grids and columns and rows, they think "I want to change my address", not "open my customer record, edit the three fields under address and save to the database" - only a sadist or a developer would think that way.

And finally I touched on things like NoSQL databases, which neatly solve a communication problem - they stop us thinking about How to store information and let us focus on What we are storing and Why.

And lastly I tried to show the link between CQRS and these business problems - how it made you focus on the language, on tasks and objectives and how it let you detach the How from the What and Why.

But most importantly, what I was trying to do in that presentation was to throw some non-mainstream ideas out into the audience, to spark discussion and debate, and to get people to think - Maybe There is a Better Way

If only a few of those in the audience went away and Googled some of the ideas I was talking about, it will be another step towards moving development away from it's heavy focus on technology and technological solutions - and towards a people and business driven focus, where technology is an artifact, not the deciding factor.

ThinkDDD Resource Site

Jak Charlton — Tue, 04 May 2010 00:08:00 GMT

For a while now I have been wanting to replace the creaking DDDStepByStep.com site - I originally set it up using Community Server, but it soon became obvious after deployment that CS just wasn't up to the job - the server regularly ran out of resources just running the vanilla product and became a pain to keep rebooting.

So, to try yet again to sort out the information, I setup a new website thinkddd.com and used it as an opportunity to play with Ruby, and the Radiant CMS. Deployment to Heroku for such a low bandwidth site also worked out as free - so the new site cost basically nothing except for time.

The slides from my recent presentation to the Sydney alt.net user group are available from here.

Hopefully it is a lot clearer and easier to navigate, so if you are interested in Domain Driven Design - enjoy.

Agile is Performance, Feedback, Revision

Jak Charlton — Thu, 15 Apr 2010 02:45:08 GMT

I came across a great video link yesterday, entitled “Performance, Feedback, Revision”. It’s a Canadian rapper named Baba Brinkman covering the theory of evolution and the work of Charles Darwin, and he equates evolution with how he writes his lyrics, Performance, Feedback, Revision.

As I listened to it, it struck me that this is Agile … we develop, we run retrospectives, and we revise what we have.

Evolution may be more like waterfall, and take millions of years to evolve something as fundamentally pointless as the appendix, and in Agile we may look to have Performance, Feedback and Revision inside a week or two.

I can’t quite believe I just showed this video to a client to explain Agile, but it seemed to get the idea across pretty well. I’m considering using it more often, but I promise to draw the line at learning to rap.

But from now on, in big letters across the top of my white boards will go the words:

Performance, Feedback, Revision

Around the World in 90 Days

Jak Charlton — Tue, 02 Mar 2010 06:38:00 GMT

Around 3 months ago, I packed up in the UK, and headed to the other side of the world, to Australia.

Last Autumn I was chatting to Richard Banks over Twitter, and he mentioned that Readify were looking to hire good .Net developers in Australia. I joked that if they wanted to pay my air fare over I would certainly do an interview.

Sadly Readify don't pay for the air fare, but after a bit more joking I took Richard seriously and dropped them a copy of my CV. A few months later, after a fairly thorough interview (part of which took place over Skype and Live Meeting... sadly the reason for no free holiday), I took the decision to join Readify as a Senior Consultant.

For the last 18 years I have been a self employed consultant / contractor in the UK, and have made a fairly good living at it. In fact this will be the first time I have been an employee since I was aged 22. So this was a fairly big decision.

What swung it for me was the general positive attitude of the Readify people I spoke to, and their focus on technical excellence and good development practices. This is a company that is rapidly growing, and despite the general world economic conditions, Australia managed to avoid the vast majority of the financial crisis.

As part of their expansion, Readify are still recruiting, and due to the shortage of good developers in Australia, they are willing to hire from overseas, and deal with all of the immigration and visa issues needed to get you into the country. So far at least one person I have chatted to over Twitter has applied, and if you look through the list of Readify employees you are likely to see a number of people who are highly respected in the .Net world - it's a great bunch of people.

Apparently only 1% of applicants to Readify get through the interview process, so if you are looking for a new career, are open to moving, and are pretty darn hot around .Net and good development practices, maybe you should think of applying. http://readify.net/Careers/

ASP.NET MVC in the Wild

Jak Charlton — Tue, 09 Feb 2010 11:02:00 GMT

Just before I emigrated to Australia, I took on a small contract to build a website for a UK company who wanted to start up a new kind of UK recruitment site, one where employers could advertise directly, and more specifically one where recruitment agencies couldn’t.

The result was Empty Lemon (www.emptylemon.co.uk) – I leave you to figure out why it is called Empty Lemon, took me a fair while to be honest!

This site was built in around 4 weeks, using ASP.NET MVC2, NHibernate, Fluent NHibernate, Windsor, JQuery, Automapper, and a host of other great tools and libraries.

The site is actually, despite the occasional bit of code I’m not too delighted with, a great example of how fast a site can be developed with ASP.NET MVC and the chosen tooling. In just four weeks we went from more or less nothing, to a full domain model (all be it a fairly simple one) running over a SQL Server 2008 database, using a combination of NH, Fulltext search and the Spatial services from SQL Server 2008.

I also want to thank James Gregory, Steve Strong and Hadi Hariri for (sometimes unknowingly) helping me write the site, with useful snippets of code, and even in Steve’s case a lot of work on getting NH to play fair with the spatial services.

So, if you are looking for a job in the UK, or if you are looking to employ someone, I strongly suggest you take a look at www.emptylemon.co.uk. It’s a little empty right now, but some large employers are coming on board soon, and other companies are signing up day by day – these things are always chicken and egg, so if you like the site, feel free to tell others about it – after all, who really likes recruitment agents anyway ;)

Solving Complexity with Message Based Architectures

Jak Charlton — Thu, 04 Feb 2010 07:55:00 GMT

Ultimately most complexity in software comes not from the requirements, the business logic, or even the underlying systems. Most complexity comes out of a poorly considered and managed architecture, and this is commonly seen in tightly coupled systems that rapidly degrade into Big Balls of Mud.

The key to simplifying architectures is to decouple their component parts, making each more specialised and less dependent on the others. This allows every component to focus on a specific task without risk of those tasks becoming compromised by, or interfering with, others.

The basic premise of most architectures that succeed in this way is that they are message based, rather than being glorified CRUD systems pulling data to and from databases. Operations are central to the system, data is ultimately just the storage for the results of those operations.

This separation of concerns allows development teams to concentrate on component parts with little impact upon other teams, and no more reliance upon them than an agreement on the messages they will listen for and those they will publish. While messages share some characteristics of service interfaces, they differ mostly in their granularity and lack of dependence upon one another. Messages allow true decoupling, where traditional service layers only promise this and usually end up creating even more tightly coupled systems.

"Message based" does not inherently mean SOA, and in many ways differs substantially. Although both can be considered message based, SOA brings significant overhead and generalisation, along with an organisation wide remit. Simple message based systems operate on the basis of message within the application rather than across an enterprise.

Poor software architecture ends up impacting more than just the software system under development, and frequently the impact can be felt across an organisation; in the way development teams approach problems, in the way they respond to new requirements, and throughout the SDLC.

More often than not, the SDLC is a reflection of the weaknesses in software architecture, and is frequently trying to play 'catch up' to attempt to deal with the issues weak architecture brings. These compromises and aspects of the SDLC then start to propagate and impact other areas of the development and support process. Often decisions about software architecture become the underlying drivers for the entire IT department, and the SDLC turns into a point of conflict as it becomes more about trying to restrain and control a big ball of mud than it does about providing a framework for providing business value.

What Is The Difference Between an IoC Container and MEF?

Jak Charlton — Fri, 18 Dec 2009 06:50:00 GMT

Today a conversation sparked off on Twitter, started by Jimmy Bogard and Matt Hinze, and then carried on by myself and Glenn Block. The basic starting point was what the difference was between using an IoC container like Windsor or StructureMap and using MEF (the Managed Extensibility Framework)

I started off by answering a question from Jimmy, he wanted the one paragraph sales pitch for MEF that didn’t include technical terminology … I guess he was asking why MEF is a better option than one of the IoC containers we all love and hopefully all use religiously.

My key answers were:

MEF allows you to write your application as a series of much simpler mini applications, and have those brought together to create a much great whole

IoC container = service level components, MEF = Autonomous Components. A MEF "part" will probably use an IoC container

This sparked some comments from Glenn where he wanted to clarify that sometimes an IoC container is the right choice, sometimes MEF is, and sometimes both are. Luckily we are both pragmatic, so not much to disagree with there. He pointed to two of his blogs on the subject, I recommend reading them highly, they will do a much better job than I could at explaining the detail of the decision points, Should I use MEF for my general IoC needs? and Should I use MEF with an IoC container?

What’s Wrong With IoC Containers?

What an IoC container is very good at is low level control of your code, your dependency graphs, your lifestyle management, and gives you great low level ways to deal with facilities, factories, decorators and AOP. In just about any code base that goes past spike stage, I use an IoC container of some sort, it is just a fools errand to try and write and application without one these days. The keyword “new” is in my code smells book.

However, what an IoC container is pretty poor at is enforcing boundaries. Sure you could use an IoC container to write much larger applications, but as Glenn rightly pointed out, the central configuration becomes a headache. In fact, the central registration can rapidly become the equivalent of using global variables, you never really know what is using what as everything has access to everything else.

Most of the containers give you the capability to have child containers to try and deal with this situation, and while they do limit the global access issue, they are pretty hard to manage in their own right, and you can without some proper thought end up with a spaghetti mess of containers replacing your spaghetti mess of components.

An Architectural Solution To The Problem

To me, MEF is an architectural solution, while an IoC container is a code level solution. They both have their places in an application, but what MEF is exceptionally good at is creating process boundaries.

One of the core underlying principles of good use of an IoC container is that you only ever reference a container at a process boundary – in most applications that means it is probably referenced in your bootstrapping code inside your web application (maybe Global.ascx), or inside an HttpModule around your web services, but from that point on, your code should never reference the container again – the magic of an IoC container comes from relinquishing all object construction and dependency graph resolution to it.

But this means in a larger application, you may end up with massive containers with hundreds of registered components, and in reality most operations will only use a small handful at a time.

In most larger systems you would naturally have parts of the system you want to devolve into discrete components, or Autonomous Components. Doing this with an IoC container starts to lead you down an architectural design geared around the limitations of the IoC container, and not necessarily driven by good architectural principles.

What MEF allows you to do is to break the system apart easily, or as Glenn referred to it, MEF allows you to easily create composite applications. These composite parts can operate as Autonomous Components, in almost total isolation from the core application, and therefore they provide a natural boundary without having to use things like web of WCF service layers just for that purpose. Oddly the MEF terminology that Jimmy wanted to avoid in his original question was “part” and yet it fits naturally in this context.

Russian Dolls

As I started off by saying, I would almost always end up using an IoC container in any application I was to write, even a fairly small scale one. If we look at these MEF parts as now being Autonomous Components, and therefore operating as mini applications, it now becomes apparent that they will themselves need a way to create service components and dependency graphs, and while MEF could in theory do a lot of that, it’s just not the best fit for the job.

So it’s pretty certain that inside your MEF part you will be using yet another IoC container for object resolution, but now you don’t have a massive central registry, but one specifically geared to the MEF part in question.

And this pattern can recurse if needed, each time providing a clean boundary within your application.

A Real World Example

The last “large scale” system I worked on where I had the opportunity to put some of these ideas into practice was an insurance portal. This portal was responsible for providing insurance quotations, but across a massively diverse and inconsistent set of products.

The solution we came to in the end was a combination of all of the above principles – we had a web application that held a Castle Windsor container in Global.ascx, that container was responsible for instantiating everything from controllers backwards, and all of the service components they used.

But some of those controllers needed to call our product plugins, and in this case we actually used Castle Windsor to use MEF to instantiate the plugins (MEF parts). Within each product plugin we had another set of bootstrapping code that had it’s own instance of a Castle Windsor container, and it’s own registry specific to that plugin.

The plugins were actually used by a number of points in the application, some in process, and some out of process, either on other servers or just behind service or service bus boundaries – what MEF allowed us to do was create an independently deployable package that was self contained and self sufficient, without the need for managing masses of registration information – trying to use an IoC container for this would have been a mini-nightmare, but with MEF it was almost a pleasure.

Combining NHibernate, NH Spatial and SQL Server 2008 Fulltext Queries

Jak Charlton — Sun, 22 Nov 2009 22:42:00 GMT

Following Kevin Pang’s request on my last post regarding my problems getting NHibernate, NH Spatial and SQL Server Fulltext queries to work, I am finally getting around to giving some tips and pointers, and the code that Steve Strong of iMeta knocked up for me.

I needed to get these things working together as I was writing a recruitment website, and they had two basic requirements, the ability to search in candidates CVs and job descriptions for keywords, and the ability to search for those things within a certain distance – so an obvious search would be “find me all the C# developers within 10 miles of Central London”

Step One – NHibernate

Firstly, I’m not dumb, and at the beginning of the project I had decided to use NHibernate for all data access. The site didn’t have much of a domain model, but what it did have was ideally suited to a relational DB and simple persistence and querying by an ORM like NHibernate. What it also had going in it’s favour was the 2nd level caching capabilities of NHibernate, which would significantly reduce the workload on a site that was expected to grow quickly (or at least the owners hoped it would)

Win for NHibernate there – the icing on the cake was the excellent NHProfiler which would prove to make development a breeze over other options, and which also proved to make my querying a lot smarter and more bug free than my first horrid attempts.

Step Two – Spatial Searching

MS SQL Server 2008 introduced Spatial searching to it’s list of features – in fact it’s not just for searching, and not just by distance either, it can do a host of clever geographical type calculations and querying.

My first tests were done with “old fashioned” T-SQL and within a short while I had a reasonable distance search working, roughly speaking it looked like:

SET @from = geography::STGeomFromText('POINT(-0.094693 51.520223)', 4326); -- EC2M (London

SELECT * FROM JobSlot j
WHERE j.Location.STDistance(@from)<=(@distance * 1609.344)

Step Three – Fulltext Searching

Long ago, back in the age before the dawn of time, MS SQL Server had Fulltext searching, via the FREETEXT and CONTAINS syntax. Give it a keyword, or a list of them, and it will dig inside your text to find just what you want.

To use it on the site I was writing, we needed to index the content of CVs (resumes to your foreigners!) and some of those were in docx format – so my first step was to download the Microsoft Office 2007 iFilters. SQL Server uses the (now) age old technology of iFilters to extract the text from documents, and this filter pack allows it to look inside Office 2007 documents – if you want to index any other weird document formats, or even your own document format, there is probably an iFilter available, or you can write your own. With the iFilters installed, SQL Server could now index all of my content just fine.

SELECT *
FROM Candidate
WHERE FREETEXT(*, N'c#')

Combining It All

So far, so good. SQL Server can do pretty much all I need it to do, and combining those two queries is simplicity in TSQL, but as I was using NHibernate for all my data access, I wanted to try an get it all working there too.

NHibernate has support for the Spatial features of SQL Server 2008, along with other spatial databases, via the NHSpatial stuff in the NH Contrib project at http://nhcontrib.svn.sourceforge.net/svnroot/nhcontrib/

A download and compile of this will have you making HQL queries to do similar things to my TSQL spatial query above, in fact, the HQL query is remarkably similar to the TSQL:

select j from JobSlot j where NHSP.Distance(j.Loc, :point) <= (:distance * 1609.344)

Then I needed to get the Fulltext part working, after some chatting with Steve Strong, he came up with the following code:

            WKTReader wtk = new WKTReader();
            IGeometry point = wtk.Read("POINT( -0.123356 51.524142 )");
            point.SRID = 4326;

            string query =
                @"select j from JobSlot j
                  from freetexttable('JobSlot', :keywords) ftt
                  where NHSP.Distance(j.Loc, :point) <= (:distance * 1609.344)";

            var results = session.CreateQuery(query)
                .SetParameter("point", point, SpatialDialect.GeometryTypeOf(session))
                .SetParameter("distance", 1)
                .SetParameter("keywords", "hello world")
                .SetFirstResult(5)
                .SetMaxResults(10)
                .List();

And to get this to work we need 2 more classes, though the FreeText class is more an example of how to create an ISqlFunction for NHibernate:

	public class MyDialect : MsSql2008GeographyDialect
	{
		public MyDialect()
		{
			RegisterFunction("freetext", new FreeText());
			RegisterFunction("freetexttable", new StandardSQLFunction("freetexttable", null));
			RegisterFunction("contains", new StandardSQLFunction("contains", null));
		}
	}

	public class FreeText : ISQLFunction
	{
		public bool HasArguments
		{
			get { return true; }
		}

		public bool HasParenthesesIfNoArguments
		{
			get { return true; }
		}

		public virtual SqlString Render(IList args, ISessionFactoryImplementor factory)
		{
			SqlStringBuilder builder = new SqlStringBuilder();
			builder.Add("freetext(");

			if (args.Count == 1)
			{
				builder.Add("*, ");
				builder.AddObject(args[0]);
			}
			else
			{
				builder.AddObject(args[0]);
				builder.Add(", ");
				builder.AddObject(args[1]);
			}

			builder.Add(")");

			return builder.ToSqlString();
		}

		public virtual IType ReturnType(IType columnType, IMapping mapping)
		{
			return null;
		}
	}

Why I’m Not Using The Above Code

Yep, in my usual fashion I go through explaining all of that, just to say, I’m not doing it that way.

Eventually, I went back to another approach, that of using TSQL dynamically constructed in my code. I did this for a number of reasons, but primarily as it was easier for me to tune the query and add more options and variants in old fashioned SQL than in HQL by adding more extensions on to NHibernate. It also left the code base in a much easier to follow and extend state for the developers who would be maintaining the project – with the amount of investigation it took me to get this far, I figured extending it could be a nightmare for my successors.

The benefits I got from NHibernate I still retained, as this is a search function it was returning DTOs anyway, not entities, and these were just as efficient to read with a DataReader as with NHibernate (in fact a lot easier). When one of the results from the search is selected, the item is retrieved by NHibernate and we are back in ORM land.

It was also little effort to cache my SQL query results, so I also had all that NH was offering me in that respect too – a simple hash of the query parameters allowed me to cache the results object.

So there ya go, how to use Spatial and Fulltext with NHibernate, and why I didn’t do it! :)

When NHibernate Won't Do What You Think It Should

Jak Charlton — Mon, 02 Nov 2009 13:57:00 GMT

There are times, when you just think something should work, and one way or another it just doesn't. I've been having one of those weeks with NHibernate. I have been using MS SQL Server 2008 spatial searching, along with SQL Server Full text search, and using NHibernate for almost all my data access.

The only problem is that NHibernate out of the box doesn't like to play nicely with Spatial or Fulltext. After head scratching, head bashing, and general teeth grinding I figured better to cut my losses and ask some people that actually knew what they were doing.

Firstly I fired a twitter off in the direction of Ayende to get a feel for whether I was doing something wrong - with CreateSqlQuery NHibernate didn't want me to use the Spatial stuff - NHibernate uses a colon : to identify parameters, but in SQL 2008 Microsoft decided to use a double colon :: in it's spatial bits, so for example to declare a point you would use:

geography::STGeomFromText('POINT(51.520223 -0.094693 )', 4326)

That promptly causes NH to stop you from carrying on with a missing parameter. Ayende did indeed confirm it's not right, and after a few helpful suggestions he ran out of quick wins and suggested I might want to take advantage of his commercial support for NHibernate. I'm not adverse to such a solution, but felt this one might get complex, and Ayende's payment structure didn't fit my needs - this is a small project for a small client, and I could work around NH's limitations by using stored procedures (yes, I know, heresy).

After a bit more head scratching I remebered that iMeta also provide NH commecial support now, and having taken a look at their pages, I figured it would work much better for my limited budget.

After a quick chat with Hadi Hariri, he directed me on to Steve Strong, the man who wrote a great deal of the Linq to NHibernate stuff, and a bit of an expert on the parser.

Steve ripped through my queries with lighning pace, and had me a viable solution that afternoon - now that's what I call customer service. Today Steve has continued to sort out niggles with the problem, and it looks like we may have identified a few bugs in NHibernate Spatial and possibly the parser, which Steve will now push back into the NH trunk for future geenrations to benefit from.

So, althought NHibernate is open source software, and in theory you can fix this stuff yourself, this is just a post to point out that sometimes the best use of your time and effort is in things you can deal with quickly, and sometimes it is worth paying some real specialists to help out - the cost saving could be immense. If you have problems with NHibernate, I encourage you to take a look at the offerings available, especially the iMeta one :)

Just because NHibernate is free - doesn't mean it isn't worth paying for!