Solving Complexity with Message Based Architectures

Jak Charlton — Thu, 04 Feb 2010 07:55:00 GMT

Ultimately most complexity in software comes not from the requirements, the business logic, or even the underlying systems. Most complexity comes out of a poorly considered and managed architecture, and this is commonly seen in tightly coupled systems that rapidly degrade into Big Balls of Mud.

The key to simplifying architectures is to decouple their component parts, making each more specialised and less dependent on the others. This allows every component to focus on a specific task without risk of those tasks becoming compromised by, or interfering with, others.

The basic premise of most architectures that succeed in this way is that they are message based, rather than being glorified CRUD systems pulling data to and from databases. Operations are central to the system, data is ultimately just the storage for the results of those operations.

This separation of concerns allows development teams to concentrate on component parts with little impact upon other teams, and no more reliance upon them than an agreement on the messages they will listen for and those they will publish. While messages share some characteristics of service interfaces, they differ mostly in their granularity and lack of dependence upon one another. Messages allow true decoupling, where traditional service layers only promise this and usually end up creating even more tightly coupled systems.

"Message based" does not inherently mean SOA, and in many ways differs substantially. Although both can be considered message based, SOA brings significant overhead and generalisation, along with an organisation wide remit. Simple message based systems operate on the basis of message within the application rather than across an enterprise.

Poor software architecture ends up impacting more than just the software system under development, and frequently the impact can be felt across an organisation; in the way development teams approach problems, in the way they respond to new requirements, and throughout the SDLC.

More often than not, the SDLC is a reflection of the weaknesses in software architecture, and is frequently trying to play 'catch up' to attempt to deal with the issues weak architecture brings. These compromises and aspects of the SDLC then start to propagate and impact other areas of the development and support process. Often decisions about software architecture become the underlying drivers for the entire IT department, and the SDLC turns into a point of conflict as it becomes more about trying to restrain and control a big ball of mud than it does about providing a framework for providing business value.

Commercial Suicide - Integration at the Service Level

Jak Charlton — Tue, 19 May 2009 07:03:00 GMT

In my previous post I described the problems with trying to integrate your organisation at the database level, and the fallacies surrounding the idea of the One True Authority Database. I also alluded to this being a problem with services too.

When you try to create a monolithic and authoritarian database that knows all, sees all, and brings everything into a single coherrent whole, you are heading down a slippery slope. Different applications have different requirements, and even within the same job role, the meaning of Customer can have a totally different meaning in two different contexts. I also argued that data has no value without context, and the context is the application that is meeting the business requirements, be these user interaction, system interaction, or reporting.

SOA to the Rescue!

I'm not arguing without ample evidence behind me here - the idea of integration at a database level was one that was well discredited many years ago. I grant you many organisations haven't got the message yet, and that many vested interests still keep trying to make the magic solution work, but fundamentally a long time back we in IT recognised that the One True Authority Database (OTADB) was a "bad thing" - and this lead us to the promised land - initially Web Services and then on to Service Oriented Architectures.

SOA is an attempt to resolve the OTADB issue - take all those database calls, and even calls to different databases, and unify them through a set of coherrent and cohesive services - SOA can provide us with the abstraction over out data stores that is required - and now we can have the One True Authority Service Layer.

With SOA we can define services to wrap our databases, services to update data, service to read and query data, and even services to unify those "different" Customers into a single coherrent and canonical data model.

And that's where it all goes wrong again.

Anti-Pattern: The Canonical Data Model

Back when our database guys were trying to integrate all this information, what they were trying to achieve, but maybe didn't realise, was a canonical data model - this is the "holy grail" of SOA. Martin Fowler describes it thus:

How can you minimize dependencies when integrating applications that use different data formats? …Design a Canonical Data Model that is independent from any specific application. Require each application to produce and consume messages in this common format

Sounds perfect! We now have a way of abstracting our systems from each other, and our canonical data model will pull this information all together at the end.

So, I'm going out on a limb here, disagreeing with Martin Fowler is rarely a smart move.

Except I'm not disagreeing with Martin Fowler so much as I am disagreeing with those that try to implement his ideas and principles. I don't recollect anywhere that Martin says that all data can be squeezed into this model, or that this model is the "one true view". In fact, later evolutions of Martin's work have included things like Domain Driven Design, which is almost the opposite of a canoncial data model - DDD says that you have a different model for every domain or context, each specifically engineered to solve the problem at hand. Given the choice, I am certainly in the DDD camp here - after all data without context is worthless, and it is the domain (and DDD) that provides the context.

The key and important part of a canonical data model is it provides a mapping between inconsistent formats and differing views of data.

So as an ideal, the canonical data model is fine - in practice it is often a major anti-pattern.

There is still no One True Authority, and SOA really never promised this, nor did the canonical data model - what they promised was enough abstraction to allow applcaitions to work independently, and to fulfill their specific requirements independently. And then to leave it to the service layers to unify this data.

So Service Oriented Architectures HAVE Come to the Rescue?

Not quite - again my argument here isn't so much about SOA, or any of the things around it, but more the people who try to implement it, often without understanding what they are trying to achieve, the problems that SOA or a service layer solves, or even the amount of effort that is required to get a canonical data model right.

The key part here is that a canonical data model is a massive excercise to do right. Essentially it means analysing every applciation, every service and every database within your organisation and extracting their individual requirements and current functionality, and trying to distill that down to a set of common definitions.

While this may be eventually achievable, in practice it is almost never the case. This "holy grail" of SOA is much like the "holy grail" of integration at the database level, eventually it boils down to "the problem is too complex, too fast moving and too indeterminate to define". In development we figured out long ago that we needed Agile to help us move quickly as requirements shifted and changed, but the OTADB and the SOA canonical data model do not allow this to happen, they seek to solidify the design early.

SOA done "right" may well make this problem smaller, but SOA is rarely done right.

Can Domain Driven Design Help?

DDD was largely an attempt to address these problems - by identifying that individual applications and contexts required different data, DDD allowed us to recognise that square pegs and round holes do not match. Eventually with a big enough hammer you may be able to force it through, but there is going to be a lot of effort and a lot of collaterel damage.

Domain Driven Design includes terms like "Domain", "Bounded Context" and "Anti Corruption Layer" specifically to deal with the issues around trying to use a "one solution fits all" hammer to crack what is quite a simple nut.

DDD is largely unconcerned with what other things Domains talk to, or even how, all that matters is that within a Domain there is one logical view of the world, tailored to the requirements of that specific Domain. Beyond that Domain, there may be a large central database, there may be a perfect canonical data model, but at application level, we have the Domain, Bounded Contexts and Anot Corruption Layers to protect us from all of that.

DDD gives us a loosely coupled way of thinking.

So SOA is Evil and the One True Authority Database is Evil?

Well hardly, but unfortunately we live in an imperfect world, and the people implementing these noble ideas are often far from perfect. They do not have mystical powers of future vision, and they do make mistakes.

By creating properly decoupled systems, and in this I certainly do not mean "systems that use web services", we can isolate our applications from external change and influence, and ensure we meet the requirements we have today, and can react quickly to future requirements as they appear.

Big Design Up Front is still BDUF, whether it is done at a database level, a service level, or in a canonical data model.