Commercial Suicide - Integration at the Service Level

In my previous post I described the problems with trying to integrate your organisation at the database level, and the fallacies surrounding the idea of the One True Authority Database. I also alluded to this being a problem with services too.

When you try to create a monolithic and authoritarian database that knows all, sees all, and brings everything into a single coherrent whole, you are heading down a slippery slope. Different applications have different requirements, and even within the same job role, the meaning of Customer can have a totally different meaning in two different contexts. I also argued that data has no value without context, and the context is the application that is meeting the business requirements, be these user interaction, system interaction, or reporting.

SOA to the Rescue!

I'm not arguing without ample evidence behind me here - the idea of integration at a database level was one that was well discredited many years ago. I grant you many organisations haven't got the message yet, and that many vested interests still keep trying to make the magic solution work, but fundamentally a long time back we in IT recognised that the One True Authority Database (OTADB) was a "bad thing" - and this lead us to the promised land - initially Web Services and then on to Service Oriented Architectures.

SOA is an attempt to resolve the OTADB issue - take all those database calls, and even calls to different databases, and unify them through a set of coherrent and cohesive services - SOA can provide us with the abstraction over out data stores that is required - and now we can have the One True Authority Service Layer.

With SOA we can define services to wrap our databases, services to update data, service to read and query data, and even services to unify those "different" Customers into a single coherrent and canonical data model.

And that's where it all goes wrong again.

Anti-Pattern: The Canonical Data Model

Back when our database guys were trying to integrate all this information, what they were trying to achieve, but maybe didn't realise, was a canonical data model - this is the "holy grail" of SOA. Martin Fowler describes it thus:

How can you minimize dependencies when integrating applications that use different data formats? …Design a Canonical Data Model that is independent from any specific application. Require each application to produce and consume messages in this common format

Sounds perfect! We now have a way of abstracting our systems from each other, and our canonical data model will pull this information all together at the end.

So, I'm going out on a limb here, disagreeing with Martin Fowler is rarely a smart move.

Except I'm not disagreeing with Martin Fowler so much as I am disagreeing with those that try to implement his ideas and principles. I don't recollect anywhere that Martin says that all data can be squeezed into this model, or that this model is the "one true view". In fact, later evolutions of Martin's work have included things like Domain Driven Design, which is almost the opposite of a canoncial data model - DDD says that you have a different model for every domain or context, each specifically engineered to solve the problem at hand. Given the choice, I am certainly in the DDD camp here - after all data without context is worthless, and it is the domain (and DDD) that provides the context.

The key and important part of a canonical data model is it provides a mapping between inconsistent formats and differing views of data.

So as an ideal, the canonical data model is fine - in practice it is often a major anti-pattern.

There is still no One True Authority, and SOA really never promised this, nor did the canonical data model - what they promised was enough abstraction to allow applcaitions to work independently, and to fulfill their specific requirements independently. And then to leave it to the service layers to unify this data. 

So Service Oriented Architectures HAVE Come to the Rescue?

Not quite - again my argument here isn't so much about SOA, or any of the things around it, but more the people who try to implement it, often without understanding what they are trying to achieve, the problems that SOA or a service layer solves,  or even the amount of effort that is required to get a canonical data model right.

The key part here is that a canonical data model is a massive excercise to do right. Essentially it means analysing every applciation, every service and every database within your organisation and extracting their individual requirements and current functionality, and trying to distill that down to a set of common definitions.

While this may be eventually achievable, in practice it is almost never the case. This "holy grail" of SOA is much like the "holy grail" of integration at the database level, eventually it boils down to "the problem is too complex, too fast moving and too indeterminate to define". In development we figured out long ago that we needed Agile to help us move quickly as requirements shifted and changed, but the OTADB and the SOA canonical data model do not allow this to happen, they seek to solidify the design early.

SOA done "right" may well make this problem smaller, but SOA is rarely done right.

Can Domain Driven Design Help?

DDD was largely an attempt to address these problems - by identifying that individual applications and contexts required different data, DDD allowed us to recognise that square pegs and round holes do not match. Eventually with a big enough hammer you may be able to force it through, but there is going to be a lot of effort and a lot of collaterel damage.

Domain Driven Design includes terms like "Domain", "Bounded Context" and "Anti Corruption Layer" specifically to deal with the issues around trying to use a "one solution fits all" hammer to crack what is quite a simple nut.

DDD is largely unconcerned with what other things Domains talk to, or even how, all that matters is that within a Domain there is one logical view of the world, tailored to the requirements of that specific Domain. Beyond that Domain, there may be a large central database, there may be a perfect canonical data model, but at application level, we have the Domain, Bounded Contexts and Anot Corruption Layers to protect us from all of that. 

DDD gives us a loosely coupled way of thinking.

So SOA is Evil and the One True Authority Database is Evil?

Well hardly, but unfortunately we live in an imperfect world, and the people implementing these noble ideas are often far from perfect. They do not have mystical powers of future vision, and they do make mistakes.

By creating properly decoupled systems, and in this I certainly do not mean "systems that use web services", we can isolate our applications from external change and influence, and ensure we meet the requirements we have today, and can react quickly to future requirements as they appear.

 Big Design Up Front is still BDUF, whether it is done at a database level, a service level, or in a canonical data model.


Posted 05-19-2009 8:03 AM by Jak Charlton



Sandor Davidhazi wrote re: Commercial Suicide - Integration at the Service Level
on 05-19-2009 5:43 AM

"Big Design Up Front is still BDUF, whether it is done at a database level, a service level, or in a canonical data model."

This shoud have been the starting sentence and the rest of the article detailing practical solutions. :)  Do you plan on posting about those?

Think before coding wrote re: Commercial Suicide - Integration at the Service Level
on 05-19-2009 6:25 AM

Once again, very good post jak !

Do you plan to talk about service buses then ?

Daniel wrote re: Commercial Suicide - Integration at the Service Level
on 05-28-2009 2:10 AM

Hi Jak!

I think that you have two very interesting points there. Using a message bus to synchronize data is an interesting solution to the application scaling problem. I think that your point is correct that data has no value if it's not in the right context. However, as one of the commenters in your previous post mentioned, it feels kind of strange to abandon the data handling capabilities that the DBMS actually has for a completely new messaging infrastructure. As he says, the DBMS is designed for maintaining data integrity.  

I know this is an old debate but what's your view on it?

Jak Charlton wrote re: Commercial Suicide - Integration at the Service Level
on 05-28-2009 3:43 AM


I am not advocating abandoning the things a DBMS is good at - I am advocating using it where it is the right tool, and using other options, or other patterns where they fit the problem space better.

What an RDBMS is good at is maintaining a relational model, where data is suited to, and suitable for, being stored in 3rd normal form.

What an RDBMS is also good at is allowing complex querying across those relations.

In practice however, data rarely fits the 3NF well, and querying across relational data is horribly inefficient and resource consuming.

Using a DMBS for persistence, for heavy lifting, and for large data manipulations are all good uses.

Using a DBMS (or service layer) for integration, maintaining conformity or consistency or business logic is a bad use.

I'll also reply to the commenter on the other thread too...

About The CodeBetter.Com Blog Network
CodeBetter.Com FAQ

Our Mission

Advertisers should contact Brendan

Google Reader or Homepage Latest Items
Add to My Yahoo!
Subscribe with Bloglines
Subscribe in NewsGator Online
Subscribe with myFeedster
Add to My AOL
Furl Latest Items
Subscribe in Rojo

Member Projects
DimeCasts.Net - Derik Whittaker

Friends of
Red-Gate Tools For SQL and .NET


SmartInspect .NET Logging
NGEDIT: ViEmu and Codekana
NHibernate Profiler
Balsamiq Mockups
JetBrains - ReSharper
Web Sequence Diagrams
Ducksboard<-- NEW Friend!


Site Copyright © 2007 CodeBetter.Com
Content Copyright Individual Bloggers


Community Server (Commercial Edition)