Are additional layers of abstraction warranted?

I was presented with the following discussion opener concerning the management of complexity via the introduction of abstraction...

How do we handle complexity? I asked a few software architects I know and all of them answered, “Abstraction.” Basically they're right, but being a math major in college, there is a principal I believe that software architects and designers miss - complexity is constant. In other words, if you’re designing a system that is inherently complex, you can’t reduce it. In chemistry I remember learning that energy is constant, it can’t be created or destroyed, it's just there. In software, if the solution to a problem is complex, the complexity is always going to be there.
What about abstraction? Abstraction basically hides complexity. This is good, right? The problem is, in a lot of designs, once abstraction hides complexity the designers tend to forget about it. If you work hard at a good design to hide the complexity and forget about it, it will come back and haunt you some day. So what does one do?

I love this discussion! The heart of this is determining if the introduction of abstractions, for the sake of hiding complexity, is truly an overall benefit. In robotics, we're ever dealing with increasing levels of abstraction for this very reason. For example, in Architectural Paradigms of Robotic Control, I briefly discussed the 3T architecture which has three separate layers, implemented as increasing levels of abstraction. E.g., the skill/servo layer would likely be implemented in C++, the sequencing/execution layer might be implemented as a sequence modeling language, such as ESL or NDDL, while the planning/deliberative layer might be implemented with a higher abstraction yet, such as with the Planning Domain Definition Language or Ontology with Polymorphic Types (Opt).

In response to the concerns put forth, I would tend to agree that abstraction hides complexity and that it does make it more likely that you may be bitten by the hiding of the complexity. With that said, encapsulation of complexity into well formed abstractions is an inevitable step to facilitate taking on increasingly complex problems. For example, in the .NET world of data access, Fluent NHibernate is really just a way of hiding the complexities of NHibernate. NHibernate is really just a way of hiding the complexities of ADO.NET. ADO.NET is really just a way of hiding the complexities of communicating to a database via TCP/IP sockets, or whatever underlying mechanism is employed. Along the same vein, tools such as Herbal, NDDL, and ESL are similarly provided as a means to provide an abstraction for hiding complexity.

Because these layers of complexity have been encapsulated in a manageable fashion, we're now able to take on project work which would be far too complex to manage if we were using a lower level of implementation, e.g., pure C++, or Assembly for that matter. Indeed, there will be times when the added layers of abstraction will make it more difficult to tweak a low level capability, but the improved complexity management that the abstractions provide should far outweigh the sacrifice of losing some low level capabilities.

I think the crux for determining if an abstraction is worthwhile:

  • Does the abstraction reduce complexity of interacting with the underlying layer that it encapsulates?
  • Does the abstraction make it easier to tackle increasingly complex problems?
  • Does the abstraction provide enough tweak points to accommodate the 5-10% of times that more low level control is needed?
  • Does the abstraction increase maintainability and ease of understanding of the overall goals of the system?

If the answers to the above are yes, then I believe that the encapsulation of complexity, codified as a new layer of abstraction, is pulling its weight. Otherwise, you might not want to throw that Assembly language reference away just yet.

Billy McCafferty


Posted 08-18-2010 4:27 PM by Billy McCafferty

[Advertisement]

Comments

Sean Kearon wrote re: Are additional layers of abstraction warranted?
on 08-19-2010 6:56 AM

I like the idea of the problem complexity being constant. When I started programming the guy I worked for used to say that the job of the developer was to manage the complexity (of the solution).  I still see a lot of what I do as managing complexity, that is making my job of producing a solution easier.

Tom wrote re: Are additional layers of abstraction warranted?
on 08-19-2010 9:52 AM

The leader about constant complexity is a complete red herring.  Of course abstraction doesn't magically make the complexity disappear; anyone who thinks that is deluded.  But anyone who thinks abstraction doesn't help manage that complexity is equally deluded.  And if they don't believe me, they can delete all their compilers and assemblers, virtual machines and garbage collectors, device drivers and operating systems and can go back to programming CPU instructions directly in binary on punched tape.

Erhun wrote re: Are additional layers of abstraction warranted?
on 08-19-2010 9:56 AM

This discussion is useful for understanding the abstraction to learn or set up an algorithm.  Also I like the idea that "once abstraction hides complexity the designers tend to forget about it. If you work hard at a good design to hide the complexity and forget about it, it will come back and haunt you some daydevlicio.us/.../item-clock.png."  

Paul wrote re: Are additional layers of abstraction warranted?
on 08-19-2010 10:07 AM

If complexity is constant and taking your example, the langauge in which the TCP/IP stack is coded is just abstracting doing everything in native machine code.  That would mean the complexity of ADO.NET is the sum of every thing it is based upon broken down into hand coding zeros and ones.  Whilst sometimes that feels right, it can't be true.   Likewise, what about a simple problem that is made complex by it's solution - where did that extra complexity come from.  If it's constant then a simple problem should stay contant regardless of the solution used to solve it.

technogeist wrote re: Are additional layers of abstraction warranted?
on 08-19-2010 10:43 AM

High level languages also provide a mental firewall from the complexity of the machine level(s) lurking below. Although it's quite possible to write apps in pure assembler. It's just too ridiculous to contemplate managing anything complex at that level.

Abstraction is great way to put complexity into a black box, and if done correctly it won't manifest itself outside of the box.

Mark wrote re: Are additional layers of abstraction warranted?
on 08-19-2010 10:47 AM

It's very true that once a team gets comfortable with a framework, they forget about the underlying infrastructure, and are at a loss when it breaks. That's where abstraction comes back to haunt you. If you support the systems you develop, you have to remember everything that can go wrong. That is, everything. Fortunately, many frameworks are pretty good...but the end product is only as good as its weakest link.

Artie Gold wrote re: Are additional layers of abstraction warranted?
on 08-19-2010 11:48 AM

The complexity is inherent; all you can do is confine it via abstraction. The art involved is in both maintaining correctness and avoiding complication (here defined as the ratio between the complexity of the solution and the complexity of the problem being solved).

Tony wrote re: Are additional layers of abstraction warranted?
on 08-19-2010 1:11 PM

I agree with this. I have been saying that things can be complex but keep them from becoming complicated (which I more or less started to use after reading: codebetter.com/.../complex-vs-complicated.aspx ).

Sean Kearon wrote re: Are additional layers of abstraction warranted?
on 08-20-2010 3:21 AM

I think it's important to remember here that we have little control over the complexity of the problem but have greater control over the complexity of the solution.  (Okay, you can do a lot to reduce the problem complexity, but let's say it has been reduced to its limit!).

An important part of designing good software is to minimise the internal complexity of our solution.  It's equivalent to a Mathematician seeking an elegant proof as opposed to one that is does the job but is less well structured.

I see the point here being that all abstractions we introduce should simplify the solution.  If they don't then they are not warranted for the given problem.

Harald M. wrote re: Are additional layers of abstraction warranted?
on 08-20-2010 6:44 AM

While your questions are correct, they fall very short of giving a guideline how to answer them. In my experience (as an architect who writes frameworks for 40+ people), abstractions are expensive: Understanding a wholly new concept takes on the order of one year; understanding changes to concepts takes on the order of months - both if you are constantly working with them. To maintain an abstraction (=keep it understandable and usable in an ever-changing world) is work - mostly explanations/teaching in person or via text/diagrams, more rarely implementation work.

So after an initial rush to build more and more "big" abstractions, I (and we) now hold back and let smaller, incompatible and contradicting abstractions blossom in parts of our applications.

The effect is

* less work on the whole

* catastrophic "abstraction clashes" at some times:

** Try to use NHibernate and EF in one application ...

** Same wih user continuations and a navigation concept like old MS GUI application block

etc.

Ben Ashley wrote re: Are additional layers of abstraction warranted?
on 08-20-2010 8:14 AM

I believe abstraction is good... up to a point.  I recently started working at a web-development firm, and inheriting a lot of C# web-projects that relied heavily on abstraction layers.  It's a mess.  Hundreds of objects that wrap some Stored-proc calls, that are managed by many managers, which themselves are abstracted away in another freakin' assembly.  It appears this philosophy is based on some of Microsoft's own Tech articles.

It is unmanageable, cumbersome, and unintuitive.

My proposal for a complete re-write of the next version has been accepted readily.

Ricker Silva wrote re: Are additional layers of abstraction warranted?
on 08-20-2010 10:53 AM

Very interesting point of view. I think complexity of course is constant, even in we have shrunk it to its minimum. But I disagree in abstraction hiding complexity. I see it this way:

Abstraction layers aim to reduce some variables that within a specific reference frame can be dismissed, allowing us to find solutions to specific parts of the problem, which combined with solutions we found for other parts of the problem can grow at least a first version of the whole solution we were looking for.

It's just like maths. Newton said that objects same mass fall at the same speed in vacuum. Look, in vacuum. being IN VACUUM is an abstraction, not hiding complexity, air is there, but allowing you to find a solution that under some conditions is perfectly suitable to solve the problems, in this case, for almost all kinematics.

So I think abstractions are just a part of a method to find easily manageable solutions rather than hiding complexity. The hiding complexity issue is more due to lack of knowledge of the reference frame your in, and that's just another problem we need to address, may be, with another abstraction.

Steve wrote re: Are additional layers of abstraction warranted?
on 08-20-2010 11:05 AM

Managing complexity and hiding complexity are two different things. I've had similar discussions a lot of times, and a lot of developers hate the idea that e.g. by writing lots of small classes/functions instead of a few larger ones they aren't eliminating complexity, only moving it into the call graph. Still true, though.

Even so, there are useful principles to follow to help ensure that small changes in requirements don't require large changes in the design - and often that is what developers mean by "reducing complexity". The complexity we're interested in is the complexity we have to cope with - not the complexity that's done and dusted and hidden in the background.

Roy wrote re: Are additional layers of abstraction warranted?
on 08-20-2010 2:37 PM

Since I was the one that made the statement to Billy that started this discussion, I think I’ll chime back in.  I’m moving a little away from Billy’s initial question to talk about the issue of complexity.  The first thing we must understand is that not all complexity is equal, as architects and systems designers the complexity on which we seem to focus on is probably the least complex complexity of all." Complexity has to be the biggest challenge for software development today. We tend to focus on local complexity, methods, classes, and other constraints while we ignore the more serious system-wide complexities like interdependency.  I’ve been writing software for almost 40 years. One thing I learned is, all application has some degree of complexity.  Abstraction helps but what does abstraction really do? I think all it does is to provide a vision of the complexity that others can grasp. Said a different way, abstraction provides the perception of simplicity or something that someone can understand, but the complexity has not been reduced or as I said it’s constant.  

We tend to focus on abstraction and other process to “hide” complexity and this is a good thing, or so it seems.  To me the problem is one of understanding why the complexity?  

DrewL wrote re: Are additional layers of abstraction warranted?
on 08-20-2010 8:09 PM

The lead-in made the assumption that the only way to deal with complexity is abstraction.  The first thing that comes to my mind when dealing with complexity is to break it into less complex pieces.

When I think about abstraction I think about creating an interface layer to perform often used functions.  Another use for abstraction is to encapsulate lower level logic.  For example, I have objects in a data model.  I don't care if the data values came from a SQL database, file or web service.

I often see people use abstraction in shared libraries.  Sometimes this is for legitimate busines rules often it to force an individuals rules upon other programmers.   The attitude is "so no one f***s with my code."  These abstraction layers tend to have a lot of private members.  So there is no way to modify or extend the functionality.  

If you are creating an abstraction layer to help make life easier then good.  If you use an abstraction layer to lock down your code then bad.  In any case it is not a software architect's way to deal with complexity of a system.

Tim Yen wrote re: Are additional layers of abstraction warranted?
on 08-24-2010 7:03 PM

I think the correct way to decrease complexity is encapsulation, which is a form of abstraction.

Break parts down into smaller parts, but dont necessarily add layers or inheritance heirarchies.

Dan Smith wrote re: Are additional layers of abstraction warranted?
on 08-25-2010 9:50 AM

I think the "abstraction hides complexity" argument frame is the wrong way to think about the issue.  Hiding is a binary concept, either something is hidden or it is not hidden.  Instead, abstraction organizes complexity and allows for control of that complexity.  With that frame we can then judge how well the abstraction organizes and controls the complexity instead of being concerned if the complexity is hidden or not.  This is what underlies your final summary of judging if an abstraction is worthwhile or not.

Yes, you can say that FluentNHibernate hides the complexity of NHibernate mapping, but what it really does is organize the complexity in a more useful and productive format while allowing good control over the inherent complexity required in an advanced ORM.

Alex wrote re: Are additional layers of abstraction warranted?
on 10-23-2010 12:28 AM

doors.txt;15

Alex wrote re: Are additional layers of abstraction warranted?
on 10-23-2010 8:12 PM

doors.txt;15

Alex wrote re: Are additional layers of abstraction warranted?
on 10-24-2010 11:31 AM

doors.txt;15

Alex wrote re: Are additional layers of abstraction warranted?
on 10-25-2010 6:04 AM

doors.txt;15

aa wrote re: Are additional layers of abstraction warranted?
on 11-01-2010 2:34 AM

doors.txt;10;15

aa wrote re: Are additional layers of abstraction warranted?
on 11-01-2010 3:14 PM

doors.txt;10;15

aa wrote re: Are additional layers of abstraction warranted?
on 11-01-2010 5:28 PM

doors.txt;10;15

aa wrote re: Are additional layers of abstraction warranted?
on 11-02-2010 5:55 AM

doors.txt;10;15

aa wrote re: Are additional layers of abstraction warranted?
on 11-02-2010 5:55 AM

doors.txt;10;15

About The CodeBetter.Com Blog Network
CodeBetter.Com FAQ

Our Mission

Advertisers should contact Brendan

Subscribe
Google Reader or Homepage

del.icio.us CodeBetter.com Latest Items
Add to My Yahoo!
Subscribe with Bloglines
Subscribe in NewsGator Online
Subscribe with myFeedster
Add to My AOL
Furl CodeBetter.com Latest Items
Subscribe in Rojo

Member Projects
DimeCasts.Net - Derik Whittaker

Friends of Devlicio.us
Red-Gate Tools For SQL and .NET

NDepend

SlickEdit
 
SmartInspect .NET Logging
NGEDIT: ViEmu and Codekana
LiteAccounting.Com
DevExpress
Fixx
NHibernate Profiler
Unfuddle
Balsamiq Mockups
Scrumy
JetBrains - ReSharper
Umbraco
NServiceBus
RavenDb
Web Sequence Diagrams
Ducksboard<-- NEW Friend!

 



Site Copyright © 2007 CodeBetter.Com
Content Copyright Individual Bloggers

 

Community Server (Commercial Edition)