Billy McCafferty : Patterns

Checklist for Developing Message-Based Systems

Billy McCafferty — Tue, 23 Mar 2010 21:22:00 GMT

Even when developing the most basic CRUD application, we ask ourselves a number of questions - whether we realize it or not - during the initial phases of development concerning the architecture and construction of the project. Where will the data be persisted? What mechanism will be used to communicate with the database? How will data from the database be transformed into business objects? Should separated interfaces and dependency injection be employed to maintain a clean separation of concerns between application logic and data access objects? What UI components will be leveraged to speed development of the UI layer? When developing message-based systems (see Message-Based Systems for Maintainable, Asynchronous Development for an introduction) it's immensely helpful to formalize such questions about how the systems will be designed and developed. Creating a checklist of such items to decide upon, and formalizing answers for the application context, helps to:

Standardize how components (applications participating in the message-based system) will communicate with the messaging middleware;
Provide architectural guidance throughout project development concerning how components will leverage message end-points;
Define the data model that will be used for sending commands and exchanging data among components;
Assist new developers on the team with getting up to speed with maintaining and extending the message-based system; and
Support consistency (and predictable maintainability) amongst the components of the system.

This post provides an addendum, if you will (I'm sure you will), to Message-Based Systems for Maintainable, Asynchronous Development for complementing the article with a checklist of questions and architectural topics that should be discussed, decided upon, and/or spiked before beginning development of your budding message-based system. Accordingly, this does not describe a methodology for developing message-based systems, but simply a checklist of topics which should be taken into consideration before development begins. Many of the checklist items below serve as good starting points for team discussion and for the development of architectural spikes for demonstrating implementation details.

Messaging Middleware

What messaging middleware solution will form the background of the message-based system?
How are channels addressed in the selected middleware? (E.g., as a string or as a port number?)
How does the nomenclature of the middleware match up to standard elements of message-based systems and design patterns? E.g., in Robot Operating System (ROS), a channel is called a "topic." Likewise, is their middleware specific nomenclature for elements such as component, publisher, subscriber, point-to-point channel, publish/subscribe channel, message, command message, request-reply message, router, message end-point, and other messaging design patterns?
What mechanism is available for peeking at messages going over a point-to-point channel?
What utilities are available for viewing active channels, publishers, and subscribers?
What other tools does the middleware provide for debugging and observing the messaging system? Will custom utilities need to be developed to autment development and debugging capabilities?

Integration with Messaging Middleware

How will message end-points publish a messages?
How will message end-points subscribe to a channel for messages?
How will message end-points poll a channel?
What technique will be used to decouple the component (participating application) from the message end-point when sending a message? E.g., dependency injection, separated interface, or other?
What technique will be used to decouple the component from the message end-point when receiving a message or polling a channel? E.g., callback, separated interface, or other?
Should exceptions raised by the messaging middleware be wrapped in component specific exceptions for looser coupling? If so, what will be the standard exceptions? Additionally, how will such exceptions be logged and tracked for debugging; e.g., email alerts, logging, exception message channel?

Channels and Routing

What channels will be created and what data type should each channel carry?
Will a hierarchical naming strategy be leveraged for naming and organizing channels? If so, what are the naming conventions for the project?
Has an invalid message channel been setup with a logger to capture and record invalid messages?
How should a predictive router be implemented, when needed?
How should a reactive router (e.g., a message filter) be implemented, when needed?

Message Elements

What data elements may the header of a message contain and how should each be specified? E.g., return address, message expiration, timestamp, origin, format indicator, etc.
How should a message identifier and correlation identifier be generated and included in the header?
What is the messaging data model supported and/or enforced by the messaging middleware? E.g., ROS supports http://www.ros.org/wiki/msg.
What canonical data model will be used to imbue message content with domain specific semantic meaning? E.g., JAUS.
How will message translators and/or message mappers be leveraged for converting between each component's domain model and the canonical data model?
How will a command, document, event and request-reply message be composed?

Other Topics

How will components maintain information that is associated with a correlation identifier?
Are there existing components which need to interact with the messaging middleware which cannot be modified? If so, how will each communicate with the messaging middleware?
How will each component participate with the message based system; e.g., as a polling consumer, as an event-driven consumer, as strictly an event publisher, as a combination of these?
Should exceptions raised by a component participate in a global exception tracking and debugging mechanism?

As stated previously, this checklist is not intended to provide a methodology for developing message-based systems, but should serve as a good basis to make sure "T's are crossed and I's are dotted" when deciding upon the major architectural aspects and development techniques that will be leveraged throughout the development of your (team's) message-based system.

Enjoy!
Billy McCafferty

Message-Based Systems for Maintainable, Asynchronous Development

Billy McCafferty — Tue, 02 Mar 2010 04:22:00 GMT

Preface (you know it’s good if there's a preface)

In Architectural Paradigms of Robotic Control, a number of architectures were reviewed including deliberative, reactive, and hybrid architectures. Each of these exhibit a clean separation of concerns with layering and encapsulation of defined behaviors. When implemented, the various capabilities, such as planners and mobility controllers, are encapsulated into discrete components for better reusability and maintainability. A pivotal aspect not discussed in the previous article is how the various system layers and components communicate with each other, such as reporting sensor feedback and sending commands to actuator controllers. Effectively resolving this communication challenge is not only important to robotic systems but to many other industries and domains for the successful integration of disparate applications.

To give credit where credit is due, this article pulls quite heavily from the patterns, taxonomy, and best practices presented in Enterprise Integration Patterns (Hohpe, 2003). This well organized book is chock full of hard learned lessons and solid guidelines for developing maintainable message-based systems. This article should not be seen as an adequate replacement for that book (it’s more like cliff notes with a spackling of robotics bias); indeed, Enterprise Integration Patterns should have a prominent place on your bookshelf if you're developing message-based systems – so read this post and then browse http://www.eaipatterns.com/ while waiting for your copy to arrive to delve deeper.

A Need for Message-Based Systems

A few industries in particular, such as finance, healthcare and robotics, are demanding integration of an intimidating number of separate technologies that may be spread across computers, networks, and/or built upon a variety of technological platforms. Not only is this integration tricky, it can come with a significant cost to performance and maintainability if not implemented correctly. Accordingly, a solution is needed which facilitates loosely coupled integration while accommodating the performance demands of the task at hand. Taking a message-oriented approach to inter-application communications is one such way to accommodate these demands in a maintainable manner without sacrificing performance. This article gives an introduction to developing message-based systems using messaging middleware, describes taxonomy for discussing messaging topics and patterns, and includes a number of best practices.

Before delving further, it's important to clarify a few terms that will be used frequently:

Messaging Middleware (aka, a message bus): a 3rd party application which provides messaging infrastructure and capabilities (e.g., MSMQ, MS Concurrency and Coordination Runtime (CCR), Robot Operating System (ROS)),
Component: a stand-alone application or piece of executable code which communicates with the messaging middleware,
Message-based system: the entirety of the system including all integrated components and the messaging middleware.

As stated, messaging provides one means of facilitating inter-component communications. But as with any design approach, the project requirements must be carefully considered to determine if messaging is the appropriate mechanism for integration. While messaging is robust and facilitates integration, it also adds complexity and indirection. So before deciding to use messaging as the means of integration, consider all component integration options including (Hohpe, 2003):

File Transfer: wherein a component produces files of shared data which other components consume,
Shared Database: each component stores and retrieves data from a common database,
Remote Procedure Invocation: each component exposes specific procedures to be invoked remotely for exposing behavior and exchanging data,
Messaging: each component connects to a common messaging system, using messages to invoke behavior and exchange data.

Determining which integration approach is most suitable to your project's needs is beyond the scope of this article, focusing instead specifically on messaging. In turn, we'll review important elements of developing a message-based system including: message channels, messages, message routers, and message endpoints.

Message Channels

When a component sends information to another component in a message-based system, it adds the information to a message channel. The receiving component then retrieves the information from the message channel. Different channels are created for each kind of data to be carried; having a separate channel for each datatype better enables receiving components to know what kind of data will be retrieved from a given channel. For using a channel, each channel is addressable for sending and retrieving messages to/from them. How a channel is addressed varies depending on the messaging middleware being leveraged, but it's usually a port number or a unique string identifier. As a good practice for keeping channels organized, if string identifiers are available, a hierarchical naming convention may be employed to label channels by type and name; e.g., a channel carrying laser scans might be called "Perception/LaserScans."

There are two basic kinds of message channels:

Point-to-Point Channels (aka - client/server style): routes messages for components to talk directly with other components; e.g., a remote procedure call to another component. A message over a point-to-point channel only has a single receiver; so while the sender may not necessarily know who the receiver is, the sender can rest assured that the message will only be received by one receiver – it’s a FIFO queue.
Publish-Subscribe Channels (aka - broadcast): routes messages for components to publish data and an arbitrary number of components to subscribe to that data. A copy of the message is generated for each subscriber on the channel.

In addition to channels intended to carry information among components, it is a good practice to setup an invalid message channel that bad-formed or unreadable messages may be forwarded to for logging and to assist with debugging.

Messages

With a function call, a simple parameter or object reference may be passed and retrieved by the invoked method, sharing the same memory space. But when passing data between two processes with separate memory spaces, the data must be packaged into a "message" adhering to an agreed upon format which the receiver will be able to disassemble and understand. The sender of the message passes the message via a message channel. The receiver retrieves the message from the message channel and transforms the message into internal data structures appropriate for the task at hand.

A message is made up of two parts:

Header: describes the data being transmitted and details concerning the message itself; e.g., origin, timestamp information, message expiration (if content is time-sensitive), message identifier, correlation identifier, return address, etc., and
Body: the data content that the receiver is looking to use.

When sending a message, the sender intends for the message to be used, or responded to, in a particular way. The intention of the message may be described as being one of the following:

Command Message: invokes a procedure in another application,
Document Message: passes a set of data to another application,
Event Message: notifies another application of a change in state, and
Request-Reply: requests a reply from another application.

Event messages deserve a bit more discussion. In its simplest form, an event message would simply be informational, letting subscribers know that an event has occurred; e.g., a new laser scan is available. If subscribers would like details concerning the event, they would send a request-reply to the sender of the event to provide further details; e.g., the laser scan details. Alternatively, the event could be a document as an event to inform subscribers that an event has occurred along with the details of that event; e.g., a new laser scan is available with laser scan details included. The size of the event details and the rapidity in which the event occurs should be considered when deciding between publishing simple event messages and document messages as events.

Request-reply messages could also use a bit more describing. A request-reply is usually implemented as two point-to-point channels. The first channel delivers the request as a command message, while the second carries the reply back to the requestor as a document message. To keep the replying component more loosely coupled and reusable, the requestor should include a return address indicating the channel that the replier should use to publish the reply. After receiving the reply, a challenge for the requestor is to then correlate the reply to the original request. If the requestor is sending a number of requests in succession, it will likely be difficult to keep clear – if it matters – which request a reply is associated with. To resolve this, every request may include a unique message identifier that the replier would then include as a correlation identifier. (A message could have both a message Id and a correlation Id.) The requestor uses the correlation identifier to “jog its memory” concerning which request the response is for. But frequently, a request-reply is in context of a particular domain object, such as a terrain map or a bank transaction; but the correlation Id doesn’t include such information. To assist, the requestor can maintain a mapping (e.g., hashtable) between message Ids and relevant domain object Ids which are related to the original request. When the reply is received, the mapping may be used to load the appropriate domain objects and take further action, accordingly.

Obviously, it is important that the senders and receivers of a message system agree upon the format that messages will take for clear interoperability, better reusability of components, and extensibility of the system. Consequently, a canonical data model should be well defined that all applications will adhere to. The canonical data model does not dictate how each application's domain model must be structured, only how each application must format data within messages. Message translators are developed to convert the sending application's domain model into the canonical data model before sending a message; receivers of messages then use their own message translators to translate the message into their own domain. This mechanism allows applications built on completely different technologies (e.g., C#, Lisp, and C++) to communicate with each other and exchange data. Many off the shelf messaging systems define their canonical data model which must be adhered to. For example, the Robot Operating System (ROS), which we'll looking at in more detail in subsequent posts, defines their canonical model at http://www.ros.org/wiki/msg. But the canonical data model need not be limited to defining the types of primitives available and how to include them in messages.

Domain-specific canonical data models may augment message formatting rules, adding semantic meaning to the data within a message. For example, the Joint Architecture For Unmanned Systems (JAUS) is a set of message guidelines for the domain of unmanned systems, such as autonomous vehicles. The JAUS guidelines provide domain specific rules for communicating data, such as propulsion and braking commands, sensor events, pose and location information, etc. To demonstrate, JAUS message types include (Siciliano, 2008):

Command: initiate mode changes or actions,
Query: used to solicit information from a component,
Inform: response to a query,
Event set up: passes parameters to set up an event, and
Event notification: sent when the event happens.

A challenge in dealing with canonical data models is how to handle changes to the model. In order to support backwards compatibility of existing components when the canonical data model changes, new message channels could be created to carry the messages adhering to the model; e.g., "Perception/LaserScans_V1" and "Perception/LaserScans_V2." Alternatively, the existing channels could continue to be leveraged to carry messages adhering to different version of the canonical data model. To do so, the message, within its header, would include a format indicator, such as a version number or format document (e.g., DTD) reference. But if a sender knows that receivers of a particular message are mixed in what format is being used, a component would need to send two messages, one for each version of the canonical data model. Certainly, this is an important consideration when deciding which components should (or even can) be upgraded to newer formats, and in what order.

Message Routers

While the heavy lifting of the message routing is handled by the messaging middleware itself, there are times when it is useful to augment the middleware with custom message routers to support unique scenarios.

Suppose the destination of a message may change based on the number of messages that have already passed over a channel. In this scenario, the sender of a message may not know how many messages have been passed over a channel since other senders may have been publishing messages on the same channel. Consequently, a message router may subscribe to the channel to determine where each message should be forwarded to, based on the described business rules. Once the destination is determined, the router would then place the message on a subsequent channel to be delivered to the appropriate destination. This intermediary routing is described as predictive routing as the message router is aware of every possible destination and the rules for routing, accordingly. If the routing is based on content within the message itself, such as threshold values, then the custom router is known as a content-based router.

A drawback to using message routers is that if the routing rules change frequently, the message router will need to be modified just as often. To help remedy this, if the rules are expected to change frequently, configurable routing rules (e.g., via XML) could be employed to enable easier management of routing rules.

Let's now consider another scenario wherein it's left up to the subscribers to determine which messages they're interested in; i.e., subscribers will be responsible for filtering out the messages they're uninterested in. In this reactive routing scenario, each subscriber would provide a respective message filter which is similar to a message router, but simply forwards, or does not forward, a message onto a subsequent channel that the destination subscriber is listening to. Frequently, message filters decide to forward, or not forward, based on content in the message itself; e.g., only forwarding orders that have a coupon included. While being similar to a message router in basic functionality, a message filter only has one possible channel to forward the message onto.

Deciding between predictive and reactive filtering must take into account a number of considerations. Is the message content sensitive? Do you need to minimize network traffic? Do you need to be able to add and remove subscribers easily? Is the predictive router becoming a bottleneck of message dissemination? For further guidance on selecting among routing options, see (Hohpe, 2003), ppg. 241-242.

Message Endpoints

Each messaging middleware option (e.g., CCR, ROS) has unique requirements for communicating with it. Each has its own API, its own means of addressing channels, and its own rules for packaging messages. Ideally, the components of the system should not be aware of the specifics of communicating with the messaging middleware. Furthermore, while unlikely to occur, the middleware should be able to be replaced with another, requiring little, if any, changes to the components. Accordingly, the components must be loosely coupled to the messaging middleware. Message endpoints provide the bridge between the domain of each component and the API of the messaging middleware.

Message endpoints are similar in nature to repositories when communicating with a database. Repositories encapsulate the code required to communicate with a database to store and retrieve data while being able to convert information from the database into domain objects. If the database changes, or if the mechanism for database communication changes (e.g., ADO.NET to NHibernate), then, ideally, only the repositories are affected. The rest of the application knows little about database communications outside of the repository interfaces. Likewise, components of a message-based system should not be aware of messaging details outside of the message endpoint interfaces, which provide the means to send and receive data. The message endpoint accepts a command or data, converts it into a message, and publishes it onto the correct channel. Additionally, the message endpoint receives messages from a channel, converts the content into the domain of the component, and passes the domain objects to the component for further action. Internally, the message endpoint implements a message mapper to convert between the component domain objects and the canonical data model.

While posting to a channel is rather straight forward, a message endpoint may receive a message by acting as a:

Polling Consumer: wherein the receiver looks at a channel on a regular basis for new messages and/or as soon as it completes the processing of a previous message. This frees the receiver from having to deal with messages as soon as they arrive on the channel in favor of dealing with messages went it’s ready and willing. A consideration to keep in mind is that messages may queue up on a channel while waiting to be retrieved by the polling consumer. Additionally, a polling consumer may take up threads and resources while polling a channel, even if the channel is empty.
Event-Driven Consumer: wherein the message is given to the receiver as soon as it arrives on the channel. The benefits to this include avoiding messages queuing up while being able to process messages asynchronously. But the receiver is no longer in control of the timing in which it processes messages and must handle messages as soon as they arrive on a channel.

Many messaging middleware solutions include support for transactions that the message endpoints may leverage, making the endpoints transactional clients. To illustrate the need for this, suppose the receiver of a request-reply command crashes just moments after the command message is consumed and removed from the channel. When it recovers, the command message is lost and the sender will never receive a reply. Using a transaction, the command message is not removed from the channel until the response is completed and sent. Committing the transaction removes the command message from the channel and adds the reply document message to the reply channel.

There are a few recommendations which should be considered when developing message endpoints. In accordance with the SRP, message endpoints should be able to receive messages or send messages, but not both in the same message endpoint. Furthermore, a message endpoint should only communicate with one message channel. If a component needs to send a message on separate channels, it would leverage multiple message endpoints to do so. If your components are developed in line with DDD, it would be each component's application services layer which would communicate with the message endpoints, preferably via their interfaces instead of concrete instances. This facilitates swapping out the message endpoints with mock objects for unit testing. Leveraging separated interfaces and dependency injection helps enable this approach. While these guidelines introduce more objects and indirection, they are proven practices for increasing maintainability of the component and reusability of the message endpoints.

Performance of Message Based Systems

One would likely be quick to assume that developing a message-based system has a huge cost to performance. Domain objects are converted to messages, messages are passed over channels, routers and filters intercept and forward messages, and messages are converted back into domain objects...this sounds like a heck of a lot going on. But because each component executes in its own thread or process, they need not wait for other components to complete their job before being able to move on to handling another message or task.

In the figure above (adapted from Hohpe, 2003, pg. 73), note that a sequential process requires that each message life cycle is completed in full before moving on to the next message. But in an asynchronous, message-based system, each component can move onto a subsequent message just as soon as it has completed its part in the last one. This greatly compensates for the extra overhead imparted by the messaging infrastructure.

With that said, there are some component responsibilities which take a long time to complete and may impede the speed by which messages are processed. For example, imagine a component which takes images from a web cam and extracts information such as human figures or road signs. This is likely a time consuming process and would impact the turn around time in which the component could process subsequent messages. To alleviate this bottleneck, multiple instances of the same component may subscribe to the same point-to-point channel. Recall that a point-to-point channel ensures that each message only has a single receiver. The instances of the component then become competing consumers of each message. So if one instance of the component is still processing an earlier message, another instance can grab the next message that arrives for concurrent processing. Power in numbers!

Monitoring and Debugging

Due to the widely asynchronous nature of message-based systems, attention must be given to monitoring and debugging techniques to observe system behavior and iron out problems.

Logging message content is an invaluable measure towards getting a clear look at what communications are taking place. To facilitate this, logging could be added directly into sending and receiving components, but logging message content should not be a concern of components; accordingly, using a monitoring utility to log such information is a cleaner separation of concerns. Certainly a benefit of publish-subscribe channels is that a monitoring component may subscribe to all messages and log the information to a file or console window. While it’s just as valuable to monitor messages on point-to-point channels, if a monitor were to consume a message over a point-to-point channel, the message would be noted as consumed and would no longer be available to the intended receiver. Because such monitoring capability is so helpful in developing and debugging, many messaging middleware options include a "peek" option which allows a monitoring utility to review message contents on a point-to-point channel without actually consuming the message. This capability should be taken into consideration when comparing messaging middleware alternatives.

In addition to monitoring message content sent over channels, it’s also assistive to monitor active subscriptions to various channels to accurately determine which components are receiving which messages. This capability is typically built into messaging middleware solutions and is very assistive during development.

It’s possible that even if a message is routed appropriately, the receiving component may not know what to do with the message due to an incorrectly formatted header or body content. Invalid messages such as this should be forwarded to an invalid message channel which an error logging utility would monitor and log, accordingly. An invalid message channel is setup just like any other channel but with the intention of exposing such messages for debugging purposes.

Closing Thoughts

Developing asynchronous, message-based systems requires a paradigm shift from the more traditional, synchronously executed applications that most people are familiar with. It is not appropriate for every application domain and should be seen as one additional architectural option to consider when developing applications. But in some domains, such as robotics or in the integration of disparate applications, this approach to development is absolutely pivotal in providing responsive behavior without sacrificing maintainability of the overall system. Indeed, by splitting responsibilities into discrete components, loosely coupled to each other via messaging middleware, immensely complex problems can be broken down into understandable chunks while being flexible enough to accommodate changes to the underlying middleware or introduction of new components.

In the next couple of posts, we'll look at a checklist for developing message-based systems followed with examples in CCR and ROS.

Enjoy!

Billy McCafferty
http://wiki.sharparchitecture.net

P.S., for S#arp fans, we're trying to get another release together for the coming week...and this ain't no quarterly release. ;)

References

Hohpe, G., Woolf, B. 2003. Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions.

Siciliano, B., Khatib, O. 2008. Springer Handbook of Robotics.

Dependency Injection 101

Billy McCafferty — Mon, 09 Nov 2009 22:26:00 GMT

(It looks like Kevin Pang and I must have seen the same vision this weekend or had a close encounter flashback as he's concurrently putting together a great series on Dependency Injection for Dummies...be sure to check out his posts for another take on this subject!)

Simply put, dependency injection is the technique of supplying an object with its service dependency(s) from an external source.

If you’re unfamiliar with dependency injection or just starting down the road of object oriented programming, the above statement might have the same effect as a mathematics book when the author puts forth a new equation that is about to be analyzed. Symptoms may include glassy eyes, nausea, and headaches. But when taken apart, we find that the idea of dependency injection is a much simpler concept than its grandiose title implies, and is a technique that many developers use commonly, quite naturally in many cases, without even knowing it.

Encapsulation

The place to start, in understanding the motivations and implementation of dependency injection, is with the basic idea of encapsulation. Crudely explained, encapsulation is the art of determining which bits of logic belong together and putting them into discrete objects. This allows the details of the logic to be modified within the encapsulated object without necessarily impacting the objects which use it. Furthermore, encapsulation provides a neat and clean way to package up reusable code to be used by many other objects without introducing code duplication.

From a real-world perspective, encapsulation is a very useful technique when building a self-aware robot to take over the world. When building any self-aware robot, you need to start with the basics, such as mobility and sensory perception. For example, a robot might have a stereo camera for observing the visible world, sonar sensors for detecting nearby objects around it, and touch sensors to know if it has made contact with the object it’s trying to pick up. Just by describing it, we’re already starting to imply areas of logic that could be nicely encapsulated: visual perception, sonar object detection, touch feedback, and mobility.

Assume for a moment that we’ll bypass the practice of encapsulation and throw all of the code for these various modules directly into the primary method which uses them, a benign method we’ll call TakeOverTheWorld, living in a .NET assembly called CommandAndControl.dll. The code might look something like the following:

public class WorldDomination {
   public void TakeOverTheWorld() {
      // Code to open USB connection to use stereo camera
      // Code to process camera images to identify humans

      // Code to open serial port to legs for moving
      // Code to send legs serial commands to move over to location of human

      // Code to generate speech to user; e.g., "All your leaders are belong to us"
   }
}

In the code, we see the TakeOverTheWorld method taking on a lot of responsibilities and given explicit knowledge about components which are subject to change when the robot components are upgraded. Some of the responsibilities which we’re burdening the method with include:

Knowing the stereo camera is connected via USB and how to open a connection to it;
Knowing how to process camera images to identify humans;
Knowing that the legs require communications to be sent via a specific serial port;
Knowing how to send low level communications to the legs as serial commands.

So what’s the problem you ask? For starters, the TakeOverTheWorld method would likely be a few thousand lines long and would be immensely difficult to maintain. Let’s suppose that we upgrade the camera to use a better stereo imaging device which now receives commands wirelessly. You’d have to carefully, tediously walk through all of the lines of TakeOverTheWorld to find any communications with the camera, upgrade the capabilities, and test, accordingly. But now testing is much more challenging as well. Since all of the other code, such as the mobility code, will also run whenever we invoke TakeOverTheWorld, then we’ll need to make sure that the rest of the robot is in perfect working order while we switch out one component. In other words, it’s simply not possible now to test the various components in isolation of each other. This approach quickly turns into spaghetti code, becoming a maintenance nightmare, difficult to debug, and exhibiting decreasing code quality as time progresses. What started out as a simple project to develop a robot to conquer the world is turning out to be about as effective as C-3PO…after being blasted by a storm trooper that is.

The first step that we can take, to make our robot for world domination a little easier to understand and maintain, is to encapsulate the various modules into their own, respective classes. Let’s look at the TakeOverTheWorld code again with a sprinkling of better encapsulation by moving the camera related responsibilities to a class called StereoCamera.cs and the mobility related responsibilities to MobilePlatform.cs. Furthermore, let’s move these two classes to a new class library assembly called Services.dll.

// Within the Services package (or class library)

public class StereoCamera {
   public void Connect() {
      // Code to open USB connection to use stereo camera
   }

   public MeaslyHuman IdentifyHumanInFieldOfVision() {
      // Code to process camera images to identify humans
   }
}

public class MobilePlatform {
   public void Connect() {
      // Code to open serial port to legs for moving
   }

   public void MoveTo(Location location) {
      // Code to send legs serial commands to move over to location 
   }
}

// Within the CommandAndControl package

public class WorldDomination {
   public void TakeOverTheWorld() {
      StereoCamera camera = new StereoCamera();
      camera.Open();
      MeaslyHuman human = camera.IdentifyHumanInFieldOfVision();

      MobilePlatform legs = new MobilePlatform();
      legs.Connect();
      legs.MoveTo(human.Location);

      // Code to generate speech to user; e.g., “All your leaders are belong to us”
   }
}

As demonstrated, we introduced two new classes, StereoCamera and MobilePlatform, and moved the related responsibilities to those classes. We also introduced a new package as a new class library, called Services.dll, to contain the two new classes. The package diagram now looks as the figure shown to the right.

This basic refactoring has resulted in a number of benefits:

There are far fewer lines of code in the TakeOverTheWorld method and easier to digest for whomever may be looking at it next;
The TakeOverTheWorld method is now much easier to understand the overall flow of control without getting bogged down in low level details;
The TakeOverTheWorld method now knows nothing about the details related to connecting to the camera via the USB connection (other than simply calling Connect), nothing about the mathematics behind processing camera images to identify humans, nor the details of opening a serial port and sending low level serial commands to move from point A to point B; and
The StereoCamera and MobilePlatform classes are now quite reusable from a variety of other objects that will surely need to use them.

Overall, this encapsulation refactoring has led to a design that is easier to maintain and will likely reflect in the general level of maintainability and quality. So things are better, but we still have much to improve. I suppose we’re beginning to resemble C-3PO, but only after Chewbacca put his head on backwards. (OK, so C-3PO didn’t have diabolical plans for world dominion, but let’s ignore that subtlety for now.) As we proceed, we’ll examine what other areas we’ll want to improve and learn a bit more about the definition of dependency injection along the way.

Services

Earlier, we defined dependency injection as the technique of supplying an object with its service dependency(s)… What exactly is a service dependency? A service, in the context of dependency injection, is any object which provides access to an external resource. In the refactoring exercise we performed on our code, and as implied by the package diagram, two services were introduced. The first provided access to the stereo camera; the second provided access to the legs for moving the robot around. In your own project, services may include web services, data access objects, file manipulation utilities, third party APIs, and many others.

Because services depend on external resources (e.g., a physical component or database), they are inherently more complicated to test, require that the external resource be available when we test our code, and are subject to change when the underlying resource being managed changes (e.g., when the component is upgraded). When considering these arguments, we want to remain loosely coupled to the services themselves. To contrast, the refactored code above, even after applying encapsulation, is tightly coupled to the underlying services.

To illustrate the idea of tight coupling, the TakeOverTheWorld method is creating each service directly with the new keyword; i.e., the method is creating a concrete instance of the service class. Not only does the method have a concrete dependency on the service class itself, this inherently implies that the application layer that the WorldDomination class resides in must have a direct dependency on the layer which the StereoCamera service class resides in. This layer dependency is shown as a directional line in the above package diagram from the CommandAndControl layer to the Services layer.

In addition to having a concrete dependency on the service itself, the TakeOverTheWorld method must also know to call Open for the respective services (and presumably Close); i.e., the TakeOverTheWorld method needs to know how to create and initialize the services before it’s even able to use them. These are further responsibilities which this method should not be concerned with.

Ideally, the TakeOverTheWorld method would be loosely coupled to the underlying services and wouldn’t have to know about such low level details of the service itself outside of calling methods such as IdentifyHumanInFieldOfVision.

Coding to Interface

The first step we can take towards making TakeOverTheWorld less tightly coupled to the service classes is to have the TakeOverTheWorld method code to the interfaces of the services rather than to the concrete classes themselves. One might argue against introducing two new interfaces for the objects; as we add new objects, we inherently introduce additional complexity to the overall system. Also, with interfaces, not only are we introducing a small amount of additional complexity to the overall system, but we’re also introducing indirection. In other words, in order for someone to fully understand what’s going on at runtime, the developer will need to determine which concrete object is being used under the interface. It shouldn’t be difficult to do so, but it does add time to understanding the overall system. These are fair arguments and ones not to be ignored. Whenever we are faced with a decision that will lead to a better design at the expense of added complexity, we carefully need to consider if the benefits of refactoring outweigh the penalties of added complexity. In this case, coding to interface (and another refactoring we will do next) will lead to a more maintainable system, provide a means to readily unit test the various components in isolation, and will result in a more loosely coupled design.

Accordingly, we simply need to add two interfaces to the Services package, have the service classes implement them, and change the references in the TakeOverTheWorld method to invoke the interfaces rather than the classes themselves. But as we’ll see, this next refactoring still leaves us tightly coupled to the service classes. Let’s look at the code.

// Within the Services package

public interface IStereoCamera {
   void Connect();
   MeaslyHuman IdentifyHumanInFieldOfVision();
}

public interface IMobilePlatform {
   void Connect();
   void MoveTo(Location location);
}

public class StereoCamera : IStereoCamera {
   public void Connect() {
      // Code to open USB connection to use stereo camera
   }

   public MeaslyHuman IdentifyHumanInFieldOfVision() {
      // Code to process camera images to identify humans
   }
}

public class MobilePlatform : IMobilePlatform {
   public void Connect() {
      // Code to open serial port to legs for moving
   }

   public void MoveTo(Location location) {
      // Code to send legs serial commands to move over to location 
   }
}

// Within the CommandAndControl package

public class WorldDomination {
   public WorldDomination() {
      camera = new StereoCamera();
      legs = new MobilePlatform();
   }

   public void TakeOverTheWorld() {
      camera.Open();
      MeaslyHuman human = camera.IdentifyHumanInFieldOfVision();

      legs.Connect();
      legs.MoveTo(human.Location);

      // Code to generate speech to user; e.g., “All your leaders are belong to us”
   }

   private readonly IStereoCamera camera;
   private readonly IMobilePlatform legs;
}

In the code above, note that the two interfaces were introduced and now the constructor of the WorldDomination class now handles creating the service instances and setting private members, represented as interfaces, to the service instances. The package diagram reflecting this refactoring is shown at right.

Even though the TakeOverTheWorld method is now communicating with the interfaces of the services, the class containing is still tightly coupled to the service classes due to the local instantiation of the classes within its constructor. To further alleviate this coupling, we’re going to have to move a step “up” and introduce a package which will be responsible for creating the service dependencies and injecting them into the WorldDomination class. Yes, we’re finally ready to see dependency injection in action.

Service Initialization and Injection

While the previous refactorings have taken us closer towards a more loosely coupled design, we still need to remove the object creation from the WorldDomination class altogether. By introducing a new package which will contain the robot system’s “main” program, we can leverage that layer to also initialize the service dependencies and inject them into the WorldDomination class. Let’s look at the code of the new package, called Robot, along with modifications to the WorldDomination class. (The Services package, and the interfaces and classes therein, will remain unchanged.) But before looking at the code, study the package diagram at right to see where the classes live and take particular note of the direction of package dependencies. There’s one dependency that’s still problematic, which we’ll discuss after the code.

// Within the CommandAndControl package

public class WorldDomination {
   public WorldDomination(IStereoCamera camera, IMobilePlatform legs) {
      this.camera = camera;
      this.legs = legs;
   }

   public void TakeOverTheWorld() {
      // Same as before
   }

   private readonly IStereoCamera camera;
   private readonly IMobilePlatform legs;
}

// Within the new Robot package

public class Program {
   // This is the “main” method of the entire robot application.
   // I.e., it’s the method that gets called to kick off the application.
   public void Main() {
      IStereoCamera camera = new StereoCamera();
      IMobilePlatform legs = new MobilePlatform();

      WorldDomination worldDomination = new WorldDomination(camera, legs);

      worldDomination.TakeOverTheWorld();
   }
}

In the code above, the Main method initializes the services and “injects” them into the constructor of the WorldDomination class. As you can see, the injection of service dependencies into a class is really nothing more than passing the service instances as arguments to the class instantiation. So when it comes right down to it, the intimidating phrase “dependency injection” is really quite simple in practice. But there’s still a problem in our implementation that could lead to design “rot” over time. The CommandAndControl package still has a dependency on the Services package. This is bad because during later phases of the project, without a high level of discipline, developers may forego injecting the service dependencies into the WorldDomination class, or other classes in the CommandAndControl package and simply instantiate the service classes directly using the new keyword. This would make all the work we’ve done to this point to support dependency injection a moot point. To enforce our design, we need to introduce a new pattern known as dependency inversion, also known as separated interface.

Separated Interface

To enforce our design, and to eliminate the package dependency from CommandAndControl to Services, we will move the service interfaces to the CommandAndControl package and reverse, or invert, the dependency direction between the two packages. Accordingly, the refactoring steps to do this include:

Copy the service interfaces from the Services package to the CommandAndControl package.
Change the interface references in the WorldDomination class to refer to the interfaces now found within the CommandAndControl package.
Remove the package dependency from CommandAndControl to Services. Now add a package dependency from Services to CommandAndControl. (In C#, this would be a matter of deleting the assembly reference from the CommandAndControl class library and adding a new assembly reference from the Services class library.)
Modify the service classes, StereoCamera and MobilePlatform, to implement the interfaces now found in the CommandAndControl package.
Modify the Main method, in the Main class to use the interfaces found within the CommandAndControl class.
Delete the obsolete interfaces from the Services package.

With the separated interface pattern implemented, the CommandAndControl package is only aware of the service interfaces without having any dependency, whatsoever, on the service implementations. As performed in the previous refactoring, the CommandAndControl package, the WorldDomination class specifically, is being given its service dependencies via constructor injection via initialization code found in the Robot’s main method.

With this, we have implemented a basic example, including all the required pieces, for performing dependency injection. It introduced a number of new pieces to the overall application, but our design is now loosely coupled to the underlying services and enables us easily swap out services with new implementations, without having to recompile the CommandAndControl package, and provides the ability for test doubles, such as mocks, stubs, and fakes, to be injected into the WorldDomination class when unit testing. (Unit testing with test doubles is beyond the scope of this article, but I encourage you to read http://xunitpatterns.com/Test%20Double.html to begin learning more about this subject.)

Dependency Injection / Inversion of Control (IoC) Containers

There’s one final topic which is appropriate to discuss when describing dependency injection: IoC containers. (Dependency injection is also known as “Inversion of Control,” as originally described by Robert Martin. Interestingly enough, the development community typically uses the phrase “dependency injection” when talking about the technique itself and “IoC” when talking about dependency injection containers. But rest assured, there is no difference whatsoever between “Dependency Injection” and “Inversion of Control (IoC).”) An IoC Container is a utility which is leveraged to initialize and inject dependencies into the objects that use them. While it is a bit more difficult to visualize than the “manual” dependency injection we performed in the previous refactoring step, an IoC Container is nothing more than an object factory which initializes an object and satisfies its service dependencies in the process.

With most IoC Containers (e.g., Ninject, Castle Windsor, LinFu, and many others), there are only a couple of steps required to setup and use the IoC Container:

Register all of the classes which may be injected into another object. In our carefree efforts for world domination via our robot, this would include, at minimum, registering the service classes MobilePlatform and StereoCamera. While each IoC Container has its own means of registering a class for later retrieval, the underlying mechanism is often as simple as adding the service class’ object type to a hashtable for later lookup. If we were using Castle Windsor with our project, the registration lines would look like the following:
```
container.AddComponent("camera", 
   typeof(IStereoCamera), typeof(StereoCamera));
container.AddComponent("mobilePlatform", 
   typeof(IMobilePlatform), typeof(MobilePlatform));
container.AddComponent("camera", typeof(WorldDomination));
```
Note that the last line registers the WorldDomination class itself. In doing so, we can let the IoC Container take care of the work of instantiating it with its service dependencies already injected. We don’t have to hand off instantiation of this class to the IoC Container, but it provides a useful benefit. Suppose the WorldDomination class needs a third service dependency provided via its constructor somewhere down the road. If we aren’t using an IoC Container to manage its creation, we’d have to look for every place where the class is created, via the new keyword, and pass in the extra dependency. If we leave its creation to the IoC Container, then we simply add the new service interface parameter to its constructor and the IoC Container will take care of the rest, assuming you’ve registered the third dependency as well.
Invoke the IoC Container to instantiate a registered object with its dependencies already injected. For our project, the following demonstrates getting a handle to the WorldDomination class:
```
WorldDomination worldDomination = (WorldDomination) 
   ServiceLocator.Current.GetService(typeof(WorldDomination));
```
This line of code tells the IoC Container to return an instance of the requested class with its dependencies injected. This actually uses an IoC Container known as Common Service Locator. Since almost all IoC Containers do the same actions – registering and instantiating objects – the major IoC Container authors worked together to establish a common interface for all IoC Containers. In this way, you can register your objects with your favorite IoC Container and then register the IoC Container itself with the Common Service Locator. It’s enough to make your head spin, but leads to further decoupling to the IoC Container itself so that you can easily swap out one for another (e.g., Castle Windsor for Ninject) without modifying the code which depends on the IoC Container.

How you setup and use an IoC Container depends on the environment that you’re developing within. For example, if you’re developing an ASP.NET MVC application, you can use an IoC Container to instantiate the controllers, injecting their dependencies via the controller’s constructor. For example, the following two lines of code, within the Global.asax.cs file, instruct the ASP.NET MVC framework to use Castle Windsor to instantiate controllers:

IWindsorContainer container = new WindsorContainer();
ControllerBuilder.Current.SetControllerFactory(
   new WindsorControllerFactory(container));

The only thing left is to register the controller dependencies that they require for their constructors and the IoC Container takes care of the rest.

While this will be enough to gain a basic understanding of what an IoC Container is and how it is used, a bit more studying will be required to learn how to actually set up and use one. For more information about IoC Containers, including examples of setup and usage:

Common Service Locator: http://www.codeplex.com/CommonServiceLocator
Castle Windsor Tutorial: http://www.castleproject.org/container/gettingstarted/
LinFu in 5 Minutes: http://www.codeproject.com/KB/cs/LinFu_IOC.aspx
Using Ninject: http://ninject.codeplex.com/wikipage?title=user%20guide&ProjectName=ninject
StructureMap: http://structuremap.sourceforge.net
A classic IoC Overview: http://martinfowler.com/articles/injection.html
Using IoC in a project: http://wiki.sharparchitecture.net/

Finally, for a much more thorough analysis of many of the concepts described above, I highly recommend reading Robert Martin's Agile Principles, Patterns, and Practices in C#.

Final Thoughts

Regardless of whether you’re looking to take over the world with a dominion of evil robots or simply trying to build a better web app, dependency injection and the use of an IoC Container can lead to an easy to maintain application that is loosely coupled to service dependencies. These attributes naturally lead to better quality and happier developers. (Both of which I would put squarely into a category known as “good.”) But while these techniques provide many benefits, they also introduce a bit more complexity; especially when going from “manual” dependency injection, as we did in the coding example, to the use of an IoC Container. Accordingly, your project team should carefully consider the benefits and added complexity to determine if it's right for your project. While the use of an IoC Container would be a bit overkill for a very simple application, its use is obviously pivotal in creating self-aware robotic legions for conquering the world.

Billy McCafferty
http://wiki.sharparchitecture.net/

The dojo is open... www.ITSamuraiSchool.com

Billy McCafferty — Mon, 18 May 2009 22:52:00 GMT

Codai's IT Samurai School is open for enrollment. It's Flash, so it'll take a few moments to download...but I hope you'll agree it's worth the wait... http://www.itsamuraischool.com

A few fun facts about the website and the school itself:

Born our of a short conversation at a BBQ concerning my desire to do high quality training
100% Flash front-end, outside of the checkout screens
Built on latest S#arp Architecture available from the trunk
6 months in the making, including the site and class preparation
21 classes initially, including ASP.NET MVC, Object-Oriented JavaScript, SQL Server Integration Services, S#arp Architecture and many others
5 cities initially, including Denver, Austin, San Jose, Washington DC, and Seattle
A handful of top notch trainers, including myself (IMHO)
Guest speakers and occasional, one-run only, unique courses

We look forward to seeing you in class!

Billy McCafferty

Refactoring Service Dependencies to Separated Interface

Billy McCafferty — Fri, 31 Oct 2008 00:06:00 GMT

There are two factors which have a tremendous impact on a team's likelihood to abandon unit testing:

Fragile tests which break often and require regular maintenance; and
Slow tests which add excruciating seconds, if not minutes, to the total time of unit test runs.

I find that fragile tests are often those that depend on data within a database, which can become out of synch with the assertions in the unit test, and those that depend on external services. I use the term "services" loosely to define anything that represents an external resource; examples include web services (including RESTful services), file repositories, databases, and remote procedure calls. But these dependencies on services aren't just bad for keeping unit tests running fast and stable; they can often lead to maintenance problems due to tight coupling. This post presents a common scenario wherein business logic is tightly coupled to a service and provides instruction for refactoring the concrete dependency to a more testable and loosely coupled configuration.

Separated Interface & Motivations for its Inclusion

The design pattern known as "Separated Interface" (coined by Martin Fowler; aka "Dependency Inversion" by Robert Martin) is a technique for reducing coupling to concrete dependencies; it achieves this by coding to the interface of the dependency while maintaining the implementation details of the dependency within a separate application layer. You'll find that the trick of using separated interface, as with any design pattern, is knowing when to use it appropriately, rather than how to use it, as the underlying mechanism is rather straight forward. I find it best to use on most external services, as defined earlier, but should still be considered on a case by case basis. You should ask yourself if introducing Separated Interface will make the overall application more maintainable and testable while keeping complexity to a minimum. When used appropriately, the benefits of this decoupling include:

Improving the ability to unit test "around" the dependency using mocks/stubs,
Greatly increasing the performance of domain oriented unit tests as a side effect of using mocks/stubs,
Consolidating dependency communication details to a separate tier,
Providing a plug-in architecture for external dependencies to be easily swapped without recompiling (e.g., when that 3rd party tool gets upgraded), and
Allowing developers to develop and compile the domain logic without having all the dependencies in place (e.g., not having to install 3rd party tools A, B, and C just to develop and compile unrelated code).

To illustrate, the following diagram shows two separate assemblies within the project solution.

The Project.Core assembly contains all of the domain logic for the application. In addition to domain objects, such as Customer and Order, this example assembly also contains an interface to an external service; in this case, IOrderDao, which represents an interface to a data-access object. Note that this is only the interface of that service; the domain logic has no idea (or dependency) on the underlying implementation details outside of its conformance to the interface.
The Project.Data assembly contains all of the implementation details for the interface and contains an assembly reference back to Project.Core. This is the fundamental key to Separated Interface. In most applications, the data communications and external dependencies are depended on by the business logic. With Separated Interface, the reverse is true wherein the service implementation depends on the interfaces within the domain layer - the true core of the application.

This pattern typically presents two challenges. The first is, how does the domain logic get passed a reference to the actual concrete dependency if all it knows about is an interface? The dependency can get passed to the domain object via its constructor or a setter either manually or with an IoC container, such as Castle Windsor or Ninject. (See http://www.codeproject.com/KB/architecture/DependencyInjection.aspx for an example of doing so.) The second and more challenging problem is, how can we get the external resource to depend on our domain layer interface when it's a third party tool? Recall that we want the service to depend on the domain, not the other way around. In these cases, it's appropriate to create a facade which wraps the complicated details of communicating with the dependency; then, we simply add an interface for the facade to our domain layer. The following example will walk through looking at a real world smell of a concrete dependency on a third party API - exposed as a RESTful service - and making appropriate modifications to introduce Separated Interface.

Real World Example (aka "the smell")

To set the appropriate context, assume that (smelly) code has been developed to fulfill the following user story: On a nightly basis, the application will download any new Observations added during the previous day, input into ExternalApp, so that they may be copied locally. (An "observation" is the domain name of the record we're retrieving.) The idea is that an external application exists, aptly called ExternalApp, which exposes a RESTful service to retrieve newly added records. Our nightly process ties into the service, retrieves the appropriate records, parses the response, and copies the appropriate information locally.

Our application currently has three tiers:

The MySynch.Tests assembly contains unit tests to verify these processes. (No, of course we didn't call it "MySynch," but that'll work for now.)
The MySynch.Core assembly contains all of the business logic for communicating with the RESTful service and processing the results. Assume it contains one object called ObservationMigrator which encapsulates this logic.
The MySynch.Console application which is the nightly process, which we won't concern ourselves with here.

The following unit tests, found in the folder MySynch.Tests/Core, verify that the records can be retrieved and that the data migration is successful:

[TestFixture]
public class ObservationMigratorTests
{
    [Test]
    public void CanRetrieveObservationRecords() {
        ObservationMigrator migrator = new ObservationMigrator();
        IList<Observation> observations = migrator.GetNewObservations();
        Assert.That(observations.Count, Is.GreaterThan(0));
    }
    [Test]
    public void CanMigrateObservationRecords() {
        ObservationMigrator migrator = new ObservationMigrator();
        Result result = migrator.MigrateRecords();
        Assert.That(result.CompletedSuccefully);
    }
}

The first test is really verifying if the communications with the RESTful service - the third party tool - are successful. The second test is verifying if the data migration occurs successfully, assuming the communications with the RESTful service are successful. Without even looking at the underlying code, these tests are already unveiling a number of smells:

It's obvious that the ObservationMigrator isn't adhering to the Single Responsibility Principle (SRP). It's providing means to communicate with the service, via GetNewObservations, and it's responsible for migrating those records to whatever destination. Consequently, it's likely that it's inherently complex and that it's likely to be touched often during maintenance for different reasons.
The second test is really just trying to make sure that the migration occurs successfully without necessarily wanting to retest proper communications to the RESTful service; that was what the first test was doing. The problem with this is that if the service goes down, in addition to breaking the first test, it'll start breaking the second test which is really domain-oriented in nature. It would be great if we could test the migration process without having to retest the communications to the service.
Finally, communicating with any external resource via unit tests, especially web-based services, adds seconds to running the unit tests. This can add minutes with hundreds of unit tests. Again, if all we really want to test in the second test is the domain logic for completing the migration, then we shouldn't be reconfirming communications are working and adding wasted time to running the tests.

Indeed, if we look under the covers at the ObservationMigrator class, we confirm our concerns with respect to SRP and HTTP requests embedded into the business logic:

public class ObservationMigrator
{
    public IList<Observation> GetNewObservations() {
        WebClient webClient = new WebClient();
        byte[] response = webClient
            .DownloadData("http://www.example.com/RESTfulServiceToRetrieveRecords");
        string responseContent = Encoding.ASCII.GetString(response);
        IList<Observation> observations = new List<Observation>();
        // Lots of complicated parsing...like, crazy complicated
        return observations;
    }
    // Result is a DTO which let's the caller know if everything succeeded
    public Result MigrateRecords() {
        IList<Observation> observations = GetNewObservations();
        foreach (Observation observation in observations) {
            // Migrate each observation to the appropriate repository
        }
        return new Result(...);
    }
}

Refactoring Mechanics

In refactoring this code, we'll want to follow the steps below to reduce the amount of work done between tests and to minimize the amount of breaking code in the system at any given point in time. Our goal is to always be very close to a check-in...the "don't anybody check anything out for the next two days" won't fly here.

Refactor with Extract Class to pull the RESTful communication and parsing logic into it's own class. This extracted class becomes the facade to the underlying RESTful API.
Refactor with Extract Interface to create an interface for the new API facade.
Refactor with Introduce Constructor Injection to replace the creation of the dependency with a constructor parameter.
Refactor with Introduce Stub Object to modify the testing of the migration test to focus on the domain logic rather than on retesting communications with the API.
Refactor Architecture by Tiers to move the implementation details to a separate tier.

Step 1: Extract Class

So without further ado, let's start with step 1, extracting a class to act as a facade to the underlying RESTful API. As always, we begin with a test; let's do so by adding a new test as follows:

[TestFixture]
[Category("ExternalApp Integration")]
public class ObservationRetrieverTests
{
    [Test]
    public void CanRetrieveRecordsToImport() {
        ObservationRetriever observationRetriever = new ObservationRetriever();
        IList<Observation> observations = observationRetriever.GetNewObservations();
        Assert.That(observations.Count, Is.GreaterThan(0));
    }
}

Note that we've added this fixture to a category called "ExternalApp Integration." Doing so, we're able to conveniently turn off this slow-running category of unit tests while we're working on other areas of the application. Compiling this breaks, so the next step is to add just enough code to MySynch.Core to get the test to compile:

public class ObservationRetriever
{
    public IList<Observation> GetNewObservations() {
        throw new NotImplementedException();
    }
}

You should now be able to run the test and see it fail with a NotImplementedException. Yes, you want it to fail...a failing test is your next goal to fix. So now let's write enough code to get it to pass by copying the GetNewObservations() method from ObservationMigrator into ObservationRetriever:

public class ObservationRetriever
{
    public IList<Observation> GetNewObservations() {
        WebClient webClient = new WebClient();
        byte[] response = webClient
            .DownloadData("http://www.example.com/RESTfulServiceToRetrieveRecords");
        string responseContent = Encoding.ASCII.GetString(response);
        IList<Observation> observations = new List<Observation>();
        // Lots of complicated parsing...like, crazy complicated
        return observations;
    }
}

All the unit tests should now be passing again. But we've introduced a problem...the smell of duplicated code. We can remove this smell by replacing the ObservationMigrator.GetNewObservations() method with the creation of an ObservationRetriever object and the invocation of the method therein:

public class ObservationMigrator
{
    public Result MigrateRecords() {
        ObservationRetriever observationRetriever = new ObservationRetriever();
        IList<Observation> observations = observationRetriever.GetNewObservations();
        foreach (Observation observation in observations) {
            // Migrate each observation to the appropriate repository
        }
        return new Result(...);
    }
}

You'll need to drop ObservationMigratorTests.CanRetrieveObservationRecords() to get it to compile; but all the tests should still be passing. We've officially completed step 1 by extracting a class from the ObservationMigrator class into a facade class which only concerns itself with communicating with the RESTful API and parsing the results.

Step 2: Extract Interface

Step 2 is introducing the interface to our service. Instead of creating a new test, we simply modify the ObservationRetrieverTests.CanRetrieveRecordsToImport() method to invoke GetNewObservations() via an interface rather than the concrete object itself:

[TestFixture]
[Category("ExternalApp Integration")]
public class ObservationRetrieverTests
{
    [Test]
    public void CanRetrieveRecordsToImport() {
        // Note the "I" that's been introduced below
        IObservationRetriever observationRetriever = new ObservationRetriever();
        IList<Observation> observations = observationRetriever.GetNewObservations();
        Assert.That(observations.Count, Is.GreaterThan(0));
    }
}

Since a quick compilation shows that this isn't compilable (and it shouldn't be), let's add an interface to MySynch.Core which reflects the method on our facade class, and have ObservationRetriever then implement that interface. First, the interface that we need to add to MySynch.Core:

public interface IObservationRetriever
{
    IList<Observation> GetNewObservations();
}

Then, simply have the ObservationRetriever class implement this interface: public class ObservationRetriever : IObservationRetriever { ...

All unit tests should be passing again...oh, happy days again.

Step 3: Introduce Constructor Injection

The ObservationMigrator class still has a concrete dependency on the ObservationRetriever...damn that concrete dependency!! So let's now introduce constructor injection to remove this tight coupling. Guess where we're going to start...that's right, back in a unit test. But here again, we don't need to add a new test, we simply need to alter an existing one, ObservationMigratorTests.CanMigrateObservationRecords(). The modified test should be as follows:

[Test]
public void CanMigrateObservationRecords() {
    IObservationRetriever observationRetriever = new ObservationRetriever();
    ObservationMigrator migrator = new ObservationMigrator(observationRetriever);
    Result result = migrator.MigrateRecords();
    Assert.That(result.CompletedSuccefully);
}

All we're intending to do here is to pass an instance of IObservationRetriever to the migration utility rather than having it create its own. Tying to compile breaks the build, so now let's write just enough code to get this puppy to compile...usually we'd write just enough to compile and then just enough to pass, but I'm getting tired of writing, so we'll combine the two here:

public class ObservationMigrator
{
    public ObservationMigrator(IObservationRetriever observationRetriever) {
        // Let's do a design-by-contract (DbC) check to make sure we're not 
        // given a null service dependency. For more about this, see 
        // http://devlicio.us/blogs/billy_mccafferty/archive/2006/09/22/Design_2D00_by_2D00_Contract_3A00_-A-Practical-Introduction.aspx
        Check.Require(observationRetriever != null, 
            "observationRetriever may not be null");
        this.observationRetriever = observationRetriever;
    }
    public Result MigrateRecords() {
        IList<Observation> observations = observationRetriever.GetNewObservations();
        foreach (Observation observation in observations) {
            // Migrate each observation to the appropriate repository
        }
        return new Result(...);
    }
    private IObservationRetriever observationRetriever;
}

With these changes, everything should now compile and the tests should be passing again.

Step 4: Introduce Stub Object

As mentioned in the previous gripes against the original code, the ObservationMigratorTests.CanMigrateObservationRecords() test shouldn't be worried about communicating with the RESTful service; it should only be concerned with ensuring the migration occurs assuming the RESTful service is working as expected. The ObservationRetrieverTests.CanRetrieveRecordsToImport() test is already verifying that the service is working, so it's just adding extra time and complexity to the ObservationMigratorTests.CanMigrateObservationRecords() test. Therefore, let's now introduce a stub object to simulate communications with the RESTful service. Keep in mind that this modification isn't being done to test new functionality; instead, it's a refactoring to an existing unit test to properly focus the test's attention on the domain logic. Accordingly, the ObservationMigratorTests now reflects the following:

[TestFixture]
public class ObservationMigratorTests
{
    [Test]
    public void CanMigrateObservationRecords() {
        IObservationRetriever observationRetriever = new ObservationRetrieverStub();
        ObservationMigrator migrator = new ObservationMigrator(observationRetriever);
        Result result = migrator.MigrateRecords();
        Assert.That(result.CompletedSuccefully);
    }
    private class ObservationRetrieverStub : IObservationRetriever
    {
        public IList<Observation> GetNewObservations() {
            IList<Observation> observations = new List<Observation>();
            observations.Add(new Observation(...));
            observations.Add(new Observation(...));
            observations.Add(new Observation(...));
            return observations;
        }
    }
}

In the unit test above, a stub object has been created which implements IObservationRetriever and fakes communications with the external service to retrieve three new Observation records. The ObservationMigrator.MigrateRecords() method happily uses this stub, not knowing that it's actually not talking to the external RESTful service. We're now focusing this unit test on testing the domain logic, outside of communicating with the RESTful service. Again, the ObservationRetrieverTests fixture of tests is already testing the communications with this service and related parsing functionality...no need to retest it here.

In such a simple case, where the interface only has one method to implement, a stub is perfectly adequate. But we're passing three observations to the migration step without verifying that all three were processed...we might as well have just passed one, or none at all for that matter. This test could be extended in two ways to verify the underlying migration processed each and every Observation appropriately. The first is that we could include the status of how many observations were migrated within the Result object returned from the migration step. The other way is to use a mock object, instead of a stub, which can verify that the expected behavior is occurring under the scenes. Although mocking is beyond the scope of this post, I encourage you to take a look at http://xunitpatterns.com/ and Rhino Mocks for more information.

Step 5: Refactor Architecture by Tiers
(i.e., add the "Separated" part of "Separated Interface")

After all of this, we're finally at the point of culmination: the introduction of the Separated Interface. But by now, it's rather anticlimactic as the hard work has already been done. Nonetheless...on with the anticlimactic conclusion! A problem with the solution as it stands is that the service facade still physically resides in the MySynch.Core assembly. Consequently, there's nothing we can do to prevent developers from bypassing our elegant interface and go straight to the ObservationRetriever class itself. This is a wonderful side benefit of introducing Separated Interface; it forces developers to adhere to certain design decisions. To mitigate this concern, and to remove all coupling from MySynch.Core to the implementation details of ObservationRetriever, we'll refactor the architecture itself to move the implementation details to a separate tier. Let's head back to the tests assembly and add a new folder called ExternalAppIntegrator; recall that "ExternalApp" is the name of our 3rd party tool which exposes the RESTful service. Now move the ObservationRetrieverTests class to this new folder. This new folder represents the new class library that we'll be introducing. The ObservationRetrieverTests class should then be modified to reflect the change in namespace and a using statement to pull in the class library which we have not yet created (remember...write the test and then write the code to get the test to pass):

using NUnit.Framework;
using NUnit.Framework.SyntaxHelpers;
using System.Collections.Generic;
// The compiler will throw an error on the following line since 
// the ExternalAppIntegrator.Core class library doesn't exist yet
using ExternalAppIntegrator.Core;
namespace Tests.ExternalAppIntegrator.Core
{
    [TestFixture]
    [Category("ExternalApp Integration")]
    public class ObservationRetrieverTests
    {
        [Test]
        public void CanRetrieveRecordsToImport() {
            IObservationRetriever observationRetriever = 
                new ObservationRetriever();
            IList<Observation> observations = 
                observationRetriever.GetNewObservations();
            Assert.That(observations.Count, Is.GreaterThan(0));
        }
    }
}

We've introduced some work for us here, but we will overcome! Firstly, create a new class library called "ExternalAppIntegrator.Core" and add a reference to it from our MySynch.Tests assembly. Next, move the class MySynch.Core/ObservationRetriever.cs to the ExternalAppIntegrator.Core class library...be sure you move it completely by deleting the original. Ack...a new problem now! We've moved the class but it depended upon IObservationRetriever. Fear not! Simply add a reference to MySynch.Core to the ExternalAppIntegrator.Core class library. Since we've removed all other dependencies to the ObservationRetriever from MySynch.Core, all the unit tests will still pass. As a matter of fact, we're done!

But wait, how do we know it's working? Although not necessary, since the unit testing has proven that all of the inner workings are in order, we can make sure our Separated Interface is working as expected with the rest of the migration process by going back to ObservationMigratorTests.CanMigrateObservationRecords() and replacing the creation of the stub object with our concrete implementation: IObservationRetriever observationRetriever = new ObservationRetriever(); With that said, I'd quickly change it back to using the stub so that we can keep that unit test focused on the domain rather than communications with the RESTful service.

In Conclusion

We've taken some rather smelly dependencies and moved them to a dedicated class library which concerns itself with the management of a facade over the external API. We finished by having this facade implement an interface found without our domain layer so that the domain layers could communicate to the external API without having a direct, concrete dependency on the implmentation details. The hope is that this has made our code more testable, less coupled, and more maintainable down the road. The cost has been the introduction of increased complexity. Accordingly, care should be taken before the introduction of any design pattern to ensure that the pattern itself is "pulling its weight," so to speak.

I have found immense value from this design pattern; it has been particularly handy for removing direct dependencies on data access objects. The sample code that comes along with S#arp Architecture has a full example of just such an instance. Happy coding!

Billy McCafferty

(Re)Assessing Best Practices

Billy McCafferty — Mon, 15 Oct 2007 18:47:00 GMT

As we progress as developers, we strive to seek out the "best" way to perform our craft. The chosen methods to attain this lofty goal always bring with them a number of developmental trade-offs. Some techniques may simplify the code but lessen fine grained control while others enable greater power while introducing complexity. Model-View-Presenter with ASP.NET is a wonderful example of the latter. MVP aims to facilitate test-driven development of ASP.NET applications with the unfortunate introduction of added complexity. So while developers will be more confient in the quality of their product, MVP hinders their ability to easily maintain code. Yes, the tests will (hopefully) inform the developer that a bug has been introduced, but the inherent complexity of MVP makes it difficult for later team members to become comfortable with the code base and maintain it as project development continues.

Fortunately, as time progresses quickly in our field, resources and tools become available which enhance our ability to write powerful applications while simplifying the coding process itself. The introduction of NHibernate, for example, eliminated vast amounts of data-access code while still providing powerful facilities for managing transactions and dynamically querying data. Castle MonoRail (and Microsoft's upcoming spin-off of this framework) now does for writing testable and maintainable .NET web applications what NHibernate (and the upcoming LINQ to Entities) did for ADO.NET. This is not to say that the previous techniques were necessarily wrong, but that they were only applicable considering the developer's toolset that was available at the time of selection.

In adapting to the evolution of our field, it is important for developers to note when an accepted technique is no longer valuable in light of current alternatives. Specifically to this example, MVP was a very powerful technique for writing ground-up, test-driven ASP.NET applications but is no longer a strong candidate for consideration when compared to the time saving benefits and simplicity of Castle MonoRail and Microsoft's upcoming MVC framework. Oddly, it is sometimes difficult to "give up" on something that worked perfectly fine before, but that's the nature of our business...one tenet that's not likely to change anytime soon.

Billy McCafferty

Considering a View's Get/Set Roles with MVP

Billy McCafferty — Mon, 19 Mar 2007 15:40:00 GMT

[Update May 7, 2007: Added sequence diagram, submitted by Yanic Inghelbrecht, to better explain "Use Update Method in lieu of Getter" technique.]

There are many ways to implement Model-View-Presenter; Supervising Controller and Passive View are just two implementation examples. A question was asked on my overview of the subject as to whether or not a view's properties should have both setters and getters. For example, consider an "add/edit customer" view. This view first shows the available customer data, via a setter on the view, and then allows the user to update the information displayed. Should the view have both a setter and a getter for the customer class? (As I understand it, if you're using Passive View, this is a moot question as the view should not be intelligent enough to interpret and/or update the domain model directly.) But if you're using Supervising Controller or another approach which allows the view to know about the domain model (or DTOs for the domain model) then this is a great question to discuss. I'll list the (un)successful approaches I've used and pros/cons of each. (Here, I use "entity" to mean any domain object that's stored within the DB.)

Return Ready-to-Save Entity from View via Getter
With NHibernate, if you have a ready-for-storage entity, you may simply call SaveOrUpdate on that object. If a new entity is being created, then the view may need to work with factories and/or be aware of the business rules for creating a ready-to-save entity. The internal consistency of that object can, obviously, be verified by the presenter or validation logic built into the domain object itself, but I never liked this approach as it gives the view a lot of responsibility for creating a new entity and creates a lot of untestable code.

If an existing entity is simply being updated and returned via the getter, the view must have a handle on the existing entity to update it. Accordingly, the view can:

Re-create the entity from scratch, set its ID accordingly, and return the ready-to-save object. This approach is wrought with problems. If the view gives the recreated object a wrong ID, you've now corrupted your data. In my overview of NHibernate, I exposed ID as a publicly, settable property. In my upcoming update to this article, I recommend not exposing the ID setter. Only the ORM or entity-loader should be able to set this critical property. As another drawback, if the entity keeps track of auditing information, such as DateCreated, this approach will overwrite it. There are other drawbacks to this approach and it should be avoided.
The view can talk to a DAO directly to get the existing entity, update it's properties, and return it to the view. Although not as bug-prone as the previous suggestion, this now gives the view the responsibility of coordinating with the data-access layer directly. This grays the line between the responsibilities of the view and the other layers of the application and adds non-trivial business logic to the view which is difficult to test.
The view can be given a direct handle to the loaded entity by the presenter via another setter. This seems a bit round-about and depends on a two-step process. Additionally, the view now has two setters which may be confusing and difficult to maintain.

Return DTO from Getter
Instead of returning a fully ready-to-save object from the view, the view could, alternatively return a DTO consisting of only the appropriate information to be updated. The presenter could, in turn, get the existing object from the DB (or create a new one), transfer the DTO information onto the persistent object, and save it back to the DB. The primary benefit is that you can completely separate the view from knowing anything about the domain model while still being able to communicate more than one primitive at a time. The major drawback is that DTOs now need to be maintained along with their domain model pairs. So if you add a new, updateable property to the domain object, you'll then need to add the property to the DTO and then modify the DTO-to-domain-object transfer logic within the presenter. (Alternatively, You could also add an UpdateWith(DTO) to the entity so that the transfer logic is more contained, but it still leads to further maintenance.) But the bottom line is that this is a good approach if you really need to keep your views from having a direct dependency on the domain layer. Because of this clean separation, this brings the added benefit of removing temptation for domain logic to bleed into views since the view is limited to communicating with the Presenters/DTO layer. But again, it comes with a lot of additional maintenance to keep the DTOs synched with the domain model.

Use Update Method in lieu of Getter
A simpler approach that I'm having luck with is including a separate update method on the view and not having a getter for the entity. Going back to the "add/edit customer" example, a "Customer { set; }" would be on the view interface to show the data, but not a getter. The view interface would then also include a "void Update(Customer customer);" method. Accordingly, the view would then accept a customer object from the presenter and update its properties from the view. The communications would be as follows:

The above image was created and submitted by Yanic Inghelbrecht via Trace Modeler. Thanks Yanic!

The sequence diagram is interpreted as follows:

User clicks "Save Changes" button which invokes btnSaveChanged_OnClick in the code-behind.
Code-behind calls presenter.SaveChangesTo(customerId).
Presenter talks to the Customer DAO (or service layer) to get the loaded entity from the DB, using the ID passed to it from the view. (If the ID was 0, then the Presenter creates a new entity via a constructor or factory.)
Presenter calls view.Update(customerToBeSaved), passing the entity loaded from the DB.
View updates the entity with values from a form.
(Optionally, when appropriate) Presenter, or more appropriately, the domain layer via something like the Validation Application Block, verifies that the updated entity is internally valid.
Presenter calls customerDao.SaveOrUpdate(customerToBeSaved).

With this approach, the view remains ignorant between whether a new entity is being created or an existing one is being updated. Furthermore, separate DTOs aren't being maintained while still minimizing the responsibilities of the view. As an added bonus, there's very little code that's not testable. The drawback to this is that it gives the view direct access to the persisted objects and, thus, the domain layer itself.

A Place for View Getters
There are other scenarios in which getters on the view are valuable. One such example is an IsDefaultView/IsPostBack getter so that the Presenter can determine if post-back-specific actions are necessary. But in my own experiences, I usually pass contextual data, such as this, into the InitView, or other method, of the Presenter. But at times, exposing a getter on the view can be simple and useful.

Recommendations
If your project (team) requires a very clean separation between the view and domain layer, then I recommend considering the "Return DTO from Getter" approach. This keeps the view very contained while allowing you to be more efficient than passing around a ton of primitives. But like CRUD stored procs, the DTOs require a lot of maintenance. Alternatively, if you have a smaller and/or more experienced team, you can bypass the DTO layer altogether in favor of the "Use Update Method in lieu of Getter" approach. This is simple, minimizes responsibilities of the view but, on the downside, allows the view to do more damage, so to speak. But with proper encapsulation, the view's access to the domain layer can still be limited. I've been developing/maintaining my current project for 1 1/2 years and have found the latter approach to be the most maintainable and simple. But either way, both approaches lead to highly testable code.

Billy McCafferty