Billy McCafferty

A Few Thoughts on DDD, DTOs, View Models, Repositories and Separation of Concerns

2010-03-06T16:41:00Z

What I've loved most about developing an open-source project is the ideas that I get from others who look at the work and either A) validate ideas, B) suggest that something stinks, or C) call a royal WTF and force people (e.g., me) to explain ideas more fully. It's usually during these explanation attempts that light bulbs start to come on and ideas are refactored and become more substantiated (or at least more defendable). The S#arp Architecture forums have been a gold mine, IMO, for the discussion of the practical application of balanced DDD techniques on real-world projects. After feedback that I've already gotten on my recent post concerning application services and CQS and listening to and adding my two cents concerning ideas on the S#arp forums, I'd like to share a few ideas (that I shared on the forum) concerning drawing the line among application layers, the management of DTOs and View Models, and a general discussion of separation of concerns. (As Chris Carter correctly (and a tad bluntly) explained, don't take these ideas at face value...use them as a point of discussion and contemplation to come to your own conclusions with your project team...and let me know what you think.)

Every great developer knows that any problem can be solved by adding another layer of abstraction. ;)

Obviously, with additional layers of abstraction and indirection, we also introduce complexity and additional objects which need to be maintained. Over the past year, I have had struggles concerning just how far the "DTO paradigm" should be taken to effectively separate layers from the domain and "best practices" for how they should be used. I don't think the answer can be found in a best practice (but I also hate to leave it simply at "it depends"). I truly feel that the most important practice is for a project development team to decide, and firmly agree upon, which approach will be taken on a project and for the agreed upon approach to be followed with discipline (and enforced via code reviews) by the team.

On a couple of recent projects, the biggest problems we ran into weren't concerning which approach to leveraging DTO for layer separation was "best," but on coming to a firm layering decision and having the team discipline to adhere to those practices. Because of these "gray" areas, which we didn't come to a firm team agreement on, there were inconsistent practices used by the developers which led to rot, particularly between the controller (or code-behind) and application services layer.

So while we can discuss which approach is best or most effective, I think the greater concern is for the team to pick a direction, to stick to it, and to diverge or refactor as a team decision.

But as I said, I don't want to leave it at "it depends." ;) So after reading everyone's ideas here and contemplating further on past projects and the ideas I recently put forth in http://devlicio.us/blogs/billy_mccafferty/archive/2010/03/05/better-application-services-and-cqs-with-s-arp-architecture-1-0-q3-2009.aspx, here are a couple of ideas that I've been trying to adhere to (and/or will be adhering to on my next project). The layers referenced below may be viewed in a package diagram here for better clarity of what's discussed.

Usually, one object should be used to populate a form while a second should be used to collect data from a form. The former would contain information for default/selected values along with drop-down values, for examples, while the latter would encapsulate which options were selected when the user submitted the form. Personally, I use a <EnityName>FormViewModel to populate the form and use an entity to collect from the form...but I see it equally valid to instead use an intermediary DTO to collect from the form if further separation is warranted. Additionally, I feel it is OK for a FormViewModel object to have references to domain objects to make it easier to transfer data to the form for input population. This is an arguable point and would be just as valid to only pass DTOs in the view model for a cleaner separation between view and domain, with the acceptance that there will be more objects and maintenance that comes with that approach.
I believe that view models should be maintained either in the application services layer OR in a separate ViewModels class library if a project team feels the additional separation is warranted (although I can't envision much reason to do so). (I'll be modifying my sample project post to reflect the former.) Either way, they should not be maintained in .Core to keep a very clean separation from domain concerns. While the name "ViewModel" implies that they are a view concern, they only describe what should be displayed, not how it should be displayed. Accordingly, there is still a very clear separation of concerns between the view layer (which the user interacts with) and the view model classes. Furthermore, the application services expose methods which should be usable by any type of client. In line with this, a ViewModel or FormViewModel class does not impart a particular mechanism for the implementation of the view layer - again, it only describes what data should be available to the view. So even if view model classes are maintained in app services, this does not preclude there use by various client types; e.g., they could be equally consumed by ASP.NET, ASP.NET MVC, Spark, SilverLight, etc. (The only caveat is that if the view model classes maintain references to domain objects, then it will be more difficult to expose them for serialization via web services and WCF. Accordingly, a project team should firmly decide if ViewModels may contain references to domain objects.)
I believe that DTOs should be maintained in a separate assembly and have no reference to .Core or be maintained in .Core if warranted with the acceptance of decreased "shareability." DTOs should almost always be serializable; i.e., transferable via web services and WCF. This allows the DTOs class library to be easily shared with a consumer of web services which may expose these objects, for easier deserialization. Some of the motivations for deciding when to use DTOs would be to provide a clean separation between the view and app services (if a project team feels its warranted), to expose data to web service consumers, and to better enable Command/Query Separation.
Repositories may return Entities, Value Objects or DTOs, in addition to other "basic" types; e.g., primitives, arrays, etc. Ideally, repositories would not be used by domain objects (even via their interfaces) and would have their interfaces maintained in the application services layer. If the repository interfaces are maintained in .Core, then this forces DTOs to be maintained in .Core as well with the possibility that domain references may "leak" into DTOs. If the repository interfaces are maintained in .ApplicationServices, then DTOs may be maintained in a cleanly separated library without worrying about the line between DTOs and domain objects being grayed. But data interfaces (and DTOs) may be maintained in .Core if the team feels the simplification is warranted.

With all this said, I still think the most important decision is deciding a path to take for your project, having the project team agree upon it, and maintaining the discipline to stay on course.

Your feedback on these ideas is most welcome!

Billy McCafferty

Better Application Services and CQS using S#arp Architecture 1.0 Q3 2009

2010-03-05T18:47:00Z

Updated 2010.03.09 to reflect small modifications that were decided through subsequent discussions on S#arp forum and other DDD posts.

Obviously, S#arp Architecture is the bee's knees when it comes to developing ASP.NET MVC applications. ;) But as a project evolves and gets larger, "out of the box" S#arp Architecture 1.0 guidance runs into a few pain points. Particularly, there's poor use of the application services layer, the separation between controllers and application services is not very clear, entity listing pages become performance bottlenecks as the domain model gets sizable, there is no command/query separation (CQS) "out of the box", and unit tests require re-occurring maintenance to deal with changes in the number of constructor parameters to controllers and application services. While the amicable and adroit Alec Whittington (who is taking over the lead role from me on S#arp Architecture) is hard at work upgrading S#arp Architecture to accommodate recent dependency upgrades and accommodating ASP.NET MVC 2, I wanted to take a stab at resolving some of the architectural issues that I've run into, on S#arp projects over the past year.

I've developed and included a sample project, built on S#arp Architecture 1.0 Q3 2009, for the following key reasons:

To resolve some "pain points" that develop as S#arp projects grow to large sizes,
To demonstrate better use of the application services layer,
To demonstrate better command/query separation of the entity listing pages for dramatically better performance as the domain model grows,
To create an architectural spike for a new project I'm working on, and
To collect feedback from the S#arp community to determine if this is an appropriate architectural direction for the next release of S#arp Architecture.

Please post any feedback and/or suggestions you may have in the comments below or, more preferably, to the S#arp Architecture forums at http://groups.google.com/group/sharp-architecture.

Setup instructions

Unzip the sample project to a BetterAppServices folder
Create a new database called BetterAppServices
Using SQL Enterprise Manager, run:
1. /BetterAppServices/db/Schema/CreateBetterAppServicesDb_ChangesWillBeLost.sql
2. /BetterAppServices/db/StoredProcedures/CreateGetProductCategorySummaries.sql
3. /BetterAppServices/db/StoredProcedures/CreateGetProductSummaries.sql
Open /BetterAppServices/BetterAppServices.sln with VS 2008
In VS 2008, open BetterAppServices.Web/NHibernate.config and change the connection string to point to your BetterAppServices database
Right click the BetterAppServices.Web project and "Set as StartUp Project"
Run (F5) the project to see everything in action.

Changes from "out of the box" S#arp Architecture 1.0 Projects

This project is more of an architectural spike more than anything else at the moment. Accordingly, the CRUD scaffolding generator has been removed and non-essential unit tests have been removed to focus on the architecture itself. Many of the changes will be incorporated into the S#arp CRUD scaffolding generator; this will either be available in the next release, or will be provided as an add-on, as this new approach does add complexity and introduces a major breaking change to existing 1.0 projects.

Major changes include:

/db Folder Changes
- Added /db/Schema/CreateBetterAppServicesDb_ChangesWillBeLost.sql. This gets auto-regenerated when unit tests are run. The motivation was to have the DB schema automatically maintained while developing.
- Added /db/StoredProcedures/CreateGetEntityNamePluralSummaries.sql. These SPs provide command/query separation for the entity listing (Index.aspx) pages, which frequently became a performance bottleneck.
Cross-Project Changes
- Moved /BetterAppServices.Core/DataInterfaces/* to /BetterAppServices.ApplicationServices/DataInterfaces/. This is the only major, possible breaking change for existing applications. This was done to further remove the potential of domain objects using data repositories directly, and to allow the repositories to return objects from a new Dtos project for command/query separation, among other benefits (dicussed below). Decided not to do this to support domain services which may require the use of repositories and to make upgrading from previous version much simpler in some cases. Having the interfaces in .Core doesn't necessitate that domain objects use them; in fact, my rule of thumb is for all domain objects to avoid the use of repositories unless an exceptive case exists.
- Replaced all uses of "using BetterAppServices.Core.DataInterfaces;" to "using BetterAppServices.ApplicationServices.DataInterfaces;" to support the above mentioned change.
- Replaced all uses of IRepositoryEntityName with IEntityNameRepository. Be default, all entities now have an explicit repository. This avoids the need to manually change from the generic to the explicit, when the need arose, in unit tests and in constructors; which was frequently occurring.
/BetterAppServices.Core/QueryDtos (/BetterAppServices.Dtos)
- Added a Dtos class library to provide an appropriate location for View Models and DTOs. While this project currently has a dependency on BetterAppServices.Core, so that the View Models can contain references to domain objects, this dependency could be severed for better separation between the view and the domain. The caveat is that much more work would be required maintaining a more complete DTO layer and transferring data via DTOs in all of the CRUD pages. Decided that a separate assembly was overkill. Consequently, added this namespace to contain query DTOs for transferring results of "report" queries into objects. With this example project, only the listing page uses "pure" DTOs for much better performance. The other CRUD pages still communicate directly with domain objects to keep things much simpler. This is something that can be argued either way and it should depend on the project needs to determine if a more separated approach is warranted.
- Added EntityNameDto.cs to act as a summary object to be bound to results from stored procedures (e.g., the entity listing pages), or other DTO needs.
/BetterAppServices.ApplicationServices/ViewModels

Added this namespace and EntityNameFormViewModel.cs to hold data related to adding and updating the EntityName. This object gets populated and passed to the entity form pages, accordingly.

/BetterAppServices.ApplicationServices
- Added reference to BetterAppServices.Dtos so that app services can return DTOs.
- Added EntityNameManagementService.cs. This app service class removes the CRUD logic from the controllers, makes the logic more reusable, and creates an appropriate class to add additional application service logic to.
- Added IEntityNameManagementService.cs. This app service interface makes unit testing more maintainable because you can mock the service interface and not worry about when the concrete class' constructor arguments change.
- Added /DataInterfaces/IEntityNameRepository.cs. As described above, every entity now has an explicit repository interface to avoid changes down the road when custom interfaces would inevitably be introduced.
/BetterAppServices.Data
- Added /BetterAppServices.Data/EntityNameRepository.cs to implement the associated interface. An important thing to note is that method provided invokes a stored procedure to act as a reporting means in line with command/query separation. This is useful for entity listing pages which would frequently, previously, end up loading a huge portion of the domain model to show summary information.
- Added /BetterAppServices.Data/NamedQuery/GetEntityNameSummaries.hbm.xml and set its compile action to "Embedded Resource." This provides the "short cut" for invoking the stored procedure. It could easily be modified to accept paging parameters.
- Added reference to BetterAppServices.Dtos so that repositories can return DTOs for better command/query separation.
/BetterAppServices.Web
- Added reference to BetterAppServices.Dtos to be able show DTOs on web pages.
- Changed ComponentRegistrar.AddCustomRepositoriesTo "BetterAppServices.Core" to "BetterAppServices.ApplicationServices" to reflect the new location of the data repository interfaces.
/BetterAppServices.Web.Controllers
- Added reference to BetterAppServices.Dtos.
/BetterAppServices.Tests
- Modified MappingIntegrationTests.CanGenerateDatabaseSchema to save DB schema to /db/Schema/CreateBetterAppServicesDb_ChangesWillBeLost.sql every time the unit test is run.

To reiterate, many of the changes above can be incorporated into a CRUD scaffolding generator; the focus with this example project is on providing an archtectural spike of the proposed architectural revisions.

Even if you don't use S#arp Archtiecture, this sample project should serve as a good example of using application services and basic use of command/query separation (CQS). Although the CQS in the sample project could be taken much further, I felt that the sample provides a good balance between practical maintainability and a more austere separation of concerns.

Enjoy!

Billy McCafferty

Message-Based Systems for Maintainable, Asynchronous Development

2010-03-02T04:22:00Z

Preface (you know it’s good if there's a preface)

In Architectural Paradigms of Robotic Control, a number of architectures were reviewed including deliberative, reactive, and hybrid architectures. Each of these exhibit a clean separation of concerns with layering and encapsulation of defined behaviors. When implemented, the various capabilities, such as planners and mobility controllers, are encapsulated into discrete components for better reusability and maintainability. A pivotal aspect not discussed in the previous article is how the various system layers and components communicate with each other, such as reporting sensor feedback and sending commands to actuator controllers. Effectively resolving this communication challenge is not only important to robotic systems but to many other industries and domains for the successful integration of disparate applications.

To give credit where credit is due, this article pulls quite heavily from the patterns, taxonomy, and best practices presented in Enterprise Integration Patterns (Hohpe, 2003). This well organized book is chock full of hard learned lessons and solid guidelines for developing maintainable message-based systems. This article should not be seen as an adequate replacement for that book (it’s more like cliff notes with a spackling of robotics bias); indeed, Enterprise Integration Patterns should have a prominent place on your bookshelf if you're developing message-based systems – so read this post and then browse http://www.eaipatterns.com/ while waiting for your copy to arrive to delve deeper.

A Need for Message-Based Systems

A few industries in particular, such as finance, healthcare and robotics, are demanding integration of an intimidating number of separate technologies that may be spread across computers, networks, and/or built upon a variety of technological platforms. Not only is this integration tricky, it can come with a significant cost to performance and maintainability if not implemented correctly. Accordingly, a solution is needed which facilitates loosely coupled integration while accommodating the performance demands of the task at hand. Taking a message-oriented approach to inter-application communications is one such way to accommodate these demands in a maintainable manner without sacrificing performance. This article gives an introduction to developing message-based systems using messaging middleware, describes taxonomy for discussing messaging topics and patterns, and includes a number of best practices.

Before delving further, it's important to clarify a few terms that will be used frequently:

Messaging Middleware (aka, a message bus): a 3rd party application which provides messaging infrastructure and capabilities (e.g., MSMQ, MS Concurrency and Coordination Runtime (CCR), Robot Operating System (ROS)),
Component: a stand-alone application or piece of executable code which communicates with the messaging middleware,
Message-based system: the entirety of the system including all integrated components and the messaging middleware.

As stated, messaging provides one means of facilitating inter-component communications. But as with any design approach, the project requirements must be carefully considered to determine if messaging is the appropriate mechanism for integration. While messaging is robust and facilitates integration, it also adds complexity and indirection. So before deciding to use messaging as the means of integration, consider all component integration options including (Hohpe, 2003):

File Transfer: wherein a component produces files of shared data which other components consume,
Shared Database: each component stores and retrieves data from a common database,
Remote Procedure Invocation: each component exposes specific procedures to be invoked remotely for exposing behavior and exchanging data,
Messaging: each component connects to a common messaging system, using messages to invoke behavior and exchange data.

Determining which integration approach is most suitable to your project's needs is beyond the scope of this article, focusing instead specifically on messaging. In turn, we'll review important elements of developing a message-based system including: message channels, messages, message routers, and message endpoints.

Message Channels

When a component sends information to another component in a message-based system, it adds the information to a message channel. The receiving component then retrieves the information from the message channel. Different channels are created for each kind of data to be carried; having a separate channel for each datatype better enables receiving components to know what kind of data will be retrieved from a given channel. For using a channel, each channel is addressable for sending and retrieving messages to/from them. How a channel is addressed varies depending on the messaging middleware being leveraged, but it's usually a port number or a unique string identifier. As a good practice for keeping channels organized, if string identifiers are available, a hierarchical naming convention may be employed to label channels by type and name; e.g., a channel carrying laser scans might be called "Perception/LaserScans."

There are two basic kinds of message channels:

Point-to-Point Channels (aka - client/server style): routes messages for components to talk directly with other components; e.g., a remote procedure call to another component. A message over a point-to-point channel only has a single receiver; so while the sender may not necessarily know who the receiver is, the sender can rest assured that the message will only be received by one receiver – it’s a FIFO queue.
Publish-Subscribe Channels (aka - broadcast): routes messages for components to publish data and an arbitrary number of components to subscribe to that data. A copy of the message is generated for each subscriber on the channel.

In addition to channels intended to carry information among components, it is a good practice to setup an invalid message channel that bad-formed or unreadable messages may be forwarded to for logging and to assist with debugging.

Messages

With a function call, a simple parameter or object reference may be passed and retrieved by the invoked method, sharing the same memory space. But when passing data between two processes with separate memory spaces, the data must be packaged into a "message" adhering to an agreed upon format which the receiver will be able to disassemble and understand. The sender of the message passes the message via a message channel. The receiver retrieves the message from the message channel and transforms the message into internal data structures appropriate for the task at hand.

A message is made up of two parts:

Header: describes the data being transmitted and details concerning the message itself; e.g., origin, timestamp information, message expiration (if content is time-sensitive), message identifier, correlation identifier, return address, etc., and
Body: the data content that the receiver is looking to use.

When sending a message, the sender intends for the message to be used, or responded to, in a particular way. The intention of the message may be described as being one of the following:

Command Message: invokes a procedure in another application,
Document Message: passes a set of data to another application,
Event Message: notifies another application of a change in state, and
Request-Reply: requests a reply from another application.

Event messages deserve a bit more discussion. In its simplest form, an event message would simply be informational, letting subscribers know that an event has occurred; e.g., a new laser scan is available. If subscribers would like details concerning the event, they would send a request-reply to the sender of the event to provide further details; e.g., the laser scan details. Alternatively, the event could be a document as an event to inform subscribers that an event has occurred along with the details of that event; e.g., a new laser scan is available with laser scan details included. The size of the event details and the rapidity in which the event occurs should be considered when deciding between publishing simple event messages and document messages as events.

Request-reply messages could also use a bit more describing. A request-reply is usually implemented as two point-to-point channels. The first channel delivers the request as a command message, while the second carries the reply back to the requestor as a document message. To keep the replying component more loosely coupled and reusable, the requestor should include a return address indicating the channel that the replier should use to publish the reply. After receiving the reply, a challenge for the requestor is to then correlate the reply to the original request. If the requestor is sending a number of requests in succession, it will likely be difficult to keep clear – if it matters – which request a reply is associated with. To resolve this, every request may include a unique message identifier that the replier would then include as a correlation identifier. (A message could have both a message Id and a correlation Id.) The requestor uses the correlation identifier to “jog its memory” concerning which request the response is for. But frequently, a request-reply is in context of a particular domain object, such as a terrain map or a bank transaction; but the correlation Id doesn’t include such information. To assist, the requestor can maintain a mapping (e.g., hashtable) between message Ids and relevant domain object Ids which are related to the original request. When the reply is received, the mapping may be used to load the appropriate domain objects and take further action, accordingly.

Obviously, it is important that the senders and receivers of a message system agree upon the format that messages will take for clear interoperability, better reusability of components, and extensibility of the system. Consequently, a canonical data model should be well defined that all applications will adhere to. The canonical data model does not dictate how each application's domain model must be structured, only how each application must format data within messages. Message translators are developed to convert the sending application's domain model into the canonical data model before sending a message; receivers of messages then use their own message translators to translate the message into their own domain. This mechanism allows applications built on completely different technologies (e.g., C#, Lisp, and C++) to communicate with each other and exchange data. Many off the shelf messaging systems define their canonical data model which must be adhered to. For example, the Robot Operating System (ROS), which we'll looking at in more detail in subsequent posts, defines their canonical model at http://www.ros.org/wiki/msg. But the canonical data model need not be limited to defining the types of primitives available and how to include them in messages.

Domain-specific canonical data models may augment message formatting rules, adding semantic meaning to the data within a message. For example, the Joint Architecture For Unmanned Systems (JAUS) is a set of message guidelines for the domain of unmanned systems, such as autonomous vehicles. The JAUS guidelines provide domain specific rules for communicating data, such as propulsion and braking commands, sensor events, pose and location information, etc. To demonstrate, JAUS message types include (Siciliano, 2008):

Command: initiate mode changes or actions,
Query: used to solicit information from a component,
Inform: response to a query,
Event set up: passes parameters to set up an event, and
Event notification: sent when the event happens.

A challenge in dealing with canonical data models is how to handle changes to the model. In order to support backwards compatibility of existing components when the canonical data model changes, new message channels could be created to carry the messages adhering to the model; e.g., "Perception/LaserScans_V1" and "Perception/LaserScans_V2." Alternatively, the existing channels could continue to be leveraged to carry messages adhering to different version of the canonical data model. To do so, the message, within its header, would include a format indicator, such as a version number or format document (e.g., DTD) reference. But if a sender knows that receivers of a particular message are mixed in what format is being used, a component would need to send two messages, one for each version of the canonical data model. Certainly, this is an important consideration when deciding which components should (or even can) be upgraded to newer formats, and in what order.

Message Routers

While the heavy lifting of the message routing is handled by the messaging middleware itself, there are times when it is useful to augment the middleware with custom message routers to support unique scenarios.

Suppose the destination of a message may change based on the number of messages that have already passed over a channel. In this scenario, the sender of a message may not know how many messages have been passed over a channel since other senders may have been publishing messages on the same channel. Consequently, a message router may subscribe to the channel to determine where each message should be forwarded to, based on the described business rules. Once the destination is determined, the router would then place the message on a subsequent channel to be delivered to the appropriate destination. This intermediary routing is described as predictive routing as the message router is aware of every possible destination and the rules for routing, accordingly. If the routing is based on content within the message itself, such as threshold values, then the custom router is known as a content-based router.

A drawback to using message routers is that if the routing rules change frequently, the message router will need to be modified just as often. To help remedy this, if the rules are expected to change frequently, configurable routing rules (e.g., via XML) could be employed to enable easier management of routing rules.

Let's now consider another scenario wherein it's left up to the subscribers to determine which messages they're interested in; i.e., subscribers will be responsible for filtering out the messages they're uninterested in. In this reactive routing scenario, each subscriber would provide a respective message filter which is similar to a message router, but simply forwards, or does not forward, a message onto a subsequent channel that the destination subscriber is listening to. Frequently, message filters decide to forward, or not forward, based on content in the message itself; e.g., only forwarding orders that have a coupon included. While being similar to a message router in basic functionality, a message filter only has one possible channel to forward the message onto.

Deciding between predictive and reactive filtering must take into account a number of considerations. Is the message content sensitive? Do you need to minimize network traffic? Do you need to be able to add and remove subscribers easily? Is the predictive router becoming a bottleneck of message dissemination? For further guidance on selecting among routing options, see (Hohpe, 2003), ppg. 241-242.

Message Endpoints

Each messaging middleware option (e.g., CCR, ROS) has unique requirements for communicating with it. Each has its own API, its own means of addressing channels, and its own rules for packaging messages. Ideally, the components of the system should not be aware of the specifics of communicating with the messaging middleware. Furthermore, while unlikely to occur, the middleware should be able to be replaced with another, requiring little, if any, changes to the components. Accordingly, the components must be loosely coupled to the messaging middleware. Message endpoints provide the bridge between the domain of each component and the API of the messaging middleware.

Message endpoints are similar in nature to repositories when communicating with a database. Repositories encapsulate the code required to communicate with a database to store and retrieve data while being able to convert information from the database into domain objects. If the database changes, or if the mechanism for database communication changes (e.g., ADO.NET to NHibernate), then, ideally, only the repositories are affected. The rest of the application knows little about database communications outside of the repository interfaces. Likewise, components of a message-based system should not be aware of messaging details outside of the message endpoint interfaces, which provide the means to send and receive data. The message endpoint accepts a command or data, converts it into a message, and publishes it onto the correct channel. Additionally, the message endpoint receives messages from a channel, converts the content into the domain of the component, and passes the domain objects to the component for further action. Internally, the message endpoint implements a message mapper to convert between the component domain objects and the canonical data model.

While posting to a channel is rather straight forward, a message endpoint may receive a message by acting as a:

Polling Consumer: wherein the receiver looks at a channel on a regular basis for new messages and/or as soon as it completes the processing of a previous message. This frees the receiver from having to deal with messages as soon as they arrive on the channel in favor of dealing with messages went it’s ready and willing. A consideration to keep in mind is that messages may queue up on a channel while waiting to be retrieved by the polling consumer. Additionally, a polling consumer may take up threads and resources while polling a channel, even if the channel is empty.
Event-Driven Consumer: wherein the message is given to the receiver as soon as it arrives on the channel. The benefits to this include avoiding messages queuing up while being able to process messages asynchronously. But the receiver is no longer in control of the timing in which it processes messages and must handle messages as soon as they arrive on a channel.

Many messaging middleware solutions include support for transactions that the message endpoints may leverage, making the endpoints transactional clients. To illustrate the need for this, suppose the receiver of a request-reply command crashes just moments after the command message is consumed and removed from the channel. When it recovers, the command message is lost and the sender will never receive a reply. Using a transaction, the command message is not removed from the channel until the response is completed and sent. Committing the transaction removes the command message from the channel and adds the reply document message to the reply channel.

There are a few recommendations which should be considered when developing message endpoints. In accordance with the SRP, message endpoints should be able to receive messages or send messages, but not both in the same message endpoint. Furthermore, a message endpoint should only communicate with one message channel. If a component needs to send a message on separate channels, it would leverage multiple message endpoints to do so. If your components are developed in line with DDD, it would be each component's application services layer which would communicate with the message endpoints, preferably via their interfaces instead of concrete instances. This facilitates swapping out the message endpoints with mock objects for unit testing. Leveraging separated interfaces and dependency injection helps enable this approach. While these guidelines introduce more objects and indirection, they are proven practices for increasing maintainability of the component and reusability of the message endpoints.

Performance of Message Based Systems

One would likely be quick to assume that developing a message-based system has a huge cost to performance. Domain objects are converted to messages, messages are passed over channels, routers and filters intercept and forward messages, and messages are converted back into domain objects...this sounds like a heck of a lot going on. But because each component executes in its own thread or process, they need not wait for other components to complete their job before being able to move on to handling another message or task.

In the figure above (adapted from Hohpe, 2003, pg. 73), note that a sequential process requires that each message life cycle is completed in full before moving on to the next message. But in an asynchronous, message-based system, each component can move onto a subsequent message just as soon as it has completed its part in the last one. This greatly compensates for the extra overhead imparted by the messaging infrastructure.

With that said, there are some component responsibilities which take a long time to complete and may impede the speed by which messages are processed. For example, imagine a component which takes images from a web cam and extracts information such as human figures or road signs. This is likely a time consuming process and would impact the turn around time in which the component could process subsequent messages. To alleviate this bottleneck, multiple instances of the same component may subscribe to the same point-to-point channel. Recall that a point-to-point channel ensures that each message only has a single receiver. The instances of the component then become competing consumers of each message. So if one instance of the component is still processing an earlier message, another instance can grab the next message that arrives for concurrent processing. Power in numbers!

Monitoring and Debugging

Due to the widely asynchronous nature of message-based systems, attention must be given to monitoring and debugging techniques to observe system behavior and iron out problems.

Logging message content is an invaluable measure towards getting a clear look at what communications are taking place. To facilitate this, logging could be added directly into sending and receiving components, but logging message content should not be a concern of components; accordingly, using a monitoring utility to log such information is a cleaner separation of concerns. Certainly a benefit of publish-subscribe channels is that a monitoring component may subscribe to all messages and log the information to a file or console window. While it’s just as valuable to monitor messages on point-to-point channels, if a monitor were to consume a message over a point-to-point channel, the message would be noted as consumed and would no longer be available to the intended receiver. Because such monitoring capability is so helpful in developing and debugging, many messaging middleware options include a "peek" option which allows a monitoring utility to review message contents on a point-to-point channel without actually consuming the message. This capability should be taken into consideration when comparing messaging middleware alternatives.

In addition to monitoring message content sent over channels, it’s also assistive to monitor active subscriptions to various channels to accurately determine which components are receiving which messages. This capability is typically built into messaging middleware solutions and is very assistive during development.

It’s possible that even if a message is routed appropriately, the receiving component may not know what to do with the message due to an incorrectly formatted header or body content. Invalid messages such as this should be forwarded to an invalid message channel which an error logging utility would monitor and log, accordingly. An invalid message channel is setup just like any other channel but with the intention of exposing such messages for debugging purposes.

Closing Thoughts

Developing asynchronous, message-based systems requires a paradigm shift from the more traditional, synchronously executed applications that most people are familiar with. It is not appropriate for every application domain and should be seen as one additional architectural option to consider when developing applications. But in some domains, such as robotics or in the integration of disparate applications, this approach to development is absolutely pivotal in providing responsive behavior without sacrificing maintainability of the overall system. Indeed, by splitting responsibilities into discrete components, loosely coupled to each other via messaging middleware, immensely complex problems can be broken down into understandable chunks while being flexible enough to accommodate changes to the underlying middleware or introduction of new components.

In the next couple of posts, we'll look at a checklist for developing message-based systems followed with examples in CCR and ROS.

Enjoy!

Billy McCafferty
http://wiki.sharparchitecture.net

P.S., for S#arp fans, we're trying to get another release together for the coming week...and this ain't no quarterly release. ;)

References

Hohpe, G., Woolf, B. 2003. Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions.

Siciliano, B., Khatib, O. 2008. Springer Handbook of Robotics.

Architectural Paradigms of Robotic Control

2010-02-03T22:39:00Z

Introducing a New Blog Series

Rarely do I make new year's resolution. But I found myself pinky swearing myself a single resolution for this year: write a series of tractable posts concerning the development of software for robotics. Accordingly, I intend to introduce background, techniques, and examples with respect to the following topics and more:

Software Architectural Approaches for Robotics (e.g., Deliberative, Reactive, Hybrid)
Software Design Methodologies (e.g., for Behavior Based Robotics, for Hierarchical)
Integration Patterns (Message Channels, Publish/Subscriber, RPC)
Machine Learning and Adaptation Techniques (Neural Nets, Genetic Algorithms, Fuzzy Logic)
Planning Algorithms (Discrete Planning, Continuous Spaces, Motion Planning)
Simultaneous Localization and Mapping Techniques (Kalman Filters, Particle Filters)
Tools & Frameworks (Microsoft Robotics Studio, Robot Operating System)

I'm writing this series of posts for two reasons. The first, as always, is to bounce ideas off readers as a sanity check. The second is to instill interest in readers in this rapidly evolving field. The sophistication of software algorithms within robotics is beginning to lag behind the potential capabilities of available hardware. Accordingly, if this series acts as motivational tool to get others involved in tackling the current algorithmic challenges involved in robotics, then all the better.

To make this series more interesting and applicable to the development audience that reads http://devlicio.us, I will attempt, when appropriate, to focus on patterns and general techniques that could possibly be applicable to scenarios outside of robotics. Additionally, due to this series' focus on software, I'll concentrate on leveraging simulators over physical, often expensive, robotics hardware to make it easier for readers to duplicate provided examples.

Enjoy!

Introduction to Architectural Paradigms of Robotic Control

While "architecture" is likely one of the least definable terms in software development, it is unavoidably a topic which has one of the greatest impacts on the extensibility and maintainability of an application. Indeed, a frequent cause of an application re-write is due to the architectural decisions that were made early in the project, or the lack thereof. Certainly one of the difficulties in instituting proper architectural practices in a project lies in the fact that architectural decisions must carefully be decided at many different levels of project development. Obviously, the architectural decisions at each level should be made in context of the project requirements in order to facilitate a proper balance of speed of development, scalability of development, and maintainability of the application in the future.

To illustrate, consider a basic eCommerce site vs. a corporate financial management application which integrates with various third party tools. There are two aspects of architecture which must be carefully considered. The first is deciding which project contexts will require architectural consideration. The second is defining the architectural approach to meet the needs of the given project context, and implementing any necessary prefactoring, accordingly.

For a basic eCommerce site, the project contexts requiring architectural consideration would likely include: appropriate separation between presentation and control, separation between control and the domain, separation between the domain and data access, and appropriate integration with a payment gateway. Perhaps the active record pattern might be chosen as the data access mechanism. If inversion of control seems overkill for such a small application, perhaps direct communications to the payment gateway via the domain objects might be chosen as the means of integration. Accordingly, judicious prefactoring would suggest that an architectural spike be created demonstrating appropriate use of the active record pattern with a selected tool along with an example of how and where integration with the payment gateway would take place.

For the more complex and demanding needs of the financial management application, architecture considerations, in addition to those taken into account for the basic eCommerce site, might include: what integration patterns might be leveraged to facilitate client integration, messaging patterns that might facilitate server side integration with third party financial calculators, and where inversion of control is appropriate. Perhaps RESTful services would be included to provide integration support for clients requiring such capabilities. Messaging via a publish/subscribe mechanism using a composed message processor might be selected as the ideal means for coordinating data with third party vendors on the server side. Again, proper refactoring, from an architectural perspective would include developing architectural spikes along with the foundational pieces of such an application to provide appropriate guidance for developers assisting with the project.

Developing software for robotics is no different, in this respect, than developing any other application. One must decide where the key architectural decisions need to be made and then provide adequate decisions and guidance to facilitate development in accordance with those decisions. As stated, these decisions must be made in consideration of the project needs. This post focuses on one project context, the architectural context of determining the overall approach to robotic motor control. We'll delve into three major architectural approaches to motor control along with highlighting a few specific implementations.

Deliberative vs. Reactive Robotics

Certainly the best source of intelligent systems to study and emulate are those found in nature. From the humble ant to the exalted human, evolution has honed a variety of strategies for dealing with the physical world. Accordingly, if we are to create intelligent systems, a good place to start is by emulating the behaviors and responses to stimuli demonstrated by living creatures.

In the early days of robotics, it was presumed that the most effective approach to emulating intelligence was by taking in detailed measurements provided by sensors, creating an internal representation of the world (e.g., grid-based maps), making plans based on that internal representation, and sending pre-planned commands to actuators (devices which convert energy into motion) for moving and interacting with the world. Inevitable, this approach presumes that the internal representation of the world is highly accurate, that the sensor reading are precise, and that there is enough time to carry out a plan before the world changes. In other words, this approach is highly deliberative. Obviously, the world is highly dynamic, sensor readings are sometimes erroneous, and time is sometimes of the essence when interacting with the world. E.g., you probably wouldn't want to spend much time creating an internal representation of the world if there's a Mac truck speeding towards you.

At the other extreme from deliberative is a reactive approach to interacting with the world. Taking a purely reactive approach does away with plans and internal representations of the world altogether; instead, a reactive approach focuses on using the world as its own model and reacts, accordingly. Instead of plans, reactive approaches often rely upon finite state machines to modify behavior as the world changes. Rodney Brooks changed robotics thinking upside down when he introduced this paradigm shift in the 80s. The robots he and his team produced were much faster in their reaction times, when compared to deliberative robots, and exhibited surprisingly complex behavior, appearing quite intelligent in many scenarios. But as is any extreme, a purely reactive approach to the world had it's own drawbacks; its difficulty in managing complex scenarios which demand careful planning is one such example.

The figure at right illustrates some of the differences between deliberative and reactive systems. Adapted from (Arkin, 1998).

While there are certainly pros and cons to either approach, there are some scenarios which are appropriate to one or the other. While still others may require more of a hybrid approach. Let's now take a look at each approach in more detail.

Deliberative/Hierarchical Style

One of the first successful architectural implementations on a mobile robotic platform was the well known robot known as Shakey, created at Stanford University in the 1960s. Of particular note was the robot's architecture which was made up of three predominant layers: a sensing layer, a planning layer, and an execution layer. Accordingly, as information would be made available by sensors, Shakey would make plans on how to react to the perceived world and send those plans on to the execution layer for low level control of actuators. This began the architectural paradigm known as sense-plan-act (SPA). As the SPA approach matured, additional layers were introduced in a hierarchical style, typically all of which having access to a global world model. The upper layers in the hierarchy would use the world model to make plans reaching into the future. After plan development completed at the highest levels, the plans would be broken down into smaller commands and passed to lower levels for execution. Lower layers would then further decompose the commands into more actionable tasks which would ultimately be passed on to actuators at the lowest level. In a hierarchical approach, the sensing layers participate in keeping the internal representation of the world updated. Furthermore, to better accommodate dynamic changes to the world environment, lower planning layers may suggest changes to plans based on recent changes to the world model.

The diagram at right roughly illustrates this hierarchical architectural approach. As should be noted, the sensing layers update the world model while a hierarchy of planning and execution layers formulate plans and breaks those plans down into actionable tasks. Adapted from (Kortenkamp, 1998).

4D/RCS

The current pinnacle of hierarchical architectures may be found in the reference architecture known as 4D/RCS (4 Dimensional / Real-time Control System). 4D/RCS is the latest version of the RCS reference model architecture for intelligent systems that has been evolving for decades. With 4D/RCS, six (more or less) layers are defined for creating and decomposing plans into low level action:

Unit/Vehicle/Mission Layer: this layer decomposes the overall plan into actions for individual modules. Plans are regenerated every 2 hours to account for changes to the world model.
Module Layer: converts actions into tasks to be performed on specific objects and schedules the tasks, accordingly. Plans are regenerated every 10 minutes.
Task Layer: converts actions on a single object into sequences of moves to carry out the action. Plans are regenerated every minute.
Elemental Layer: creates plans for each move, avoiding obstacles and collisions. Plans are regenerated every 5 seconds.
Primitive Layer: determines motion primitives (speed, trajectory) to facilitate smooth movement. Plans are regenerated every 0.5 second.
Servo Layer: sends servo control commands (velocity, force) to actuators. Plans are regenerated every 0.05 seconds.

At the heart of each and every layer are one or more 4D/RCS nodes which contain a behavior generation process which accepts commands from the next higher level and issues subgoals to the behavior generation process at lower levels. Furthermore, each node reports its task execution status up the chain for consideration into further planning. While plans are being solidified and disseminated, a sensory processing process in each node receives sensory input from lower levels, updates the world model, and collates the sensory data into larger units which it passes on to the next higher level; e.g., points get converted to lines which get converted to surfaces which get converted to objects with each successive rise through the 4D/RCS layers A visual summary of a 4D/RCS node quickly shows just how complex such a system can become. But sometimes the demands of a task require a respective amount of sophistication in the architectural approach. A comprehensive review of the 4D/RCS approach is discussed in (Albus, 2006).

The primary advantage to deliberative, hierarchical approaches such as 4D/RCS is that competent plans for managing complex scenarios can be generated and broken down into smaller chunks for lower levels to execute. But an obvious disadvantage to this comes in the form of much added complexity to the overall system while penalizing the speed at which the system can react to a changing environment.

Reactive Style

In the 1980s, Rodney Brooks, in an effort to overcome some of the limitations of sense-plan-act that Shakey and other such robots exhibited, introduced the concept of reactive control. "Simply put, reactive control is a technique for tightly coupling perception and action, typically in the context of motor behaviors, to produce timely robotic response in dynamic and unstructured worlds." (Arkin, 1998). In other words, by eliminating the reliance on maintaining an internal world model and avoiding large amounts of time generating plans, simple responses can be executed in reaction to specific stimuli, thus exhibiting behavior similar to that of living organisms. If there was any doubt in what Brooks was trying to imply, he boldly titled one of his works, Planning is Just a Way of Avoiding Figuring Out What to do Next. This paradigm shift away from planning turned sense-plan-act into a simpler sense-act.

Accordingly, in a sense-act paradigm, the primary focus is in carefully defining behaviors and the environment stimulus which should invoke those behaviors. This focus is deeply rooted in the idea of behaviorism; from a behaviorism perspective, behavior is defined in terms of stimulus and response by observing what an organism does. This approach of developing robotics around such behaviors, unsurprisingly enough, is commonly referred to as Behavior Based Robotics. By leveraging simple state machines, which we'll examine below, to define which behaviors are active for a given stimuli, the overall architectural complexity is greatly reduced while giving rise to responsive, seemingly intelligent behaviors. (The question of whether or not the robot is truly intelligent and/or self-aware is a debate that I'll leave to others such as Searle and Dennett.) While the reactive approach took the spotlight for a number of years, and is still very appropriate in some cases, its limitations in managing complex scenarios and performing sophisticated planning indicated that this approach was not the end all panacea for all situations.

The diagram at right clarifies that reactive systems remove planning and deal with sensory input within the context of each behavior. I.e., "Behavior 1" has no knowledge of the feedback coming from "Sensor 3." Adapted from (Kortenkamp, 1998).

A Design Methodology for Reactive Systems

Due to the simplistic nature of reactive systems, the corresponding design methodology for developing such systems is rather straight-forward (Kortenkamp, 1998):

Choose the tasks to be performed. E.g., find kitchen.
Break down the tasks in terms of specific motor-behaviors necessary to accomplish each task. E.g., find wall, follow wall, recognize kitchen.
For each action, find a condition under which each action should be taken. E.g., wall found.
For each condition, find a set of perceptual measurements that are sufficient to determine the truth of the condition. E.g., long straight line connected to obstacle.
For each measurement, design a custom sensory process to compute it. E.g., bumper sensor activated, edge length over threshold found.

When implemented, we find that the defined behaviors can be realized as states within a finite state machine and that found conditions act as the mechanism for changing from one behavior state to another. What's missing is an arbitration mechanism to determine which behavior wins out in light of competing conditions. Let's briefly look at implementation approaches to reactive systems along with how such arbitration is achieved.

Subsumption & Motor Schemas

A key concern with reactive, behavior based robotics is determining which behavior should take precedence if conditions exist to activate two or more behaviors. An arbitration mechanism needs to be introduced to resolve such competing situation. Brooks dealt with this issue by proposing an architecture known as subsumption. Simply enough, a subsumption architecture still uses a finite state machine to codify behaviors and transitions, but introduces behavioral layers which can provide input to other layers and override behaviors of lower layers. The advantage is in facilitating more sophisticated behaviors as a sum of "lesser" behaviors. It's better understood by reviewing the following example from (Arkin, 1998):

In the example, there are three behavioral layers: Avoid-Objects, Explore, and Back-Out-of-Tight-Situations. The Avoid-Objects Layer's responsibility is to avoid objects by moving away from any perceived obstacles. The Explore Layer's responsibility is to cover large areas of space in the absence of obstacles. The highest level assists the robot in getting out of tight spaces if the lower layers are unable to do so. Each discrete behavior can invoke transitions to other behaviors and/or provide input or advice to subsequent behaviors based on perceived conditions. The "trick" of subsumption is that higher levels can suppress commands between lower level behaviors; consequently, higher layers are able to handle more complex scenarios by manipulating lower level behaviors in addition to its own. To illustrate suppression, note, in the above example that the "Reverse" behavior in the Back-Out-of-Tight-Situations Layer suppresses any commands that the "Forward" behavior is sending to the actuators. By doing so, complex behaviors may emerge using a number of basic behaviors and a relatively simple architectural approach.

A major challenge in using a subsumption architecture is deciding the appropriate hierarchy of behavioral layers. In more complex scenarios, it quickly gets sticky deciding which layer should have the right to suppress which other layers. (Otherwise known as spaghetti-prone.) Additionally, since the layers have bi-directional dependencies amongst each other, in order to provide input and suppression, changes to layers can have large impacts on other layers, often resulting in shotgun surgery with any change. Ronald Arkin introduced a subsequent architecture to address these, and other concerns, with an approach known as Motor-Schema based control which does away with arbitrating competing behaviors. Each active behavior in a Motor-Schema based control system calculates a vector to be carried out by an actuator (e.g., wheels or arm). Using vector addition, a final vector for each actuator is computed and sent for execution. With this approach, the output of behaviors is combined instead of arbitrated. This avoids the need to determine suppression hierarchies and makes a more extensible application, within the limits of behavior based robotics.

Hybrid Style

Most modern architectural approaches to robotics control attempt to combine the planning capabilities of deliberative systems with the responsiveness of reactive systems. Appropriately enough, this is referred to as a hybrid style. While the deliberative approach takes a sense-plan-act perspective and the reactive approach follows with plan-act, a hybrid approach typically takes the form of plan, sense-act. So while the sense-act layers carry out behaviors, the deliberative planning layer can observe the progress of reactive behaviors and suggest direction based on reasoning, planning and problem-solving. The diagram at right crudely demonstrates this combination of the two approaches. Adapted from (Kortenkamp, 1998).

Three-Layer Architecture (3T)

James Firby's thesis proposing Reactive Action Packages (RAPs) (Firby, 1989) provided a solid approach for integrating deliberative planning with reactive behaviors in the form of a three layer architecture. A multitude of subsequent architectures have emulated a similar approach, coming to be known as 3T architecture. From Eran Gat's essay Three Layer Architecture (Kortenkamp, 1998):

Three-layer architectures organize algorithms according to whether they contain no state, contain state reflecting memories about the past, or contain state reflecting predictions about the future. Stateless sensor-based algorithms inhabit the control component. Algorithms that contain memory about the past inhabit the sequencer. Algorithms that make predictions about the future inhabit the deliberator. ... In 3T the components are called the skill layer, the sequencing layer, and the planning [deliberative] layer.

As implied by being a hybrid approach, this 3T architecture is not mutually exclusive to behavior driven robotics. Indeed, the skill layer itself is made up of unique behaviors which resemble that of reactive systems. The sequencing layer then uses the world model to determine when a change in behavior is required. The planning layer can then monitor the lower layers and perform more deliberative, time consuming processes such as path planning. The primary variation amongst 3T implementations lies in the decision between whether the planning layer "runs the show" or if the sequencing layer takes command and invokes the planning layer only when needed. Atlantis is one such example that leaves primary control to the sequencing layer to invoke the planning layer. Other examples of 3T implementations include Bonasso's Integrating Reaction Plans and Layered Competences through Synchronous Control and Standford's Junior which took 2nd place in Darpa's Urban Challenge.

Wrapping Up

The intention of this article has been to provide an introductory overview of paradigms of robotic control including deliberative/hierarchical systems, reactive systems, and hybrid systems. While each approach is appropriate in select contexts, a hybrid architecture is very adaptable for accommodating a large variety of robotic control scenarios with sufficient planning and reactive capabilities. The interested reader is encouraged to dig further into the references and links provided to learn more about these various approaches and design methodologies for implementation.

In the next post, we will examine messaging strategies to facilitate the communications amongst the layers and components of robotic systems.

References

Albus, J., Madhavan, R., Messina, E. 2006. Intelligent Vehicle Systems: A 4D/RCS Approach.

Arkin, R. 1998. Behavior Based Robotics.

Firby, J. 1989. Adaptive Execution in Complex Dynamic Worlds.

Kortenkamp, D., Bonasso, R., Murphy, R. 1998. Artificial Intelligence and Mobile Robots.

Real World S#arp Architecture

2010-02-03T21:04:00Z

With any framework, technology, tutorial, book, idea, <insert pedagogical sources here>, it's often difficult to figure how the technique should be used in the real world. Sure, things sound great when the scope of your project deals cleanly with Customers, Orders and Order Items; and sure, auto-binding works great in a nicely controlled scenarios...but how should things work in the "real world." No, not the Real World in which Brooke has a complete meltdown in season 18. I'm talking about the real world in which many scenarios aren't clean cut in determining where the controller should end and the application services layer should take over; the world in which you're trying to bind from unique data collection mechanisms to appease odd requests from the client; the world in which even the GoF would retort with "heh, that's a tricky one." Yes, we're faced with these kinds of dilemmas on a frequent basis. What I find to be truly helpful is to look at full blown, real-world applications. While a real-world application is never perfect, you can often find great gems for dealing with tricky situations and using patterns and complex scenarios. Howard van Rooijen has released just such a real-world application demonstrating the use of S#arp Architecture, AutoMapper, ELMAH, Spark View Engine, and other great tools.

In Howard's own words... "A few months ago I wrote an email to the community about a site we had just launched – http://fancydressoutfitters.co.uk that used S#arp Architecture at its core along with a whole myriad of other Open Source Frameworks and Tools (Spark, AutoMapper, PostSharp, xVal...).

In the run up to the festive period, myself and two of the development team – Jonathan George & James Broome, decided that in the spirit of giving, we wanted to gift something back to the communities that gave so much to us throughout the year; so we decided to build a new sample web application to showcase the use of these various frameworks & tools called “Who Can Help Me?” which is based on the same architectural style as http://fancydressoutfitters.co.uk.

Who Can Help Me? started out as a small web application I built a few years ago to solve a small and specific business problem within our consulting organisation (and to test out .NET 3.5, LINQ to SQL, ASP.NET WebForms & MS AJAX!). The problem was, that as the organisation grew and new members of staff started, they found it difficult to find the right people who could help them solve specific problems they’d encounter in their consulting gigs. As I have worked for the organisation for a long time (>9 years) I generally knew everyone, had worked with most of them and knew what their areas of expertise were, thus I’d get a few calls every day asking “Do you know anyone who knows about X that could help me?”. The solution was to create a searchable skills matrix that would allow people within an organisation find other people who had specific skills or expertise who could help them solve a particular problem.

So Jonathan, James & I decided to re-write the Who Can Help Me? from scratch, using the architecture style, frameworks and tools we used to build http://fancydressoutfitters.co.uk - it might seem like we've massively over-complicated the architecture for such a simple application - but we really wanted this to demonstrate some of the concepts & techniques we used to build a full scale, public facing enterprise web application.

Who Can Help Me? utilises the following:

The project is currently hosted at Codeplex: http://whocanhelpme.codeplex.com/ and we’ve also released a live demo: http://who-can-help.me. We’ve added some documentation on the Codeplex homepage and will continue to refine this and augment it with blog posts covering some topics in more depth – so if you’re interested – please keep an eye on the following blogs / twitter:

http://howard.vanrooijen.co.uk/blog/ | @HowardvRooijen
http://jonathangeorge.co.uk/ | @jon_george1
http://jamesbroo.me/ | @broomej

Thanks very much for this contribution Howard! Besides myself, I'm sure others have found this project helpful and will continue to do so in the future.

Enjoy!

Billy McCafferty

Your First S#arp Project in 15 Minutes

2010-02-01T23:11:00Z

The most daunting task when trying out or learning a new framework or technology is figuring out where the heck to start. Take S#arp Architecture for instance (arguably, the world's most insanely amazing framework ever written by mankind...more or less). As incredibly cool as S#arp is, it can be intimidating trying to figure out where to start. To make that first step a bit easier, the following will get you up and running with your first S#arp project in about 15 minutes or less. (Steps verified on WinXP Pro.)

Here's what you'll have in less than 15 minutes:

A new S#arp Architecture project generated using a VS template,
CRUD pages for managing product categories for our crazy cool store,
live communications with the application database,
and unit tests verifying that everything is working.

Ready?

Download S#arp Architecture and Generate Your Project

Download and unzip the latest release at http://cloud.github.com/downloads/codai/Sharp-Architecture/SharpArchitecture_1.0_2009_Q3.zip. We'll call the unzip location <ROOT>.
Close all running instances of VS 2008.
Install <ROOT>\tools\T4 Toolbox 9.5.20.1.msi. (You'll need to uninstall any existing version before doing so (not compatible with latest T4 Toolbox...yet.)
Install <ROOT>\tools\NUnit-2.5.2.9222.msi. (It's likely compatible with later versions of NUnit as well.)
Copy <ROOT>\VisualStudioTemplate\SharpArchApplicationTemplate.zip to "C:\Documents and Settings\<YOUR USERNAME>\My Documents\Visual Studio 2008\Templates\ProjectTemplates\Visual C#\Web" (You'll need to create the \Web folder. Also, be sure to copy the zip, don't unzip it!)
Copy <ROOT>\VisualStudioTemplate\SharpArchApplicationWizard.dll to "C:\Program Files\Microsoft Visual Studio 9.0\Common7\IDE"
Open VS 2008 and select File / New Project / Visual C# / Web / S#arp Architecture Application. For the name, enter "MyCoolStore" (no spaces!), set the location to whatever you'd like (e.g., "C:\MyStuff\Projects\"), and keep "Create directory for solution" unchecked. Then click OK. (It'll take about a minute to generate the solution...when the status changes back to "Ready" in the very bottom left of VS, you're good to go.)
Build the solution (should build without a problem).

Hook Up to a Database

Run NUnit and select File / Open Project, to then open <ROOT>/MyCoolStore/tests/MyCoolStore.Tests/bin/Debug/MyCoolStore.Tests.dll
Run the tests and see the Data tests fail due to a DB connection timeout. Time to create the database!
Create a new SQL Server database called MyCoolStore. (Other DBs are supported.)
In VS 2008, open MyCoolStore.Web/NHibernate.config and modify the following bits in the DB connection string: YOUR_SERVER, YOUR_USERNAME and YOUR_PASSWORD. Save the file.
In NUnit, run the tests to see all the tests pass...if all is green, your project is talking to the database. (If the tests under Data still fail, recheck your DB connection string.)
In VS 2008, right click the MyCoolStore.Web project and "Set as Startup Project" and click F5 after doing so to run the project. (Not much to see here yet beyond the landing page...let's add some CRUD functionality.)

Generate CRUD Scaffolding

After verifying that the web project is running, stop the debugger.
In VS 2008, open Code Generation/CrudScaffolding/ScaffoldingGeneratorCommand.tt and do the following:
- Change line 19 to:
```
new EntityScaffoldingDetails("ProductCategory");
```
- Delete the four additions to entityScaffoldingDetails, lines 25-36, and replace them with:
```
entityScaffoldingDetails.EntityProperties.Add(
   new EntityProperty("Name", "string", 
      "Seafood", "[NotNull, NotEmpty]", true)
);
```
- Uncomment the 2nd to last line (i.e., remove the "//" in front of generator.Run()) and save the file...
- The CRUD scaffolding generator will run for a few seconds...it'll be done when the cursor no longer looks like an hour glass. (Yes, that's annoying that it runs when the file is saved...that's T4.)
- Recomment out the 2nd to last line (i.e., add the "//" in front of generator.Run() back in) and save the file. (This avoids premature generation - always embarrassing - when editing this file in the future.)
In VS, rebuild the solution.
In NUnit, run all of the tests...only CanConfirmDatabaseMatchesMappings should fail. Time to add the ProductCategories table!

Create the Database Table

In NUnit, run just Tests/MyCoolStore/Data/NHibernateMaps/MappingIntegrationTests/CanGenerateDatabaseSchema (which should pass), and change the NUnit tab to "Text Output."
Copy all of the SQL that you see on the Text Output tab in NUnit and run it against the MyCoolStore database using SQL Management Studio.
In NUnit, run all of the tests again to see them pass.
In VS, click F5 to run the project and change the URL, after it loads, to http://localhost:<DEBUG PORT>/ProductCategories. You can now CRUD product categories to your heart's content!

Enforce Object Uniqueness

Let's say you now want to enforce name uniqueness...
- In VS, open MyCoolStore.Core/ProductCategory.cs and add an attribute as:
```
[HasUniqueDomainSignature(
   Message="Provide a unique name dag nabbit!")]
public class ProductCategory : Entity
{
   ...
```
- Build the solution, F5 into debug mode, and try to add two product categories with the same name...go ahead, try it, I triple dog dare you.

Wanna learn more? Go to http://wiki.sharparchitecture.net/

And for all you S#arp junkies out there...the next quarterly release is coming soon...and it's going to have proper utilization of the application services layer during CRUD generation!

Billy McCafferty

Offsetting Your Bad Code Footprint

2009-11-23T21:01:00Z

Admit it, we've all written bad code now and again. Unfortunately, your bad code over time can leave a lasting footprint on the technosphere. If you're beginning to feel guilty about your bad code footprint, do know that you have options! While reducing your bad code footprint by writing better code is a logical approach; there are some that feel doing so will adversely affect timelines and budgets (regardless of quality). But even then, you can still offset your bad code footprint by purchasing bad code offsets at http://codeoffsets.com/. (A thanks to Roy Bradley for pointing me to this guilt reduction technique.) "The Bad Code Offset provides a convenient and rational approach for balancing out the bad code we all have created at one time or another throughout our lifetime—even when we can’t go back and fix it directly.

Denominated in Source Lines of Code (SLOC), every purchase will offset the desired quantity of SLOC and pave the way toward future code excellence. Money raised through the purchase of Bad Code Offsets supports the various Open Source Initiatives that are performing vital work towards the salvation of our future code base.

Every purchase of an offset today will take us one line of code closer to our goal of universal code excellence. Banish the bugs in your past as you help usher in a new age of software development."

See, you're only one PayPal transaction away from reducing your bad code footprint!

Billy McCafferty
http://wiki.sharparchitecture.net

S#arp Architecture 1.0 2009 Q3 with NHibernate 2.1.1 GA Released

2009-11-13T01:08:00Z

I'm pleased to announce that S#arp Architecture 1.0 2009 Q3 has been released at http://github.com/codai/Sharp-Architecture/downloads. This isn't a major upgrade, just an incremental, quarterly release to help developers work with the latest, stable S#arp dependencies. Along with updated documentation (at http://wiki.sharparchitecture.net/) and migration guidance (in /VersionHistory.txt), this release has a number of incremental improvements to developing your S#arp Architecture project, including:

Upgraded dependencies including NHibernate 2.1.1 GA, Fluent NHibernate 1.0 and NHibernate LINQ 1.0
New Many-to-Many convention added to VS project template (thanks John Ledger!)
New DB schema generation (as a unit test) added to VS project template (thanks indomitablehef!)
VS generated project now uses the HiLo algorithm for identity generation
S#arp Architecture is now HornGet compatible (http://preview.tinyurl.com/ydoolwh)
S#arp Architecture has moved to GitHub since the last release (http://github.com/codai/Sharp-Architecture)
The wiki documentation has gotten much more complete (http://wiki.sharparchitecture.net)
A contrib project, led by Tom Cabanski, is well underway (http://wiki.sharparchitecture.net/SharpArchContrib.ashx)
WCF support within S#arp Architecture has been deprecated...it's moving to Sharp Contrib

If you're new to S#arp Architecture and want to learn more:

What the heck is S#arp Architecture?
http://devlicio.us/blogs/billy_mccafferty/archive/2008/04/21/asp-net-mvc-best-practices-with-nhibernate-and-spring-net.aspx
Give me something meatier to read!
http://www.infoq.com/articles/Sharp-Arch-Billy-McCafferty
But I don't have a second more than 10 minutes to even read about it!
http://www.dimecasts.net/Casts/CastDetails/75 (a bit dated but a useful intro)
Well what are other people saying about this thing? (I read "doesn't suck" somewhere...that's got to count for something!)
http://www.google.com/search?hl=en&q=s%23arp+architecture
You don't say...well where can I join in on the fun?

http://wiki.sharparchitecture.net/ and http://groups.google.com/group/sharp-architecture.

There are also plenty of other resources out there such as Virtual ALT.NET videos, DimeCasts and lots more referenced on the wiki.

Enjoy!

Billy McCafferty

Dependency Injection 101

2009-11-09T22:26:00Z

(It looks like Kevin Pang and I must have seen the same vision this weekend or had a close encounter flashback as he's concurrently putting together a great series on Dependency Injection for Dummies...be sure to check out his posts for another take on this subject!)

Simply put, dependency injection is the technique of supplying an object with its service dependency(s) from an external source.

If you’re unfamiliar with dependency injection or just starting down the road of object oriented programming, the above statement might have the same effect as a mathematics book when the author puts forth a new equation that is about to be analyzed. Symptoms may include glassy eyes, nausea, and headaches. But when taken apart, we find that the idea of dependency injection is a much simpler concept than its grandiose title implies, and is a technique that many developers use commonly, quite naturally in many cases, without even knowing it.

Encapsulation

The place to start, in understanding the motivations and implementation of dependency injection, is with the basic idea of encapsulation. Crudely explained, encapsulation is the art of determining which bits of logic belong together and putting them into discrete objects. This allows the details of the logic to be modified within the encapsulated object without necessarily impacting the objects which use it. Furthermore, encapsulation provides a neat and clean way to package up reusable code to be used by many other objects without introducing code duplication.

From a real-world perspective, encapsulation is a very useful technique when building a self-aware robot to take over the world. When building any self-aware robot, you need to start with the basics, such as mobility and sensory perception. For example, a robot might have a stereo camera for observing the visible world, sonar sensors for detecting nearby objects around it, and touch sensors to know if it has made contact with the object it’s trying to pick up. Just by describing it, we’re already starting to imply areas of logic that could be nicely encapsulated: visual perception, sonar object detection, touch feedback, and mobility.

Assume for a moment that we’ll bypass the practice of encapsulation and throw all of the code for these various modules directly into the primary method which uses them, a benign method we’ll call TakeOverTheWorld, living in a .NET assembly called CommandAndControl.dll. The code might look something like the following:

public class WorldDomination {
   public void TakeOverTheWorld() {
      // Code to open USB connection to use stereo camera
      // Code to process camera images to identify humans

      // Code to open serial port to legs for moving
      // Code to send legs serial commands to move over to location of human

      // Code to generate speech to user; e.g., "All your leaders are belong to us"
   }
}

In the code, we see the TakeOverTheWorld method taking on a lot of responsibilities and given explicit knowledge about components which are subject to change when the robot components are upgraded. Some of the responsibilities which we’re burdening the method with include:

Knowing the stereo camera is connected via USB and how to open a connection to it;
Knowing how to process camera images to identify humans;
Knowing that the legs require communications to be sent via a specific serial port;
Knowing how to send low level communications to the legs as serial commands.

So what’s the problem you ask? For starters, the TakeOverTheWorld method would likely be a few thousand lines long and would be immensely difficult to maintain. Let’s suppose that we upgrade the camera to use a better stereo imaging device which now receives commands wirelessly. You’d have to carefully, tediously walk through all of the lines of TakeOverTheWorld to find any communications with the camera, upgrade the capabilities, and test, accordingly. But now testing is much more challenging as well. Since all of the other code, such as the mobility code, will also run whenever we invoke TakeOverTheWorld, then we’ll need to make sure that the rest of the robot is in perfect working order while we switch out one component. In other words, it’s simply not possible now to test the various components in isolation of each other. This approach quickly turns into spaghetti code, becoming a maintenance nightmare, difficult to debug, and exhibiting decreasing code quality as time progresses. What started out as a simple project to develop a robot to conquer the world is turning out to be about as effective as C-3PO…after being blasted by a storm trooper that is.

The first step that we can take, to make our robot for world domination a little easier to understand and maintain, is to encapsulate the various modules into their own, respective classes. Let’s look at the TakeOverTheWorld code again with a sprinkling of better encapsulation by moving the camera related responsibilities to a class called StereoCamera.cs and the mobility related responsibilities to MobilePlatform.cs. Furthermore, let’s move these two classes to a new class library assembly called Services.dll.

// Within the Services package (or class library)

public class StereoCamera {
   public void Connect() {
      // Code to open USB connection to use stereo camera
   }

   public MeaslyHuman IdentifyHumanInFieldOfVision() {
      // Code to process camera images to identify humans
   }
}

public class MobilePlatform {
   public void Connect() {
      // Code to open serial port to legs for moving
   }

   public void MoveTo(Location location) {
      // Code to send legs serial commands to move over to location 
   }
}

// Within the CommandAndControl package

public class WorldDomination {
   public void TakeOverTheWorld() {
      StereoCamera camera = new StereoCamera();
      camera.Open();
      MeaslyHuman human = camera.IdentifyHumanInFieldOfVision();

      MobilePlatform legs = new MobilePlatform();
      legs.Connect();
      legs.MoveTo(human.Location);

      // Code to generate speech to user; e.g., “All your leaders are belong to us”
   }
}

As demonstrated, we introduced two new classes, StereoCamera and MobilePlatform, and moved the related responsibilities to those classes. We also introduced a new package as a new class library, called Services.dll, to contain the two new classes. The package diagram now looks as the figure shown to the right.

This basic refactoring has resulted in a number of benefits:

There are far fewer lines of code in the TakeOverTheWorld method and easier to digest for whomever may be looking at it next;
The TakeOverTheWorld method is now much easier to understand the overall flow of control without getting bogged down in low level details;
The TakeOverTheWorld method now knows nothing about the details related to connecting to the camera via the USB connection (other than simply calling Connect), nothing about the mathematics behind processing camera images to identify humans, nor the details of opening a serial port and sending low level serial commands to move from point A to point B; and
The StereoCamera and MobilePlatform classes are now quite reusable from a variety of other objects that will surely need to use them.

Overall, this encapsulation refactoring has led to a design that is easier to maintain and will likely reflect in the general level of maintainability and quality. So things are better, but we still have much to improve. I suppose we’re beginning to resemble C-3PO, but only after Chewbacca put his head on backwards. (OK, so C-3PO didn’t have diabolical plans for world dominion, but let’s ignore that subtlety for now.) As we proceed, we’ll examine what other areas we’ll want to improve and learn a bit more about the definition of dependency injection along the way.

Services

Earlier, we defined dependency injection as the technique of supplying an object with its service dependency(s)… What exactly is a service dependency? A service, in the context of dependency injection, is any object which provides access to an external resource. In the refactoring exercise we performed on our code, and as implied by the package diagram, two services were introduced. The first provided access to the stereo camera; the second provided access to the legs for moving the robot around. In your own project, services may include web services, data access objects, file manipulation utilities, third party APIs, and many others.

Because services depend on external resources (e.g., a physical component or database), they are inherently more complicated to test, require that the external resource be available when we test our code, and are subject to change when the underlying resource being managed changes (e.g., when the component is upgraded). When considering these arguments, we want to remain loosely coupled to the services themselves. To contrast, the refactored code above, even after applying encapsulation, is tightly coupled to the underlying services.

To illustrate the idea of tight coupling, the TakeOverTheWorld method is creating each service directly with the new keyword; i.e., the method is creating a concrete instance of the service class. Not only does the method have a concrete dependency on the service class itself, this inherently implies that the application layer that the WorldDomination class resides in must have a direct dependency on the layer which the StereoCamera service class resides in. This layer dependency is shown as a directional line in the above package diagram from the CommandAndControl layer to the Services layer.

In addition to having a concrete dependency on the service itself, the TakeOverTheWorld method must also know to call Open for the respective services (and presumably Close); i.e., the TakeOverTheWorld method needs to know how to create and initialize the services before it’s even able to use them. These are further responsibilities which this method should not be concerned with.

Ideally, the TakeOverTheWorld method would be loosely coupled to the underlying services and wouldn’t have to know about such low level details of the service itself outside of calling methods such as IdentifyHumanInFieldOfVision.

Coding to Interface

The first step we can take towards making TakeOverTheWorld less tightly coupled to the service classes is to have the TakeOverTheWorld method code to the interfaces of the services rather than to the concrete classes themselves. One might argue against introducing two new interfaces for the objects; as we add new objects, we inherently introduce additional complexity to the overall system. Also, with interfaces, not only are we introducing a small amount of additional complexity to the overall system, but we’re also introducing indirection. In other words, in order for someone to fully understand what’s going on at runtime, the developer will need to determine which concrete object is being used under the interface. It shouldn’t be difficult to do so, but it does add time to understanding the overall system. These are fair arguments and ones not to be ignored. Whenever we are faced with a decision that will lead to a better design at the expense of added complexity, we carefully need to consider if the benefits of refactoring outweigh the penalties of added complexity. In this case, coding to interface (and another refactoring we will do next) will lead to a more maintainable system, provide a means to readily unit test the various components in isolation, and will result in a more loosely coupled design.

Accordingly, we simply need to add two interfaces to the Services package, have the service classes implement them, and change the references in the TakeOverTheWorld method to invoke the interfaces rather than the classes themselves. But as we’ll see, this next refactoring still leaves us tightly coupled to the service classes. Let’s look at the code.

// Within the Services package

public interface IStereoCamera {
   void Connect();
   MeaslyHuman IdentifyHumanInFieldOfVision();
}

public interface IMobilePlatform {
   void Connect();
   void MoveTo(Location location);
}

public class StereoCamera : IStereoCamera {
   public void Connect() {
      // Code to open USB connection to use stereo camera
   }

   public MeaslyHuman IdentifyHumanInFieldOfVision() {
      // Code to process camera images to identify humans
   }
}

public class MobilePlatform : IMobilePlatform {
   public void Connect() {
      // Code to open serial port to legs for moving
   }

   public void MoveTo(Location location) {
      // Code to send legs serial commands to move over to location 
   }
}

// Within the CommandAndControl package

public class WorldDomination {
   public WorldDomination() {
      camera = new StereoCamera();
      legs = new MobilePlatform();
   }

   public void TakeOverTheWorld() {
      camera.Open();
      MeaslyHuman human = camera.IdentifyHumanInFieldOfVision();

      legs.Connect();
      legs.MoveTo(human.Location);

      // Code to generate speech to user; e.g., “All your leaders are belong to us”
   }

   private readonly IStereoCamera camera;
   private readonly IMobilePlatform legs;
}

In the code above, note that the two interfaces were introduced and now the constructor of the WorldDomination class now handles creating the service instances and setting private members, represented as interfaces, to the service instances. The package diagram reflecting this refactoring is shown at right.

Even though the TakeOverTheWorld method is now communicating with the interfaces of the services, the class containing is still tightly coupled to the service classes due to the local instantiation of the classes within its constructor. To further alleviate this coupling, we’re going to have to move a step “up” and introduce a package which will be responsible for creating the service dependencies and injecting them into the WorldDomination class. Yes, we’re finally ready to see dependency injection in action.

Service Initialization and Injection

While the previous refactorings have taken us closer towards a more loosely coupled design, we still need to remove the object creation from the WorldDomination class altogether. By introducing a new package which will contain the robot system’s “main” program, we can leverage that layer to also initialize the service dependencies and inject them into the WorldDomination class. Let’s look at the code of the new package, called Robot, along with modifications to the WorldDomination class. (The Services package, and the interfaces and classes therein, will remain unchanged.) But before looking at the code, study the package diagram at right to see where the classes live and take particular note of the direction of package dependencies. There’s one dependency that’s still problematic, which we’ll discuss after the code.

// Within the CommandAndControl package

public class WorldDomination {
   public WorldDomination(IStereoCamera camera, IMobilePlatform legs) {
      this.camera = camera;
      this.legs = legs;
   }

   public void TakeOverTheWorld() {
      // Same as before
   }

   private readonly IStereoCamera camera;
   private readonly IMobilePlatform legs;
}

// Within the new Robot package

public class Program {
   // This is the “main” method of the entire robot application.
   // I.e., it’s the method that gets called to kick off the application.
   public void Main() {
      IStereoCamera camera = new StereoCamera();
      IMobilePlatform legs = new MobilePlatform();

      WorldDomination worldDomination = new WorldDomination(camera, legs);

      worldDomination.TakeOverTheWorld();
   }
}

In the code above, the Main method initializes the services and “injects” them into the constructor of the WorldDomination class. As you can see, the injection of service dependencies into a class is really nothing more than passing the service instances as arguments to the class instantiation. So when it comes right down to it, the intimidating phrase “dependency injection” is really quite simple in practice. But there’s still a problem in our implementation that could lead to design “rot” over time. The CommandAndControl package still has a dependency on the Services package. This is bad because during later phases of the project, without a high level of discipline, developers may forego injecting the service dependencies into the WorldDomination class, or other classes in the CommandAndControl package and simply instantiate the service classes directly using the new keyword. This would make all the work we’ve done to this point to support dependency injection a moot point. To enforce our design, we need to introduce a new pattern known as dependency inversion, also known as separated interface.

Separated Interface

To enforce our design, and to eliminate the package dependency from CommandAndControl to Services, we will move the service interfaces to the CommandAndControl package and reverse, or invert, the dependency direction between the two packages. Accordingly, the refactoring steps to do this include:

Copy the service interfaces from the Services package to the CommandAndControl package.
Change the interface references in the WorldDomination class to refer to the interfaces now found within the CommandAndControl package.
Remove the package dependency from CommandAndControl to Services. Now add a package dependency from Services to CommandAndControl. (In C#, this would be a matter of deleting the assembly reference from the CommandAndControl class library and adding a new assembly reference from the Services class library.)
Modify the service classes, StereoCamera and MobilePlatform, to implement the interfaces now found in the CommandAndControl package.
Modify the Main method, in the Main class to use the interfaces found within the CommandAndControl class.
Delete the obsolete interfaces from the Services package.

With the separated interface pattern implemented, the CommandAndControl package is only aware of the service interfaces without having any dependency, whatsoever, on the service implementations. As performed in the previous refactoring, the CommandAndControl package, the WorldDomination class specifically, is being given its service dependencies via constructor injection via initialization code found in the Robot’s main method.

With this, we have implemented a basic example, including all the required pieces, for performing dependency injection. It introduced a number of new pieces to the overall application, but our design is now loosely coupled to the underlying services and enables us easily swap out services with new implementations, without having to recompile the CommandAndControl package, and provides the ability for test doubles, such as mocks, stubs, and fakes, to be injected into the WorldDomination class when unit testing. (Unit testing with test doubles is beyond the scope of this article, but I encourage you to read http://xunitpatterns.com/Test%20Double.html to begin learning more about this subject.)

Dependency Injection / Inversion of Control (IoC) Containers

There’s one final topic which is appropriate to discuss when describing dependency injection: IoC containers. (Dependency injection is also known as “Inversion of Control,” as originally described by Robert Martin. Interestingly enough, the development community typically uses the phrase “dependency injection” when talking about the technique itself and “IoC” when talking about dependency injection containers. But rest assured, there is no difference whatsoever between “Dependency Injection” and “Inversion of Control (IoC).”) An IoC Container is a utility which is leveraged to initialize and inject dependencies into the objects that use them. While it is a bit more difficult to visualize than the “manual” dependency injection we performed in the previous refactoring step, an IoC Container is nothing more than an object factory which initializes an object and satisfies its service dependencies in the process.

With most IoC Containers (e.g., Ninject, Castle Windsor, LinFu, and many others), there are only a couple of steps required to setup and use the IoC Container:

Register all of the classes which may be injected into another object. In our carefree efforts for world domination via our robot, this would include, at minimum, registering the service classes MobilePlatform and StereoCamera. While each IoC Container has its own means of registering a class for later retrieval, the underlying mechanism is often as simple as adding the service class’ object type to a hashtable for later lookup. If we were using Castle Windsor with our project, the registration lines would look like the following:
```
container.AddComponent("camera", 
   typeof(IStereoCamera), typeof(StereoCamera));
container.AddComponent("mobilePlatform", 
   typeof(IMobilePlatform), typeof(MobilePlatform));
container.AddComponent("camera", typeof(WorldDomination));
```
Note that the last line registers the WorldDomination class itself. In doing so, we can let the IoC Container take care of the work of instantiating it with its service dependencies already injected. We don’t have to hand off instantiation of this class to the IoC Container, but it provides a useful benefit. Suppose the WorldDomination class needs a third service dependency provided via its constructor somewhere down the road. If we aren’t using an IoC Container to manage its creation, we’d have to look for every place where the class is created, via the new keyword, and pass in the extra dependency. If we leave its creation to the IoC Container, then we simply add the new service interface parameter to its constructor and the IoC Container will take care of the rest, assuming you’ve registered the third dependency as well.
Invoke the IoC Container to instantiate a registered object with its dependencies already injected. For our project, the following demonstrates getting a handle to the WorldDomination class:
```
WorldDomination worldDomination = (WorldDomination) 
   ServiceLocator.Current.GetService(typeof(WorldDomination));
```
This line of code tells the IoC Container to return an instance of the requested class with its dependencies injected. This actually uses an IoC Container known as Common Service Locator. Since almost all IoC Containers do the same actions – registering and instantiating objects – the major IoC Container authors worked together to establish a common interface for all IoC Containers. In this way, you can register your objects with your favorite IoC Container and then register the IoC Container itself with the Common Service Locator. It’s enough to make your head spin, but leads to further decoupling to the IoC Container itself so that you can easily swap out one for another (e.g., Castle Windsor for Ninject) without modifying the code which depends on the IoC Container.

How you setup and use an IoC Container depends on the environment that you’re developing within. For example, if you’re developing an ASP.NET MVC application, you can use an IoC Container to instantiate the controllers, injecting their dependencies via the controller’s constructor. For example, the following two lines of code, within the Global.asax.cs file, instruct the ASP.NET MVC framework to use Castle Windsor to instantiate controllers:

IWindsorContainer container = new WindsorContainer();
ControllerBuilder.Current.SetControllerFactory(
   new WindsorControllerFactory(container));

The only thing left is to register the controller dependencies that they require for their constructors and the IoC Container takes care of the rest.

While this will be enough to gain a basic understanding of what an IoC Container is and how it is used, a bit more studying will be required to learn how to actually set up and use one. For more information about IoC Containers, including examples of setup and usage:

Common Service Locator: http://www.codeplex.com/CommonServiceLocator
Castle Windsor Tutorial: http://www.castleproject.org/container/gettingstarted/
LinFu in 5 Minutes: http://www.codeproject.com/KB/cs/LinFu_IOC.aspx
Using Ninject: http://ninject.codeplex.com/wikipage?title=user%20guide&ProjectName=ninject
StructureMap: http://structuremap.sourceforge.net
A classic IoC Overview: http://martinfowler.com/articles/injection.html
Using IoC in a project: http://wiki.sharparchitecture.net/

Finally, for a much more thorough analysis of many of the concepts described above, I highly recommend reading Robert Martin's Agile Principles, Patterns, and Practices in C#.

Final Thoughts

Regardless of whether you’re looking to take over the world with a dominion of evil robots or simply trying to build a better web app, dependency injection and the use of an IoC Container can lead to an easy to maintain application that is loosely coupled to service dependencies. These attributes naturally lead to better quality and happier developers. (Both of which I would put squarely into a category known as “good.”) But while these techniques provide many benefits, they also introduce a bit more complexity; especially when going from “manual” dependency injection, as we did in the coding example, to the use of an IoC Container. Accordingly, your project team should carefully consider the benefits and added complexity to determine if it's right for your project. While the use of an IoC Container would be a bit overkill for a very simple application, its use is obviously pivotal in creating self-aware robotic legions for conquering the world.

Billy McCafferty
http://wiki.sharparchitecture.net/

Speaking at Virtual ALT.NET Tonight

2009-11-04T17:07:00Z

If you're interested in...

Finding out what's new in S#arp Architecture Q3 2009,
Hearing about what the Contrib project has to offer,
Going into a couple of S#arp topics in detail, such as...
- Getting up and running quickly,
- Handling object associations,
- Using an Application Services layer and
- Practical implementation of Command/Query Separation (or "Domain/Reporting Separation" from my perspective).

Then join us tonight at Virtual ALT.NET, 9 - 11 PM Eastern time.

Billy McCafferty

The HornGet Project: Bringing "apt-get install" to .NET Projects

2009-10-30T02:34:00Z

For years I used nothing but Windows. When I started getting into robotics (for fun - let me know if you'd like me to do it professionally ;), I had to dive into the Linux world. After only a couple of weeks with Ubuntu, I was stunned at how far behind our Microsoft-following community is in some respects. Take Linux's "apt-get install" command, for example. Running this simple command, followed by an application name, finds the referenced application on the "Information Super Highway," downloads the binaries and installs it locally. No biggie, right? Well it also looks to see what dependencies the application has, downloads the latest of all of them, and installs them as well before installing the target application. It does all of that with just one command line (and sometimes even successfully! ;).

There are two scenarios that come to mind where something like "apt-get install" is tremendously useful when developing a project. The first, and most akin to the intent of "apt-get install" is when you're managing an open source project (e.g., S#arp Architecture) and want to provide an easy way for developers to get the latest of the project while concurrently grabbing the latest of its dependencies (e.g., Rhino tools, NHibernate, Fluent NHibernate, MVCConrib) in the process. The second scenario is if you have a project which has a large number of dependencies and dread checking for updates to the project dependencies. Which dependencies do you download? Where do you find the source? Which order should you build them in? How do you know, except through a lot of tedious snooping, which secondary dependencies are also involved in rebuilding the project dependencies?

The HornGet project is attempting (and succeeding I might add, even in its pre-beta form) to greatly ease the burden of updating dependencies of complex projects. With Horn installed and configured, it only takes a single command line to get the latest of your favorite OSS project or to rebuild your project's dependencies. Similarly to "apt-get install," you simply tell Horn what you'd like to have built and it does the rest. There's a full listing of all of Horn's supported packages found at http://code.google.com/p/hornget/wiki/SupportedPackages. As you can see, Horn supports a wide variety of projects such as NHibernate, Castle, LinFu, Ninject, AutoMapper, Subsonice, MVCContib, S#arp Architecture, and many others. Something quite unique about Horn is that it even checks for indirect, bi-directional library dependencies. For instance, NHibernate depends on a Castle project while one of the Castle projects depends on NHibernate. Horn is smart enough (through config files) to know which projects to build in which order to resolve such scenarios.

The best way to experience and appreciate Horn is to try it out. The following steps demonstrate using Horn to automatically download and build the latest of NHibernate with updated dependencies:

Install TortoiseSVN (if you haven't already) to download Horn and to be used during the build process.
Install msysgit as some of the dependencies will be downloaded from from Git distributed source repositories.
Make sure that "C:\Program Files\Git\bin" and "C:\Program Files\Git\cmd" are included in your Environment Variable's PATH (and add them if not)
Install Windows PowerShell (used during the dependency build process - it's included in Win 2008 automatically)
Open a PowerShell command prompt and run "set-ExecutionPolicy RemoteSigned" to allow Horn to invoke PowerShell commands used during the dependency build process.
Use SVN command line or TortoiseSVN to get latest of http://hornget.googlecode.com/svn/trunk/ to a folder we'll call HornRoot
(Here's the only kludgey part, which I'm sure will be cleverly resolved very soon...) Open /HornRoot/src/Horn.Console/Program.cs in Notepad, or whatever, and modify, under GetRootFolderPath(), the line "var ret = new DirectoryInfo(rootFolder);" to be "var ret = new DirectoryInfo(@"C:\...\HornRoot\" + PackageTree.RootPackageTreeName);" and save/close the file - replacing C:\...HornRoot with a real file path. NOTE: Make sure there are NO spaces in the file path that you provide. PowerShell abhors spaces in file paths.
Open a command prompt, cd to the /HornRoot/src folder and run "hornbuild.bat"...you be triumphantly rewarded with a BUILD SUCCEEDED message...but that's just the Horn project.
In the same command prompt, cd to /HornRoot/src/build/net-3.5/debug and run "horn"
- You should see the Horn usage guidelines...that's a good sign.
- At this point, Horn is now installed, ready and configured to build any of the supported packages.
As an example, run "horn -install:nhibernate" (and be patient as it'll take some to run...30 minutes to an hour or more for a project with many dependencies is possible probable)
- After executing this one liner, horn will spit out a whole bunch of "working......" statements as it figures out what NHibernate's dependencies are, downloads the source of the dependencies, builds them, and then downloads and builds NHibernate with all of the updated dependencies.
- Keep waiting...Horn is doing a LOT of tedious work for you right now. Don't be surprised if you see very long strings of "working......" statements at a time, throughout the build process; this is expected as it downloads each project dependency and builds them, accordingly. Go get another cup of coffee or something.
- Assuming the OSS world is in a Zen-like state with each other you'll get a number of BUILD SUCCEEDED messages for each project dependency and then eventually get a BUILD SUCCEEDED message for the target project itself.
- Horn outputs its results (the updated DLLs) into /HornRoot/.horn/result. Don't worry if there are more DLLs that you expected; just grab the updated ones that are applicable to your project and you're good to go.

If you've ever tried to update a bunch of project dependencies, you'll appreciate the enormous time savings in using Horn to pull together the dependencies for you. Doing this manually usually takes me at least a few hours to get latest of all dependencies, build them, and build them with S#arp Architecture. With Horn, it takes a one liner and a coffee break.

How does it do this magic you ask? Good question! Each project that Horn supports has a Boo configuration file; e.g., the S#arp Architecture Horn description is at http://hornget.googlecode.com/svn/trunk/package_tree/frameworks/sharp.architecture/sharp.architecture.boo. The file speaks for itself in describing what dependencies are necessary for the project and where to find the source of S#arp Architecture itself. So when you run "horn -install:sharp.architecture" (which takes a very long while mind you) Horn references the respective project description file and off it goes. It's really quite a remarkable feat that Horn has pulled off.

Horn is still maturing - it's not even beta yet - but it's quite a time saver already. So if you have ideas, would like to get involved, or have a project that you'd like to have added to the roster, be sure to let the Horn team know.

For more Horn:

Project home: http://code.google.com/p/hornget/
Discussion forum: http://groups.google.co.uk/group/horn-development

Enjoy!

Billy McCafferty

Where are we with P = NP?

2009-10-08T21:06:00Z

KurzweilAI.net led me to a terrific article concerning the current status of answering the question of whether or not P = NP. As the author, Lance Fortnow, aptly notes, "The P versus NP problem has gone from an interesting problem related to logic to perhaps the most fundamental and important mathematical question of our time, whose importance only grows as computers become more powerful and widespread."

The topic of P versus NP is certainly not a new one but one which is only vaguely familiar to many software developers. If nothing else, gaining a better appreciation for P versus NP problems can better prepare you for recognizing when the problem you are working on turns out to be NP-complete and arm you with a strong toolset for delivering an adequate solution. But taking the appreciation a bit further, the question of whether or not P = NP sheds an aesthetic beauty on the fact that our software development practices are so firmly embedded in the realm of mathematics...and that our code is only a shallow shell around that world. Any time spent gaining a better understanding of the underlying mathematics and algorithmic nature of programming will quickly enlighten you to seeing development in a completely different light.

Billy McCafferty

A S#arp Project Case Study

2009-10-08T18:55:00Z

Like Jimmy Bogard, I was pleased to read a post from Howard van Rooijen on the S#arp Architecture forums concerning the use of S#arp Architecture and Bogard's AutoMapper in the timely development of http://www .fancydressoutfitters.co.uk. With a big thanks to Howard, he has also taken the time to provide a number of great blog posts which walk the reader through his experiences, challenges, solutions, and lessons learned with respect to developing a commercial, greenfield application using ASP.NET MVC.

Hello S#arp Architecture Community,

I just wanted to let you know that a S#arp Architecture based e-commerce site has just been released into the wild: http://www.fancydressoutfitters.co.uk

For those who are interested - here's a little background:

- We ran the project using Scrum (managing the project with Scrum for Team System)

- Delivered in 20 weeks: 10 x 2 week iterations
- We extended the core S#arp Architecture framework (support for multiple databases, ViewModels etc...)
- Integrated other OSS projects into the solution:
  - N2CMS
  - Spark View Engine
  - xVal Validation Framework
  - AutoMapper (for entity / view model mapping)
  - PostSharp (for cross cutting concerns - logging, performance counters, caching)
  - MBUnit / Gallio for our BDD Framework
  - Solr & SolrNet for Search and Faceted Navigation
- Over 700 BDD Specs
- Solution performs very well - 1000 concurrent users per web server, generating around 180 pages per second across 2x single quad core 64bit servers.

If you're really interested two of the developers in the team (James Broome & Jon George who have both posted to this forum) & myself have blogged our thoughts & experiences developing this solution and working with S#arp Architecture:

I wanted to say a HUGE thank-you to everyone in this community for all the passion, energy, knowledge and help you provide on a daily basis.

S#arp Architecture is a wonderful framework that truly is greater than the sum of its parts.

/Howard

Thanks for sharing Howard and good luck with the venture! (There are other S#arp projects in the wild discussed in the S#arp forums...go take a look!)

Billy McCafferty

S#arp Architecture Contrib...it's alive!

2009-10-05T22:02:00Z

As if developing ASP.NET MVC applications with S#arp Architecture couldn't get any more flippin' exciting... I'm pleased to announce S#arp Architecture Contrib which promises to take S#arp Architecture into new territory, such as providing better support for WinForms, making available a number of optional add-ons (such as PostSharp enhancements), and encouraging further opinionation (is that a word?) of S#arp Architecture with various alternatives such as specific security providers and view engines, such as CRUD generation for Spark.

It's always been very important to me and the contributors of S#arp Architecture that the core framework remain tight and clean. The introduction of S#arp Architecture Contrib now opens the door to many more options for its use without compromising that vision. (To emphasize this, even the WCF modules in #Arch will be moved to the Contrib project to make the core of #Arch that much leaner.)

Tom Cabanski is leading this Contrib effort and has already put a great deal of effort into making both the code and related documentation available for your review. The source may be found at http://github.com/codai/Sharp-Architecture-Contrib with supporting documentation to be found at http://wiki.sharparchitecture.net/SharpArchContrib.ashx. Your feedback on the S#arp Architecture forums on its current direction is most welcome!

On a related note, I'll be speaking at the Nov. 4 Virtual ALT.NET meeting to discuss the first quarterly release of S#arp Architecture (coming soon!) along with a couple of specific topics related to developing S#arp projects. Appropriately enough, Tom Cabanski will be joining me to give an introductory overview of what's in the Contrib project and where it's headed.

Enjoy!

Billy McCafferty

Drawing the Line Between MVC Controllers and Application Services

2009-09-16T17:29:00Z

Frequently, when we adapt new technologies or techniques, the new fix ends up introducing new "yucky" stuff or simply pushes the previous yucky stuff to another area. Since "yucky" just might be a bit subjective here, I consider yucky stuff to include any bit of code that has no clear home or which is overly complicated for the sake of separtion of concerns. If you encounter code such as this, it doesn't necessarily mean that you have a weak understanding of the domain or task at hand, but it may be a sign that you have a grey area between application layers which needs to be better defined.

These grey areas are frequently found between where the presentation layer ends and the controller and/or business logic begins. For example, when we moved from Active Server Pages to ASP.NET, we liberated our views of business logic only to find our newly embraced code-behinds to be a bear to maintain as they intertwined abstracted presentation concerns with logic. Adding a pattern such as M V P certainly made it all testable but then moved much of the "yucky" stuff to either the presenter class or added new yucky stuff in the form of complicated event communications and/or fudgy controller logic between the code-behind and presenter classes. (If you've used MVP in a major project, you'll likely know what I'm talking about.) So now with the dawn of ASP.NET MVC with a DDD approach (nicely encapsulated within S#arp Architecture, of course), our concerns are over, right? Well, not exactly...we now have a new yucky bit to deal with: the grey area between the controllers and application services (not to be confused with domain services) . This is exactly the grey area that I've struggled with since adopting ASP.NET MVC with an eye for DDD. Where exactly do we draw the line between what goes into controllers and what goes into application services?

Joe Wilson recently posted a review on S#arp Architecture and had a useful critique of the framework which IMO provides a very clear explanation of where the line should be drawn between the MVC controller and DDD application service layers:

I've borrowed a metaphor from the SQL Server installer that talk about "surface area". I think of the MyProject.ApplicationServices layer as a wrapper around all the app logic. The methods in this layer are the surface area of my application, and they encapsulates all the implementation code my domain is dealing with. Calls into this layer can kick off multi-step processes, handle workflow, and usually have long, explicit method names. But this is the only part of my domain's surface area that is exposed externally.

To put it another way, if you're putting any logic into your controller class that you would have to copy and paste, almost verbatim, into, e.g., a WCF service class to replicate the same sort of behavior, but for a different client interface (e.g., WCF vs. MVC), then it's likely a sign that your controller is doing too much and that the code may instead belong in an application service. In a different light, if you're putting anything into the application service layer which would preclude a different client type (e.g., WCF vs. MVC) to leverage it, then you may be introducing controller concerns into the application services. Following these rules of thumb, your controllers remain more appropriately paired with the web tier of your application while your application services provide the true gateway to the rest of the application. Keeping a close watch on the application service layer's "surface area" (or more appropriately stated, bounded context), and making sure it doesn't leak into the controller layer or vice versa, is a good approach for drawing a firm line between these two layers and keeping the yucky stuff at bay.

Billy McCafferty
http://www.itsamuraischool.com