Billy McCafferty

Greenfield Development with ASP.NET MVC & S#arp Lite - Day 3

Billy McCafferty — Tue, 16 Oct 2012 22:10:00 GMT

A series of posts providing proven guidance for developing ASP.NET MVC applications from idea to well-designed implementation.

Day 3 – Define the Domain Design Model

Objective of the Day

Transform the actor/system interaction diagrams - from Day 1 - and the domain conceptual model - from Day 2 - into a design model, which will be used as the basis for implementation. The design model will include two artifacts:

Class Diagram: A class diagram extends the domain conceptual model to include data types, methods, and supporting classes necessary for implementation.
Sequence Diagrams: Sequence diagrams express the interactions among classes and the messages invoking, and responding to, those interactions, accordingly.

Inputs

Actor/System interaction diagrams
Domain conceptual model

Activities

Designing the domain conceptual model, and evolving that into the domain design model represents the transition from business-oriented to implementation-oriented requirements. While there are two outputs from this process, class diagrams and sequence diagrams, they are defined in parallel with each other, each contributing to the design of the other.

Transforming the conceptual model into a design model is most effectively done iteratively. This does not mean that each refinement need take a two-week iteration to complete. Indeed, each iterative refinement may only take 30 minutes or so to complete. Tackling this transformation iteratively allows you to learn from each pass, applying new knowledge to the class diagram and to the sequence diagrams in successive passes.

Furthermore, you need not tackle the entire domain model when defining the design model; only focus on the scope and level of detail that is immediately useful.

The steps involved with each elaboration iteration, for defining/refining the class diagram and sequence diagrams, are as follows:

Identify the bounded context which will be the focus of design model elaboration. (The bounded context need not be strictly adhered to, but should be identified to assist in focusing the elaboration effort.)
Select one or more high priority User Stories (or Use Cases) which are representative of the bounded context and, preferably, are of higher risk.
Refine (or initially define) the class diagram, focusing on classes, attributes, and associations within the context of the User Stories.
For each User Story, refine (or initially define) sequence diagrams illustrating the behavior and interactions of each class involved.
Update the class diagram to reflect any new or modified classes, behaviors, and/or attributes discovered during the design of sequence diagrams.

When defining sequence diagrams, do not feel constrained to only include classes which are already defined within the class diagram. The exercise of building out the sequence diagrams will quickly lead to the discovery of new classes and behaviors which are not yet a part of the class diagram; that’s one of the points of doing this. Furthermore, do not feel obliged to create a sequence diagram for every User Story or requirement. Sequence diagrams are time consuming to create and are quickly subject to change. Accordingly, I see sequence diagrams as disposable artifacts which need not be maintained after they have been implemented as code.

Sequence diagrams are particularly useful at the following times:

To illustrate the interactions involved with a complex User Story,
To illustrate a "boiler-plate activity," such as the interactions involved with the CRUD (create/read/update/delete) of a single domain object, in order to establish a standard practice and to serve as a useful reference for new team members, and
To tell a more junior developer exactly how you’d like something implemented.

In the final step, when updating the class diagram to reflect new or modified behaviors, use each message, passed between two classes within the sequence diagram, to indicate the need for a respective method on the class from which the message originates. In other words, if a communication is indicated between two classes in the sequence diagram, there should be a respective method (or return result from a message) in the class diagram.

Deciding to what granularity to create the sequence diagrams is a tricky decision. If too little time is spent, then the sequence diagram will provide very little guidance towards implementation. But too much time may result in granularity which is otherwise obvious or clearly assumed. As a rule, continue to break-down the sequence diagram to further levels of granularity until:

The effort is no longer reducing uncertainty and risk,
The increased granularity is no longer providing "new" information, or
The increased granularity is no longer useful to the task at hand (such as providing an initial estimate).

From the perspective of including this practice within an overall project methodology, defining/refining the design model should be roughly performed at the beginning of a project to assist with estimating and to improve understanding of the project. It should also be performed in more detail at the beginning of each development iteration to define/refine the design model for that iteration.

GRASP Patterns

The definitive challenge at this point is figuring out exactly how to transform the conceptual domain model into an appropriate design model describing classes with attributes, associations, and behavior. While sequence diagrams are a tool to visualize interactions among classes and their behaviors, the diagramming technique itself does not provide guidance on how to determine how the classes will interact and what their respective behaviors will be. Doing this correctly is a critical element to ensuring the codebase is well-designed and maintainable.

GRASP patterns provide guidance for assigning responsibility and interactions to objects; GRASP is an acronym for General Responsibility Assignment Software Patterns and were defined to provide guidance for OOA&D. While there are nine GRASP patterns in all, five key GRASP patterns should be mastered and considered continuously in the creation of the design model. What follows are the five patterns, each including the problem that arises when adding behavior to a model, and the guiding principle/solution for meeting that challenge. (Reference Applying UML and Patterns by Craig Larman, from which the following patterns are quoted, for a very well written discussion of these principles and other fundamentals of OOA&D.)

Information Expert

Problem: What is a general principle of assigning responsibilities to objects?

Solution: Assign a responsibility to the information expert - the class that has the information necessary to fulfill the responsibility.

Creator

Problem: Who should be responsible for creating a new instance of some class?

Solution: Assign class B the responsibility to create an instance of class A if one or more the following is true:

B aggregates A objects.
B contains A objects.
B records instances of A objects.
B closely uses A objects.
B has the initializing data that will be passed to A when it is created (thus B is an Information Export with respect to creating A).

An exception to this is that if the logic required to create class A is of considerable complexity, consider moving the creation logic to a helper class, called a Factory [GoF 1995].

Low Coupling

Problem: How to support low dependency, low change impact, and increased reuse? Coupling is how strongly one class is connected to, or has knowledge of, other classes. Highly coupled classes suffer from the following problems:

Changes in related classes force local changes.
Harder to understand in isolation.
Harder to reuse because its use requires the additional presence of the classes on which it is dependent.

Solution: Assign a responsibility so that coupling remains low.

This terse solution is not very helpful; indeed, it’s often simpler to instill low coupling by watching for signs of tight coupling and to refactor, accordingly. Martin Fowler’s Refactoring is by far the best resource available for becoming familiar with these "smells" along with concrete guidance for taking corrective action.

High Cohesion

Problem: How to keep complexity manageable? Cohesion is how strongly related the responsibilities of a class are. Classes with low cohesion suffer from the following problems:

Hard to comprehend.
Hard to reuse.
Hard to maintain.
Delicate and constantly effected by change.

Solution: Assign a responsibility so that cohesion remains high.

Like the solution for Low Coupling, this isn’t immediately useful advice. The point is to aim towards encapsulating related logic together, and to introduce new classes as necessary to continue to do so. Again, Fowler's Refactoring is a tremendous resource for this topic.

Controller

Problem: Who should be responsible for handling an input system event?

Solution: Assign the responsibility for receiving or handling a system event message to a class representing one of the following choices:

Represents the overall system as a Facade [GoF 1995].
Represents a User Story or Use Case scenario within which the system event occurs, often named <UserStoryName>Handler, <UserStoryName>Coordinator, or <UserStoryName>Controller.

As an aside, this GRASP pattern is nicely enabled via ASP.NET MVC with an explicit controller layer. But as the naming of a controller would imply, the scope of each controller should be limited to that implied by its name, accordingly. For example, a controller by the name of "ManageUsersController" would imply performing User-related CRUD operations; User-related report generation should likely belong to a separate controller. Incidentally, this would also better support High Cohesion - what is good for the support of one pattern is often good for another.

CRC Cards

It is sometimes difficult to figure out the very first step for adding behavior and collaboration activities to objects. A technique which may assist with getting the ball rolling or to help when brainstorming the behavior of a new bounded context within a domain is a technique known as CRC cards (Class-Responsibility-Collaborator cards).

CRC cards are very simple yet effective for defining brainstorming the responsibilities of each object. To utilize, grab some index cards, title each with a class name of interest, and write two things on each:

Responsibilities: For each CRC card (i.e., each class), write one or a few bullet points describing the extent of responsibility for the class. Keep GRASP patterns in mind when defining the scope of responsibility.
Collaborator Objects: List one or more collaborator classes which need to be interacted with in order to carry out the responsibilities of that class.

After documenting responsibilities and identifying collaborator objects, the task of transforming this information into the class diagram, and into sequence diagrams, is greatly simplified.

Refactoring & Refactoring to Patterns

GRASP patterns are a tremendous guide for answering the question of "what belongs where." But as a system evolves, changes are inevitably introduced by client request, or more frequently, by further discovery of requirements. For example, what may at first appear to be simple may quickly become complex after coding begins. In these cases, refactoring techniques can greatly assist to answer the question of "how to adapt to change." In a nutshell, refactoring is the practice of improving the design of existing code. While the details of refactoring are beyond the scope of this article, two tremendous resources (i.e., absolutely required reading for any developer) are Refactoring by Martin Fowler and Refactoring to Patterns by Joshua Kerievsky.

Putting it Into Practice

This day’s "Putting it Into Practice" will consist of a number of example artifacts, including the class diagram (both initial and revised), CRC cards, and sequence diagrams. Let’s review each in turn.

By far, transforming the concept model into the design model is the one of the trickiest tasks and the most important to get as close to "right" as possible. The rule to keep in mind is to design the simplest model which could possibly work to fit the needs of the requirements and elaborate iteratively and only when necessary. (Keeping in mind the design elaboration steps discussed previously.)

Initial Domain Class Diagram

At some point, the concept model must be transformed into an implementable design model. This is the time to put our techie hats on and start to think in terms of classes, properties and methods. To get the ball rolling, let’s take a stab at designing a minimal class diagram to support the concept model discussed earlier.

Initially, we’ll focus on the bounded context of support ticket management, and the objects which participate with this activity. We’ll leave the reporting context until later on; this will also give Acme Telecom a little more time to make up their mind on what they want to see in the report.

What follows is the class diagram reflecting the first attempt at modeling the domain as an implementable design:

There are a number of differences from the conceptual model which were carefully considered:

The concepts Management, SupportStaff, and Admin have been combined into a single class called StaffMember. The following considerations were taken into account to make this decision:
- All of these concept objects are employees of Acme Telecom.
- Each of the concepts share duplicate data.
- While each may be allowed to do different tasks (e.g., manage staff vs. open support tickets), it does not yet appear that the classes themselves would behave differently. I.e., it’s not yet apparent that an Admin class would have different methods than a Manager class.
- Having the flags "IsAdmin" and "IsManager" may indicate that we may need role-management capabilities; while this may be true, we’ll want to avoid delving into the weeds further until warranted. (With that said, we’ll want to fully think out security management before we begin coding…it’s much more difficult to incorporate security management later on in project development.)
The concepts SupportTicket and SupportCall have been merged into a single class called SupportTicket. This could have been argued either way, but the two concepts are equivalent from a data workflow perspective. Consequently, I went for the simpler approach.
SupportTicket.Status has been pulled out as an enum value as we rarely expect the status types to change. Additionally, since the status may also be considered within data workflows, having it as an enum value will simplify the workflow conditionals and provide support for the use of a finite state machine, if deemed necessary down the road. (See Kerievsky's Refactoring to Patterns for guidance on when/how to make this decision.)
Preferably, SupportTicket.IssueType would also be an enum to keep things simple. But, the requirements state that the SupportStaff must be able to dynamically add new issue types, when needed, when opening a support ticket. Consequently, IssueType will need to be a persistable domain class.

Take special note that this model does not imply anything about infrastructural details such as persistence and security. These will be added to the class diagram as they are included in the sequence diagrams. Trying to anticipate all of the infrastructural classes at this time will likely lead to over-engineering (or wrong-engineering altogether).

With that said, the architect should have a good idea of how such infrastructural concerns will be addressed in the final solution. It is appropriate to have at least discussed how the overall project will be architected and what tools will be used to support infrastructural concerns. The overall architecture will have a strong impact on the sequencing of messages through the application tiers.

For the task at hand, we’ll take into account the fact that we’ll be using S#arp Lite as our underlying framework and architecture.

Accordingly, before going further, read the article "S#arp Lite: The Basics" to get a better understanding of how a S#arp Lite project is architected: http://devlicio.us/blogs/billy_mccafferty/archive/2011/11/11/s-arp-lite-the-basicss.aspx . Pay particular attention to the package diagram, describing how the layers are organized.

Sequence Diagram for Opening a New Support Ticket

Did you read the introduction to S#arp Lite as referenced above? OK, great, we’re ready to continue. (You'll likely be lost, bewildered, confuddled and worse if you haven't read it...I hope you'll only experience one or two of those symptoms if you have read it.)

We have designed the beginnings of the class diagram. We’re now ready to elaborating, refining, and adding behavior to the class diagram by analyzing the sequence of events involved with opening a new support ticket. This particular user story was selected because it covers much of the bounded context that we’re currently working on and is a critical element of the CLogS application.

As we’ll see, the design of the sequence diagram is where we have to seriously start thinking about:

How will data be loaded from the database?
How will data be saved to the database?
How will messages travel among the Web, Controllers, Tasks, Domain, and Data layers of the project?
How will the Tasks layer expose the core API of the application to the Controllers layer?

While this is a daunting task, be patient and revise your first sequence diagram many times until you feel it’s clear and clean.

Let’s now consider a sequence diagram for the following user story:

Support Staff may open a new support ticket and provide details including: Customer, Issue Details (such as description and "dynamically managed" type), Status, and Resolution Details (if resolved immediately). Status of the call may be New, In Progress, or Resolved. ("Dynamically managed" means that if the issue type isn’t available, the Support Staff may enter a new one when entering the support ticket.)

To make things simpler, ignore any extraneous features in the first design of the sequence diagram to focus on the core of the requirements; e.g., we’ll ignore the dynamically managed issue types for now.

(Click to embiggen.)

The sequence diagram above describes the collaboration of objects in support of allowing a staff member to open a new support ticket.

While designing the above sequence diagram, four new objects were identified, which may be added to the class diagram:

ManageSupportTicketController: This MVC controller class will reside in the .Web layer of the project and will have the responsibility of handling requests for the management of a support ticket.
ManageSupportTicketTasks: This tasks class (aka "application service" class [Fowler 2002]) will reside in the .Tasks layer of the project and will have the responsibility of carrying out the coordination activities for the management of a support ticket.
SupportTicketViewModel: This view model DTO (data-transfer object) will carry information from the controller to the view and vice-versa for the purposes of allowing a StaffMember to add/update support ticket information.
IRepository<SupportTicket>: This interface will provide a means to communicate with the underlying data access mechanism to persist the new support ticket to the database.

In a nutshell, the sequence is as follows:

The StaffMember requests to open a new support ticket.
The StaffMember is presented with a form.
A sub-process facilitates the StaffMember to select the respective customer (elaborated below).
The StaffMember provides the requested information and submits the form.
The StaffMember is either:
- presented with the same form, with validation information, if the submitted information contained validation problems, or
- presented with a listing of open support tickets with a message stating that the support ticket was successfully saved.

We’ll now turn our attention to look at the sub-process for selecting a customer...

The sequence diagram above describes the sequence of events to allow a StaffMember to select a customer via an AJAX post-back on the page. The form will have a "Customer Account Number" input box on it; when it changes, an AJAX post-back will try to find the customer record. If the customer exists, the customer information will be dynamically shown; if it doesn’t, the StaffMember will need to provide a few customer details before submitting the form.

As implied, many details about a user story are discovered during the process of designing a sequence diagram. To ease organization of the requirements, the sequence diagrams and details may be simply printed out and stapled to an index card with the user story written on it, or uploaded as attachments to the user story in an online requirements management tool.

In both sequence diagrams, arguments may be made that all of the UML is not "by the book"; e.g., the use of stereotypes has been basterdized a bit to make the diagram more expressive. I bring this up to express the point that less time should be spent worrying about making a UML diagram perfect as should be spent conveying useful information. When everything’s said and done, these diagrams will likely be thrown away once they’ve been implemented as working code.

If the underlying project architecture is already established (e.g., we’re using S#arp Lite as the foundation), designing the sequence diagram is relatively straight-forward. If the intention is to grow the architecture "organically," then elaborating the sequence diagram may take a few iterations.

CRC Cards for "Ticket Resolution Report by Support Staff"

In our discussions thus far, we haven’t gotten into the details of exactly how the support ticket resolution report will be generated. So before tackling the respective sequence diagram, we’ll use CRC cards to brainstorm a bit about the classes to be involved with the user story. The CRC cards don’t need to be exhaustively detailed; just enough in order to move on to the associated sequence diagram. CRC cards don’t/shouldn’t need to be maintained throughout the project life-cycle; once they’ve served their usefulness, toss them (or archive them away to some hidden place if throwing them away saddens you).

As demonstrated, the CRC cards were used to work out the basics of the classes involved with generating a support ticket resolution report, per the requirements. Now that the general responsibilities have been defined, we’re able to confidently move on to designing the respective sequence diagram.

Sequence Diagram for Creating a "Ticket Resolution Report by Support Staff"

(Click to invigorate.)

The sequence diagram above describes the collaboration of objects in support of the creation of a support ticket resolution report for a specific staff member.

While designing the above sequence diagram, four new objects were identified, which may be added to the class diagram:

SupportTicketReportsController: This MVC controller class will reside in the .Web layer of the project and will have the responsibility of handling requests for the creation of support ticket related reports.
SupportTicketReportTasks: This tasks class will reside in the .Tasks layer of the project and will have the responsibility of carrying out the coordination activities for the creation of support ticket related reports.
IRepository<StaffMember>: This interface will provide a means to communicate with the underlying data access mechanism to retrieve staff member information.
IRepository<SupportTicket>: This interface will provide a means to communicate with the underlying data access mechanism to retrieve support ticket information.

We’ll now turn our attention to look at the revised domain class diagram, taking into account what we’ve learned with the creation of the CRC cards and sequence diagrams.

Revised Domain Class Diagram

Now it’s starting to look like a real application! (On paper anyway. >:/ ) The class diagram now contains enough classes to warrant noting which project layer each class belongs in. We also have enough information to decorate classes with methods (i.e., behavior) in addition to attributes. If you’ve been paying attention, you may notice that some of the class names are slightly different from the CRC cards and/or sequence diagrams; e.g., ManageCustomersController was called ManageCustomerController in the sequence diagram. This is perfectly fine and expected; your design should continue to evolve as conventions are established, details are discovered, and new changes are incorporated.

With the sequence diagrams established to describe how the objects collaborate, and the class diagram designed to show how the classes are organized, we’re ready to begin switching gears from design to implementation. If you’ve been designing user stories along the way, the actual implementation should be the easy part!

Outputs

Domain class diagram
Sequence diagrams

That wraps up Day 3 in this series. In Day 4, we'll begin the implementation phase of the project and get our hands dirty with some code!

Enjoy!
Billy McCafferty

Greenfield Development with ASP.NET MVC & S#arp Lite - Day 2

Billy McCafferty — Fri, 12 Oct 2012 22:00:00 GMT

A series of posts providing proven guidance for developing ASP.NET MVC applications from idea to well-designed implementation.

Day 2 – Define the Domain Conceptual Model

Objective of the Day

Transform the requirements from Day 1 into an appropriate domain conceptual model reflecting objects, attributes, and associations.

Inputs

Project Vision
User Stories or Use Cases
Data Dictionary

Activities

The domain conceptual model is a visual representation of conceptual or real-world objects in the domain of interest [Fowler, 1996]. (The domain design model, which will be discussed in Day 3, is a visual representation of the classes and behaviors which will be implemented in the software.)

The conceptual model is not a class diagram describing classes with methods. But in practice, the conceptual model will iteratively become the basis of the class diagram of the design model. Specifically, the conceptual model reflects the following information:

Domain conceptual objects,
Associations between conceptual objects, and
Attributes of conceptual objects.

The important distinction between conceptual model and design model is that the conceptual model should represent a clear business-oriented abstraction of the domain; i.e., the client should be able to look at the conceptual model and understand it fully without difficulty. Compare this to the design model which will include method names, data types, polymorphic associations, and supporting classes, such as factories and services.

To illustrate, suppose your domain includes the concept of “Customer” and “Employee.” The conceptual model would reflect two separate objects, each having duplicate attributes such as first name and last name. The design model may instead reflect a parent “Person” class, inherited by both Customer and Employee, having duplicated fields moved up to the Person superclass. Applying UML and Patterns [Larman 2004] goes into great deal concerning this process and is highly recommended read.

We’ll get into more details of the design process when we see it in action, next.

Putting it Into Practice

For Acme Telecom’s CLogS, the business requirements have been established. It is now time to begin transforming the requirements into a domain model.

The first step along this path is to define the conceptual model. We’ll define the conceptual objects, define their relationships with other objects, and add attributes, describing the pertinent information associated with each. But how exactly do we go about naming the conceptual objects? [Larman 2004] suggests looking for objects which fall into a Conceptual Class Category, such as:

Physical or tangible objects,
Places,
Transactions and transaction line items,
Roles of people,
Physical container of other things,
Things in a container,
Organizations,
Events,
Processes,
Rules and policies, or
Nouns.

For finding associations, Larman suggests looking for the following types of associations:

A is a physical or logical part of B,
A is physically or logically contained in/on B, or
A is recorded in or is a line item of B.

Now that we know what to look for, what artifacts do we look at as potential sources for objects? The primary sources include the vision, data dictionary, and user stories; i.e., the artifacts from Day 1.

As a caveat, if you find conceptual objects in the vision or data dictionary that are not included in the user stories, you should double check to make sure that the user stories are adequately representative of the requirements. While you may find extra concepts in the data dictionary, since it provides explanatory details of terms, extra concepts in the vision – but not found in the user stories – likely indicates an inadequacy, accordingly.

Looking at the inputs from the previous day, potential conceptual objects include:

Customer
Support Staff
Management
Support Call
Support Ticket
Ticket Resolution Report per Support Staff
Ticket Resolution Report per Issue Type

Compiling the list of concept objects will likely incite useful discussion, much of which will help to ensure that the business has been adequately modeled, and how the concept model will be later transformed into the design model.

At right, review an example of how the conceptual model could be visually modeled, for the concepts discussed, with associations and attributes. As shown, all of the major elements of the conceptual model have been visualized. There are some interesting things to note concerning this model:

There are many duplicated fields between Management and SupportStaff. When implemented, these will likely be combined into a single Employee class; but recall that this is the conceptual model and should include more, rather than less, detail to cover all of the unique concepts within the requirements.
SupportTicket and SupportCall have a one-to-one relationship with each other. These may have instead been merged into a single concept (and will be done so in the design model), but splitting them into separate concepts at this stage emphasizes that there is a call from the customer which leads to a ticket being opened.
TicketResolutionReport is a report concept which includes a listing of SupportTickets and a number of ResolutionMetrics. Note that there is not an explicit association between TicketResolutionReport and the SupportTicket object to emphasize that the report, rather than being a core element of the domain, is an auxiliary concept to the domain. This is a subtle and subjective point but can help to establish bounded contexts during the design of the implementation. While it would not have been wrong to have included an explicit association, doing so would not have added much useful meaning to the context of the requirements. Associations between objects in a conceptual model should indicate some degree of permanence.
As an additional note, the ResolutionMetrics attribute of TicketResolutionReport may indeed be a separate concept object itself; but at this point, it’s added as a placeholder, to be expanded upon at a later time.

As may be inferred, there’s certainly a lot of subjectivity and brainstorming that goes into creating the domain conceptual model. As such, don’t try as much to get it "just right" as to ensure that it comprehensively captures all of the key concepts and associations within the requirements.

When creating the conceptual model and you are faced with the decision to split a concept (or merge separate concepts), err on the side of splitting the concept to provide more granularity than may be necessary for the actual implementation; i.e., it’s better for the conceptual model to be too expressive than too simplistic. Once the concepts are established, you can always combine and regroup as necessary for the design model, but you don’t want to miss anything along the way.

As a final note, defining conceptual objects is more important than identifying the associations among them, and more time should be spent on object identification, accordingly.

Outputs

Domain conceptual model

That wraps up Day 2 in this series. In Day 3, we'll Define the Domain Design Model...

Enjoy!
Billy McCafferty

Greenfield Development with ASP.NET MVC & S#arp Lite - Day 1

Billy McCafferty — Fri, 24 Aug 2012 17:15:00 GMT

A series of posts providing proven guidance for developing ASP.NET MVC applications from idea to well-designed implementation.

Part I - Planning & Design

For the planning and design phase of project delivery, each "day" will have an objective, inputs, activities, and outputs:

Objective: This is a short description of the goal – or expected end result – of the day.
Inputs: These are the informational inputs, document artifacts, and decisions that should have been made before beginning the day’s activities.
Activities: This is the core element of each day, describing the interactions with the client and the practices used for ultimately transforming ideas into production code. For each day, we’ll discuss the activities to be achieved and put them into practice on our example project.
Outputs: These are the artifacts that will have been generated during the day’s activities, some of which may be used as inputs for subsequent days.

Day 1 – Define the Requirements & Actor/System Interactions

Objective of the Day

Work with the client to translate ideas into requirements. By the end of the day, you should be able to concisely describe the project’s vision, actors, and implementable requirements.

Inputs

Client project ideas, with knowledge of who will be interacting with the system.

Activities

Project Vision

The first activity of the day is to establish the project vision. The vision describes the high-level goals of the system and the business case for implementation. It’s the two-minute elevator speech describing what the system is intended to achieve.

While defining the project vision, a key activity will also be to clearly define the “actors,” or the people who interact with the system and the goal(s) of those interactions, accordingly.

This may seem like a simple, or even dismissible, step, but clearly defining the vision and actors is one of the most essential elements of the entire project. The vision serves as a good litmus test to use to verify if a client idea is relevant to the project’s vision. Additionally, by clearly defining the actors, it allows you and the client to focus on just those requirements which satisfy the needs of those actors; i.e., if an idea comes up which doesn’t serve the needs of one of the defined actors, it can be summarily discarded or put on an “idea parking lot” (so the client feels like you’re actually going to look at it again someday…which you’re not). In other words, don’t skip this!

Putting it into Practice

Our aspiring client’s company is Acme Telecom. Acme Telecom would like to create a support call tracking system to log and monitor all calls, which will be tracked as “support tickets,” coming into its support department. Currently, Acme Telecom’s paper-based system is making it difficult to assess how well its support staff is responding and closing calls. Acme Telecom doesn’t want to try to do too much at once, so the client wants to keep things very simple and small for this first release. Accordingly, the vision that we and the client were able to agree on was as follows:

Acme Telecom’s paper-based approach to support call tracking is limiting the company’s ability to improve support capabilities. Accordingly, Acme Telecom needs an easy-to-use, secure, web-based Call & Logging System (or "CLogS" for short) for its Support Staff to track and resolve support calls from Customers as “support tickets.” In addition to facilitating support ticket resolution, the system must provide high-level reporting for Management to filter support ticket logs and to assess key performance metrics, such as rate of resolution and resolution bottlenecks.

Not only does this vision define the general scope of the project, but did you notice that the actors were also clearly stated? (We can even infer hints of the domain model…but that’ll be for a later discussion.)

Taking the vision a bit further, the actors, and goals of each actor, are as follows:

Actor	Major Goals
Support Staff	Log support ticket. Update and/or resolve calls opened by self.
Customer	Have support staff resolve calls.
Management	Run filterable reports on call logs. Run "canned reports" to assess resolution performance.

With that, we have a good general scope definition of the project, we know who the primary stakeholders are, and what each would like to achieve.

At this stage, it doesn’t look like the customer would actually be interacting with the system since they’ll be calling the Support Staff directly...should we include them as an actor? Yes, due to the fact that they represent the key stakeholder to be satisfied and should be considered during requirements elaboration by asking “Is this idea going to help the Customer get their calls resolved?”

User Stories or Use Cases?

Both User Stories and Use Cases summarize the various features of the system. Each should be small enough to be estimable, but big enough that the client would be willing to pay for its implementation and measure progress. With that said, it is important to decide, very early on, if requirements will be defined using User Stories or Use Cases (or another means altogether) as there are appreciable differences between the two.

Users Stories, an element of the Scrum/XP, are very short requirement descriptions which are intended to be elaborated during development iterations; they are truly markers for further discussion. A User Story is a feature stated from the perspective of the actor(s) who will benefit from that feature. Each User Story typically includes the following details:

Description: The feature from the actor’s perspective.
Priority: Higher priority features should be addressed earlier in the project life-cycle while lower priority features may be dropped in favor of higher priority change requests, or dropped altogether to assist with meeting schedule deadlines.
Risk: Higher risk User Stories may be A) split into smaller risk User Stories, B) researched to mitigate risk, C) reflected as a higher uncertainty in the estimate, or D) dropped to reduce uncertainty in scope and schedule.
Estimate: Typically described in “points” rather than in hours to better facilitate relative-estimating techniques.

To learn more about User Stories, I encourage you to read User Stories Applied by Mike Cohn.

Use Cases, on the other hand, include much more detail for each requirement. For example, while a User Story may be described in a sentence or two, a Use Case describes prerequisites for the Use Case to be activated, a “happy path” of execution, exceptive cases to execution, and post-conditions, describing the state of the system after completion. To learn more about Use Cases, I encourage you to read Applying Use Cases by Geri Schneider.

There are pros and cons to each; google “User Story vs. Use Case” to get a swath of opinions. Personally, I prefer User Stories if I feel the client is very unsure of what the final system will be, or if the client is supportive of flexible scope; I prefer Use Cases if the end result is very definable or if the client demands fixed bid (which always makes me shudder). There’s more to it than that, but for the purposes at hand, we’re going to go with User Stories for Acme Telecom’s CLogS application.

Putting it into Practice

Now that the vision, actors, and actor-goals have been defined, we’re ready to define the requirements of the solution. We’ll do so by working with the client to define the requirements as User Stories. To keep things simple, we’ll ignore Priority, Risk and Estimate, focusing simply on the description of each. Our User Stories for CLogS are as follows:

Support Staff may open a new support ticket and provide details including: Customer, Issue Details (such as description and “dynamically managed” type), Status, and Resolution Details (if resolved immediately). Status of the call may be New, In Progress, or Resolved. (“Dynamically managed” means that if the issue type isn’t available, the Support Staff may enter a new one when entering the support ticket.)
Support Staff may manage Customer details, adding and updating Customer information when necessary.
Support Staff may view listing of all tickets opened by self, defaulting to those which are not resolved and filter the listing, accordingly.
Support Staff may search for an existing support ticket, opened by self, view details of ticket, and update its status and resolution details.
Management may view listing of all tickets, defaulting to those which are not resolved and filter the listing, accordingly.
Management may view report showing information regarding how quickly calls are being resolved, called the “Ticket Resolution Report by Support Staff.” The report may be run for all Support Staff or for a specific Support Staff employee. The manager creating the report may only include Support Staff of whom they are manager.
Admin may view the “Ticket Resolution Report by Support Staff” for any and all Support Staff.
Management may view chart showing a breakdown of call by issue type and average resolution time for each issue type, called the “Ticket Resolution Report by Issue Type.” The manager creating the report will only see stats for Support Staff of whom they are manager.
Admin may view the “Ticket Resolution Report by Issue Type” to include any and all Support Staff.
Admin may manage all details of Support Staff and Management, along with which Support Staff are assigned to which Manager. Each Support Staff may only have, and must have, one Manager.

Data Dictionary

One of the most frequently skipped steps of project delivery is also one of the simplest and helpful: the creation of a data dictionary. A data dictionary is a simple glossary of domain-related terms which serves to reduce requirements confusion between the client and project delivery team.

As project requirements become more complicated, a data dictionary becomes increasingly important to ensure everyone is speaking the same language. (Not) surprisingly, it’s frequently difficult for even client associates to agree upon term definitions; a data dictionary helps to get people talking and to (hopefully) agree upon the meaning of business terms. Forming this agreement of definitions forms the basis of Ubiquitous Language, a prominent component of Domain-Driven Design.

A few years back I developed a project management tool for a construction management project. One of the key features of this tool was a Master Budget Report which combined information from various sources to ultimately provide an Estimate at Completion for each project. The report’s data columns included Expenditures, Change Orders, Pending Obligations, Remaining Obligations, Total Obligations, and others. It took over three months of workshops involving client stakeholders from the financial group, project controls group, project managers, and upper management to agree upon what constituted a Pending Obligation vs. a Remaining Obligation and to sort out other such definitions and equations associated with the report. Imagine the frustrations that would have ensued if we had tried to develop this “standard” report when no one agreed to what was actually “standard.”

The lesson of the story is...don’t underestimate the usefulness of a well-discussed data dictionary. And knowing is half the battle.

Putting it into Practice

Acme Telecom’s CLogS is starting to take shape as vaporware; we’ve agreed to the vision and even the requirements. But during requirement elaboration discussions, there was some confusion on a few key terms. The following data dictionary helped to alleviate the confusion:

Term	Definition
Customer	A person who has an Acme Telecom account number. (Anyone else is just an annoying person wasting Support Staff’s time.)
Resolution Time	The number of business days (whole number) to resolve a call is calculated by taking the difference between the day a ticket is opened, and the day when it is resolved, excluding weekends. If a ticket is resolved in the same day it was opened, its resolution time is 0 days.
Avg. Resolution Time	This is the number of mean business days (decimal number) it takes for an arbitrary group of support tickets to be resolved; i.e., the sum of business days to resolve each ticket divided by the number of tickets.

It was easy to assume that everyone understood what "resolution time" meant when discussing requirements. But as we attempted to define this term in the data dictionary, it was recognized to be a non-trivial idea. Having this cleared up will help to ensure that anyone using the system has the same understanding of the term...assuming they’ve taken five minutes to read the data dictionary, which is quite an assumption in and of itself!

It should also be noted that the data dictionary was used to not only clarify project-related terms and calculations, but also to clarify the definition of actors.

While it’s not necessary to document every project term, any term which involves tricky calculations, subjectivity in defining, or has been identified as causing disagreement, should certainly be included. Furthermore, the data dictionary may be separated into separate lists such as Actors, Domain, and Calculations, but start simple and expand when necessary.

Actor/System Interaction Diagrams

I’ve been a strong proponent of agile development techniques for years; but a challenge I’ve always had is keeping track of the "big picture" as requirements are broken down into more granular requirements as user stories. An associate of mine, Jim Tucker, introduced me to a quick and easy technique that I’ve not seen surpassed for expressing the big picture in a clear and concise manner.

Before delving into technical design, Jim creates "Actor / System Interaction Diagrams" (as UML sequence diagrams) to document how the actors interact with each other and with the underlying system, treating the system as a black box. It’s possible to use this technique not only for the overall system, but down to the level of a single user story. This facilitates "zooming in and out" of the details as appropriate.

This is similar in nature to the back and forth between an actor’s actions and system’s responses described within a Use Case (an alternative approach to User Stories for documenting feature requirements), and to Craig Larman’s System Sequence Diagrams in Applying UML and Design Patterns. It differs from Use Cases in the fact that it’s more graphically oriented, and differs from both in that it may be easily expanded to include many participants and need not be constrained to a single feature. On a recent project, we used this technique to illustrate the high-level interactions among eight actors and multiple third party systems; the end result was simple enough for the client to understand and verify its logic, but expressive enough to be a useful artifact during technical design.

To use this approach, it’s first necessary to understand UML sequence diagrams, which is beyond the scope of this article. (Googling "UML sequence diagrams" will get you going in the right direction.) Having an understanding of sequence diagrams, Actor / System Interaction Diagrams differ from sequence diagrams in the following ways:

Actors are used in place of classes.
The system itself is treated as a single participant in the sequence diagram.
The "messages" among participants, instead of being API calls, are requests between actors and/or the system itself.

But, as usual, an example speaks a thousand words...

Putting it into Practice

The trickiest User Story in the CLogS application is as follows:

Support Staff may open a new support ticket and provide details including: Customer, Issue Details (such as description and type), Status, and Resolution Details (if resolved immediately). Status of the call may be New, In Progress, or Resolved.

This one caused the most discussion and had the most unknowns. Accordingly, it was decided that an Actor / System Interaction Diagram would be created to clarify the sequence of interactions.

The actors that will be in the sequence diagram include Customer, Support Staff, and the CLogS application. The sequence of events to handle a new support call is as follows:

Customer calls Support Staff and describes the issue.
Support Staff opens a new support ticket in CLogS which requires the following sub-steps:
1. Support Staff gets Customer account number.
  1. Support Staff uses account number to find customer.
  2. If not found, Support Staff adds a new Customer entry.
  3. If found (or after created), Support Staff generates a new support ticket for the Customer.
2. Support Staff adds issue details to the new support ticket entry.
If resolvable, Support Staff assists the Customer and marks the support ticket as resolved.
Otherwise, Support Staff gives the Customer the support ticket reference number and pinky swears to resolve the issue and get back with them.

The Actor / System Interaction Diagram above shows the actors involved and their interactions with each other. Although this level of detail isn’t necessary for every User Story, it’s a useful technique for illustrating the business processes behind more complex User Stories. Furthermore, this technique need not be limited to the User Story level; alternatively, you could create Actor / System Interaction Diagrams for major modules and/or include third party systems as additional "actors." UML sequence diagrams are quite versatile for this kind of work.

Outputs

Project Vision
User Stories or Use Cases
Data Dictionary
Actor / System Interaction Diagrams

That wraps up Day 1 in this series. In Day 2, we'll Define the Domain Conceptual Model...

Enjoy!
Billy McCafferty

Greenfield Development with ASP.NET MVC & S#arp Lite - Introduction

Billy McCafferty — Thu, 02 Aug 2012 21:08:00 GMT

A series of posts providing proven guidance for developing ASP.NET MVC applications from idea to well-designed implementation.

So you’ve landed your first project! Just days ago your client came to you and said “I’ve got a great idea and I want you to build it!” With unwavering confidence, you quickly retorted “And I’ll build it, and it’ll be friggin’ awesome!” You agree to a price, you put all the business paperwork in place, and you’re ready to get coding…you sit down at the computer, raring to go, and think to yourself “Now where the hell do I start?”

To begin with, you’ve already gotten ahead of yourself. Before a single key stroke of code is developed, it’s time to do some planning. And once the planning is done - and only after that - it’ll be time to turn those plans into maintainable, tested code.

While there are plenty of coding samples of design patterns and "best practices," it’s sometimes difficult to figure out how the developer got from an empty directory in Windows Explorer to an implemented piece of architectural and functional beauty. This series of posts will walk you through the basics of the project life cycle, from project inception to project completion, turning ideas into requirements, requirements into plans, and plans into code, with a clear and blatantly unfair bias towards the developer’s perspective.

This isn’t intended to discuss a particular project management methodology or advocate the principles of any specific development methodology. Instead, this series’ focus is on using a number of tricks of the trade of traditional Object Oriented Analysis & Design (OOAD) and various development techniques to turn ideas into working code. In other words, bits and pieces from UML, design patterns, OOAD, Domain-Driven Design , Test-Driven Development , etc…you know, like what you’d do on a real project.

Although we’ll be using S#arp Lite as the project starting point and architectural infrastructure, most of the principles and techniques we’ll be using will be applicable to just about any development implementation you’d prefer.

The project activities will be examined over seven “days” with each day focusing on a distinct element of project production. Note that each day, when mapped to a real project, may represent an hour, a day, a week, or perhaps even a month(s) depending on the activities involved and the size of the project. (So don’t take "day" too literally.)

The seven days of the project will cover three major themes of project delivery:

Part I focuses on the planning and design elements of project delivery; i.e., turning project ideas into a workable design. Part I is split into three days:
Part II focuses on the implementation phase of project delivery; i.e., turning the design into a working project. Part II is split into four days:
- Day 4 – Setup the Project & Security Infrastructure
- Day 5 – Develop CRUD & Data Management Capabilities
- Day 6 – Provide Reporting Capabilities
Part III focuses on extending the project to integrate with third-party, external services with an appropriate separation of concerns.
- Day 7 – Integrate with External Services

While this series of posts walks through the development of the project in a seemingly waterfall fashion, it does not intend to promote such a methodology; indeed, an agile/iterative approach to development is recommended but is beyond the scope of this series to discuss. If you find them useful, I'd encourage you to adapt some or all of the described techniques into your organization’s preferred methodology.

At the end of the series, you should have a basic understanding of how to turn a project idea into a well-designed, maintainable deliverable using real-world techniques, starting from nothing but an idea.

Who is the Intended Audience?

While I am attempting to write this series to be understandable by any skill level, you're expected to have basic knowledge of UML (specifically class diagrams and sequence diagrams) and to be familiar with ASP.NET MVC and C#. For materials on these subjects, refer to Pro C# 2010 and the .NET Platform [Troelsen 2010], Professional ASP.NET MVC 3 [Galloway 2011], and UML 2.0 Pocket Reference [Pilone 2006]. Other references will be provided along the way to help fill in any missing gaps and secondary subjects.

Developers of experienced-beginner to intermediate levels will gain the most from this series.

The next post in this series is Day 1 – Define the Requirements & Actor/System Interactions...

Enjoy!
Billy McCafferty

S#arp Lite: The Basics

Billy McCafferty — Sat, 12 Nov 2011 00:01:00 GMT

What is S#arp Lite?

S#arp Lite is an architectural framework for the development of well-designed, custom-built, ASP.NET MVC applications.

ASP.NET MVC 3 is a terrific platform for delivering web-based applications. But, similar to ASP.NET, it does not provide specific guidelines for how to best use it in different project contexts. That's certainly the point; it exists to provide a flexible platform which may be used in a variety of situations without being biased towards one architecture or another, beyond the fundamentals of Model-View-Controller. The benefit of this is that you can structure MVC projects almost anyway you'd like; the drawback is that it's possible to have almost no consistency among your organization's projects, even if they're all using ASP.NET MVC.

That's where S#arp Lite comes in to play. S#arp Lite comes packaged with three primary assets to provide a turnkey solution for developing well-designed, MVC applications:

A project template to facilitate the creation of new S#arp Lite projects, pre-configured to communicate with your database using NHibernate;
A set of reusable class libraries which encapsulates infrastructural concerns (including a base repository); and
Architectural guidance on how to build out a S#arp Lite project.

Currently, the architectural guidance is demonstrated via the sample project which has been included in the S#arp Lite release package. Architectural guidelines are also enforced by the direction of dependencies among the project layers. (This will be discussed in more detail below.) If you have technical questions, please post to the Google Group at http://groups.google.com/group/sharp-lite.

The overall objective is to allow your development team to more easily develop ASP.NET MVC applications which adhere to well founded principles, such as domain-driven design and test-driven development; without being bogged down with infrastructural setup and without sacrificing long-term maintainability and scalability of the solution.

As a quick side, the base repository class which S#arp Lite exposes is purposefully very simplistic. The base repository only includes the following methods:

Get(id): returns an entity from the database having the Id provided,
GetAll(): returns an IQueryable<> which may be further filtered/transformed via LINQ,
SaveOrUpdate(entity): persists an entity to the database, and
Delete(entity): deletes an entity from the database.

Keeping the base repository very light reduces bloat in the reusable libraries and places greater emphasis on the use of LINQ for retrieving results from GetAll(). Besides, you can extend it with your own code to bloat it as much as you'd like. We'll discuss this in more detail a bit later.

Who is this intended for?

The motivation for S#arp Lite came from working with many teams (including my own) who had been developing projects with S#arp Architecture. To many, S#arp Architecture is simply too big of an architectural framework to easily get your head around. When I used to discuss S#arp Architecture with teams who were considering using it, I would always suggest that their developers be very experienced and well versed with topics such as dependency inversion, low-level NHibernate, and domain-driven design.

The reality of business is that it's not likely that your team will be made up of all senior level developers who are all experts in these topics. S#arp Lite is intended to epitomize the underlying values of S#arp Architecture, strive to be equally scalable to large projects, all while being tractable to a larger audience. In other words, you should be able to have a realistically skill-balanced team and still be able to successfully deliver a S#arp Lite application.

S#arp Lite is recommended for any mid-to-large sized ASP.NET MVC project. If you have a small mom & pop store, you'd likely be better off using a less-tiered application setup. S#arp Lite scales well to very large projects; we're currently using it effectively on applications which integrate with a half dozen other systems...so it certainly holds up to meatier challenges.

What does a S#arp Lite project look like?

Creating a new S#arp Lite project is trivially simple:

Download and unzip the S#arp Lite release package from GitHub.
Follow the instructions within the README.txt to create your S#arp Lite project with Templify (a brilliant little tool).

After you've created a S#arp Lite project, you'll find the following directory structure under the root folder:

app: This folder holds the source of your project; i.e., the code that you're getting paid to write.
build: This initially empty folder is a placeholder for your build-related artifacts; e.g., a "publish" folder, NAnt or MSBuild artifacts, etc.
docs: This initially empty folder contains all of the documents for your project. Keeping them here keeps all of your docs checked in with the code.
lib: This folder contains all of the DLL dependencies for your project, such as log4net.dll, SharpLite.Domain.dll, System.Web.Mvc.dll, etc.
logs: This initially empty folder is intended to hold any generated log files. The generated project's web.config used this folder for dumping out log4net logs.
tools: This initially empty folder is intended to hold any third party install files or other dependencies which the team may need to work on the project. For example, this is where we store the latest install files of Telerik ASP.NET MVC and NUnit, used by the project. Having all of your installable dependencies, checked in with the code, makes it much easier to get "the new guy" up and running quickly.

The auto-generated folder structure is just a means to help keep your digital assets and solution files organized. The more interesting stuff is in the /app folder which houses the source code of your project. But before we delve into each of the project layers in the /app folder of a S#arp Lite project, let's take a birds eye view of the overall architecture.

The diagram above reflects the layers of a S#arp Lite project, implemented as separate class libraries and an ASP.NET MVC Web Project. Having the tiers in separate class libraries allows you to enforce the direction of dependency among them. For example, because YourProject.Tasks depends on YourProject.Domain, YourProject.Domain cannot have any direct dependencies on a calss within YourProject.Tasks. This singled-directional dependency helps to enforce how the architecture is to remain organized.

While the diagram above describes the basic purpose of each layer, it's most assistive to look at an example project to have a clearer understanding of the scope of responsibilities of each layer. Accordingly, let's examine the tiers of the MyStore example application which was included in the S#arp Lite release package.

Examining the Layers of MyStore Sample Application

The MyStore sample application, included in the release zip, demonstrates the use of S#arp Lite for a fairly typical CRUD (create/read/update/delete) application. It includes managing data and relationships such as one:one, one:many, many:many, and parent/child. Let's take a closer look at the relational model.

The diagram above represents the relational object model, as implemented within a SQL Server database. It's a very simple model with basic relationships, but it still brings up a lot of interesting discussion points when we go to translate this relational model into the object-oriented design of our application.

For example, each customer entity contains address information (stored as StreetAddress and ZipCode in the Customers table); in the domain, we want the address information pulled out into a separate object called Address. Having this information as a separate object more easily allows us to add behavior to the Address object while keeping those concerns separate from the Customer object, such as integration with a USPS address validation service. (Arguably, such a service would be in a stand-alone service class, but you get the idea.)

As another example, the many:many Products_ProductCategories table shouldn't have a similarly named class in our object model; instead, we would expect Products to have a listing of ProductCategories and/or vice-versa.

The sample application includes examples of how all of this and mappings of the classes themselves have been achieved almost entirely via coding-by-convention. Let's now look at the sample application, layer-by-layer, and interesting points along the way.

MyStore.Domain

This layer of the application contains the heart of the application; it represents the core domain of our product. All the other layers exist simply to support the user's need to interact with the domain. In a S#arp Lite project, the domain layer contains four types of objects:

Domain Objects
Query Objects
Query Interfaces
Custom Validators

Let's review each in turn.

Domain Objects

Alright, so maybe this is a whole bunch of types of objects, but they all have the same purpose - they exist to implement the domain of our application. They consist of entities, value objects, services (e.g., calculators), factories, aggregates...all organized into modules, usually expressed as separate namespaces. (I highly recommend Eric Evans' Domain-Driven Design, Jimmy Nilsson's Applying Domain Driven Design and Patterns, and Agile Software Development by Robert Martin to help guide you.) There are two namespaces in the sample project: the "ProductMgmt" namespace which contains everything related to product management, and the "root" namespace for, well, everything else. Your project will likely have others.

Let's now take a look at the Customer class within the sample project as an example of an entity. There are some important items to note:

Customer inherits from Entity (which is from SharpLite.Domain.dll). The Entity base class A) signals that this class is a persisted object which has an associated table in the database, B) provides an Id property (no biggie there), and C) facilitates comparing two entities to each other. If two entities are of the same type and have the same Id, then you know they're the same object. But what if you're comparing two "transient" entities; i.e., two entities which have not yet been persisted to the database. As another example, how would you go about comparing a transient entity to entities that have been persisted.

For this we need to compare "domain signatures." A domain signature is the fingerprint of what makes the entity unique from a business perspective. In other words, which property(s) of an object would make it identifiable without having an Id property? Looking at the Customer class, we see that there are two properties decorated with the attribute "DomainSignature." Furthermore, the class itself is decorated with the attribute "HasUniqueDomainSignature." This means that no two objects may exist having the same first and last name. (This will not be appropriate in all scenarios; but should reflect the domain signature of the object in the context of the application.) The described attributes are included in SharpLite.Domain.dll and support automatic validation of the class' domain signature. So if you try to add a new customer with the same first and last name as an existing customer, a validation message will let you know this is not allowed.
```
[HasUniqueDomainSignature(...
public class Customer : Entity
{
    [DomainSignature]
    ...
    public virtual string FirstName { get; set; }
```
The Customer class pulls encapsulates the address information from the Customers table into a separate Address class. Interestingly, NHibernate "loquacious" mapping (aka - code by convention) automatically maps the related table columns into this "component" object.
Customer has an IList of Orders with a protected setter; the collection is initialized in the constructor. This is done for two reasons: 1) by design, the collection will never be null (which avoids a lot of object reference exception avoidance), and 2) we don't have to worry about NHibernate losing its pointer to the original collection loaded from the database. This collection "pattern" is simply good practice for exposing/protecting the collection.
```
public class Customer : Entity
{
    public Customer() {
        Orders = new List();
    }

    public virtual IList Orders { get; protected set; }
```
Validation is enforced with standard, .NET data annotations. No need for NHibernate.Validator or other validation mechanism when it's all available via the .NET library. And whenever you run into limitations, you can simply create a custom validator.

Query Objects

Regularly, we need to filter information returned from the database. For example, we may just want to return "active" customers vs. all the customers in the database. For performance reasons, it's obviously better to put as much filtering work on the shoulders of the database. Before LINQ providers, it was difficult to find the appropriate balance between filtering on the domain side or filtering on the database side. But with LINQ and IQueryable, filtering can be developed within the domain while it's still executed on the database. Brilliant! To facilitate this, every repository (e.g., IRepository<Customer>) exposes the method GetAll() which returns IQueryable<>.

The abso-friggin-spectacular side effect of this is that we can avoid having specialty "repository" methods which exist simply to hide away the details of the underlying data-access mechanism. There are two kinds of query objects in a S#arp Lite project:

Specification Query Objects: Specification query objects take a list and filter the results down to a smaller list, based on some criteria. In the sample project, the specification query class, MyStore.Domain.FindActiveCustomers, provides an extension method to IQueryable<> with any filter parameters passed in to the method. Alternatively, the query object could be a POCO class, accepting an IQueryable<> and filtering, accordingly. The benefit to setting up the specification query object as an extension, albeit, with additional indirection, is that multiple queries may be chained, all while taking advantage of IQuerable<>'s delayed querying (i.e., only one query will be sent to the database even if you chain multiple queries.
```
public static class FindActiveCustomersExtension
{
    public static IQueryable FindActiveCustomers(this IQueryable customers) {
        return customers.FindActiveCustomers(MINIMUM_ORDERS_TO_BE_CONSIDERED_ACTIVE);
    }

    public static IQueryable FindActiveCustomers(this IQueryable customers, int minimumOrders) {
        return customers.Where(c =>
            c.Orders.Count >= minimumOrders);
    }

    private const int MINIMUM_ORDERS_TO_BE_CONSIDERED_ACTIVE = 3;
}
```
To me, the real beauty in this is that the query object may live in the domain, and be tested as a first class citizen of the domain, without introducing any dependencies to the underlying data-access layer (whether that be NHibernate, Entity Framework, etc.).
Report Query Objects: Report query objects take a list, filtering if necessary, transforming and returning the results as a DTO or list of DTOs. Imagine that you have a summary dashboard in your application; e.g., a page which shows how many orders each customer has placed and what is each customer's most frequently purchased product. In this scenario, we'd ultimately like a list of DTOs containing each customer's name, his/her order count, and his/her favorite product. There are a few options to tackling this:
- Create a DB stored procedure, binding the results to the DTOs list (and thus put processing logic onto the DB),
- Use NHibernate Criteria, HQL or named query to retrieve the results (and tightly couple your data-access code to NHibernate),
- Traverse the object model on the domain side to collate the information (can you say n+1?), or
- Use clean and simple, data-access agnostic LINQ (and keep it in the domain).
2 points if you guess which one I'm leaning towards. Let's look MyStore.Domain.Queries.QueryForCustomerOrderSummariesExtension for an example:
```
public static class QueryForCustomerOrderSummariesExtension
{
    public static IQueryable QueryForCustomerOrderSummaries(this IQueryable customers) {
        return from customer in customers
                select new CustomerOrderSummaryDto() {
                    FirstName = customer.FirstName,
                    LastName = customer.LastName,
                    OrderCount = customer.Orders.Count
                };
    }
}
```
Again, the advantage of this is that it can live within the domain layer and act as a reusable reporting query without introducing dependencies to the underlying data-access layer.

You have a lot of flexibility on how you use query objects and where they live. For example, you could use an ad-hoc report query (i.e., a LINQ query not encapsulated by a class) which lives within a method in the tasks layer. Although I'd advise against it, you could even use an ad-hoc query within a controller's method. So the provided samples are just that, samples of a particular approach. What's most important is to agree as a team how you'll encapsulate and organize query objects. In the sample project, queries are encapsulated as query objects and stored within a "Queries" folder - one folder per namespace.

Query Interfaces

In the very unlikely event that you need to leverage the data-access mechanism directly, instead of LINQing IQueryable, the domain layer may also contain any query interfaces which define a query to be implemented by the data-access layer. (This is akin to creating custom repository interfaces in S#arp Architecture.)

The disadvantages to this approach are three-fold:

It introduces a layer of indirection to the developer,
It more tightly couples your code to the underlying data-access layer (since some of your application's logic is now in the data-access layer), and
It becomes trickier to unit test the query since you need an in-memory or live database to test the implementation.

But, I can foresee that there may be a situation where this is necessary if you have a very complicated query, need to leverage NHibernate detached queries, or simply can't do what needs to be done via LINQ. Accordingly, three steps would need to be taken to support the query:

Define the query interface in YourAppProject.Domain (e.g., MyStore.Domain.ProductMgmt.Queries.IQueryForProductOrderSummaries.cs),
Implement the concrete query class in YourAppProject.NHibernateProvider (e.g., MyStore.NHibernateProvider.Queries.QueryForProductOrderSummaries.cs), and
Register the implementation with the IoC to resolve requests to the interface (e.g., MyStore.Init.DependencyResolverInitializer).

Obviously, not as clean as using Specification and Report Query Objects, but available if absolutely necessary.

Custom Validators

As discussed previously, S#arp Lite uses .NET's data annotations for supporting validation. (You could use something else, like NHibernate.Validator if you prefer.) The data annotations are added directly to entity classes, but could instead be added to form DTOs if you feel that entities shouldn't act as form validation objects as well.

Sometimes, data annotations aren't powerful enough for the needs of your domain; e.g., if you want to compare two properties. Accordingly, you can develop custom validators and store them within a "Validators" folder. If the custom validator is specific to a class, and never reused, then I'll usually just add the custom validator class as a private subclass to the class which uses it. In this way, the class-specific validator is neatly tucked away, only accessible by the class which needs it. S#arp Lite uses a custom validator to determine if an object is a duplicate of an existing object, using its domain signature: \SharpLiteSrc\app\SharpLite.Domain\Validators\HasUniqueDomainSignatureAttribute.cs.

MyStore.Tasks

This layer of the application contains the task-coordination logic, reacting to commands sent from, e.g., a controller in the presentation layer. (This layer is also described as a Service Layer in Martin Fowler's PoEAA.) For example, let's assume that your application integrates with a number of other applications. This layer would communicate with all of the other applications (preferably via interfaces), collating the information, and handing it off to the domain layer for performing domain logic on the data. As a simpler example, if your domain layer contains some kind of FinancialCalculator class, the tasks layer would gather the information needed by the calculator, from repositories or other sources, and pass the data via a method to FinancialCalculator.

As a secondary responsibility, the tasks layer returns data, as view-models, DTOs, or entities, to the presentation layer. For example, the presentation layer may need to show a listing of customers along with Create/Edit/Delete buttons if the logged in user has sufficient rights to do so. The tasks layer would get the listing of customers to show and would determine what security access the user has; it would then return a view model containing the customers listing along with a bool (or security object) describing if the user has rights to modify the data.

It's important to note the difference between the logic found within the tasks layer and that found within the domain layer. The tasks layer should contain minimal logic to coordinate activities among services (e.g., repositories, web services, etc.) and the domain layer (e.g., calculator services). Think of the tasks layer as an effective boss (does that exist?)...the boss helps to facilitate communications among the team and tells the team members what to do, but doesn't do the job itself.

The tasks layer contains two kinds of objects:

Task Objects
View Models

Task Objects

These are the tasks themselves. The most common kind of task is coordinating CUD logic (CRUD without the read). It's so common, in fact, that S#arp Lite projects includes a (completely customizable) BaseEntityCudTasks class to encapsulate this common need. Looking at the sample project, you can see how BaseEntityCudTasks is extended and used; e.g., within MyStore.Tasks.ProductMgmt.ProductCudTasks.

As the project grows, the task-layer responsibilities will inevitably grow as well. For example, on a current project which integrates with multiple external applications, a task class pulls schedule information from Primavera 6, cost information from Prism, and local data from the database via a repository. It then passes all of this information to a MasterReportGenerator class which resides in the domain. Accordingly, although the task class is non-trivial, it's simply pulling data from various sources, leaving it up to the domain to the heavy processing of the data.

It's important to note that a task object's service dependencies (repositories, web services, query interfaces, etc.) should be injected via dependency injection. This facilitates the ability to unit test the task objects with stubbed/mocked services. With MVC 3, setting up dependency injection is very simple and makes defining your task objects dependencies a breeze:

public ProductCategoryCudTasks(IRepository<ProductCategory> productCategoryRepository) : base(productCategoryRepository) {
    _productCategoryRepository = productCategoryRepository;
}

Here we see that the ProductCategoryCudTasks class requires a IRepository<ProductCategory> injected into it, which will be provided at runtime by the IoC container or by you when unit testing.

View Models

A view model class encapsulates information to be shown to the user. It doesn't say how the data should be displayed, only what data should be displayed. Frequently, it'll also include supporting information for the presentation layer to then decide how the information is displayed; e.g., permissions information.

There's a lot of debate about where view model classes should reside. In my projects, I keep them in the tasks layer, with one ViewModels folder per namespace. But arguably, view model classes could live in a separate class library; that's for your team to decide at the beginning of a project.

MyStore.Web

There's not much to say here. A S#arp Lite project uses all out-of-the-box MVC 3 functionality for the presentation layer, defaulting to Razor view engines, which you may change if preferred. The only S#arp Lite-isms (totally a word) in this layer are as follows:

MyStore.Web.Global.asax invokes DependencyResolverInitializer.Initialize(); to initialize the IoC container (discussed below),
MyStore.Web.Global.asax uses SharpModelBinder to act as the preferred form/model binder, and
Web.config includes an HttpModule to leverage a session-per-request, NHibernate HTTP module, found within the S#arp Lite source at \SharpLiteSrc\app\SharpLite.NHibernateProvider\Web\SessionPerRequestModule.cs.

SharpModelBinder extends the basic form/model binding with capabilities to populate relationships. For example, suppose you have a Product class with a many:many relationship to ProductCategory. When editing the Product, the view could include a list of checkboxes for associating the product with one or more product categories. SharpModelBinder looks for such associations in the form and populates the relationships when posted to the controller; i.e., the Product which gets posted to the controller will have its ProductCategories populated, containing one ProductCategory for each checkbox that was checked. You can take a look at MyStore.Web/Areas/ProductMgmt/Views/Products/Edit.cshtml as an example.

Like task objects, controllers also accept dependencies via injection; e.g. MyStore.Web.Areas.ProductMgmt.Controllers.ProductsController.cs:

public ProductsController(IRepository<Product> productRepository,
    ProductCudTasks productMgmtTasks, IQueryForProductOrderSummaries queryForProductOrderSummaries) {

    _productRepository = productRepository;
    _productMgmtTasks = productMgmtTasks;
    _queryForProductOrderSummaries = queryForProductOrderSummaries;
}

In the example above, the controller requires an instance of IRepository<Product>, ProductCudTasks, and IQueryForProductOrderSummaries passed to its constructor from the IoC container. IQueryForProductOrderSummaries is an example of using a query interface, defined in the domain, for providing data-access layer specific needs. It's a very exceptive case and has only been included for illustrive purposes. You'd almost always be able to use specification and report query objects instead...or simply LINQ right off of IRepository<Product>.GetAll().

If you'd like to learn more about dependency injection in ASP.NET MVC 3, check out Brad Wilson's series of posts on the subject. And for learning more about the basics of developing in the web layer, Steve Sanderson's Pro ASP.NET MVC 3 Framework is a great read.

MyStore.Init

This nearly anemic layer has one responsibility: perform generic, application initialization logic. Specifically, the initialization code included with a S#arp Lite project initializes the IoC container (StructureMap) and invokes the initialization of the NHibernate session factory. Arguably, this layer is so thin that its responsibilities could easily be subsumed by MyStore.Web. The great advantage to pulling the initialization code out into a separate class library is that MyStore.Web requires far fewer dependencies. Note that MyStore has no reference to NHibernate.dll nor to StructureMap.dll. Accordingly, there is very little coupling (i.e., none) to these dependencies from the web layer...we like that. Among other things, this prevents anyone from invoking an NHibernate-specific function from a controller. This, in turn, keeps the controllers very decoupled from the underlying data-access mechanism as well.

MyStore.NHibernateProvider

The next stop on our tour of the layers of a S#arp Lite project is the NHibernate provider layer. With S#arp Architecture, this layer would frequently get quite sizable with custom repositories and named queries. With the alternative use of query objects and LINQ on IQueryable<>, this class library should remain very thin. This class library contains three kinds of objects:

NHibernate Initializer,
NHibernate Conventions,
Mapping Overrides, and
(very occasionally) Query Implementations.

Let's look at each in turn.

NHibernate Initializer

NHibernate 3.2.0 introduces a built-in fluent API for configuration and mapping classes, nicknamed NHibernate's "Loquacious" API. This is a direct affront to Fluent NHibernator which (as much as I have truly loved it...a sincere thank you to James Gregory) I feel is headed for obsolescence with these capabilities now being built right in to NHibernate. NHibernate 3.2's Loquacious API isn't yet as powerful as Fluent NHibernate, but will get there soon as more of ConfORM is ported over to Loquacious API. On with the show...

There is one NHibernate initialization class with a S#arp Lite project; e.g., MyStore.NHibernateProvider.NHibernateInitializer.cs. This class sets the connection string (from web.config), sets the dialect, tells NHibernate where to find mapped classes, and invokes convention setup (discussed next). Initializing NHibernate is very expensive and should only be performed once when the application starts. Accordingly, take heed of this if you decide to switch out the IoC initialization code (in MyStore.Init) with another IoC container.

NHibernate Conventions

The beauty of conventions is that we no longer need to include a class mapping (HBM or otherwise) for mapping classes to the database. We simply define conventions, adhere to those conventions, and NHibernate knows which table/columns to go to for what. S#arp Lite projects come prepackaged with the following, customizable conventions:

Table names are a plural form of the entity name. E.g., if the entity is Customer, the table is Customers.
Every entity has an Id property mapped to an "Id" identity column (which can easily be changed to HiLo, Guid, or otherwise).
Primitive type column names are the same name as the property. E.g., if the property is FirstName, the column name is FirstName.
Foreign keys (associations) are the name of the property suffixed with "Fk." E.g., if the property is Order.WhoPlacedOrder, the column name (in the Orders table) is WhoPlacedOrderFk with a foreign key to the respective type's table (e.g., Customers).

There is typically just one convention-setup class in a S#arp Lite project; e.g., MyStore.NHibernateProvider.Conventions. The only convention that isn't supported "out of the box" is a many:many relationship, which we'll discuss more below.

Mapping Overrides

There are times when conventions don't hold up. Examples include:

Many-to-many relationships,
Enum as a property type,
Legacy databases which don't stick to (your) conventions, and
Anytime a convention is not followed for one reason or another.

On the upside, this isn't too many cases...but we need to be able to handle the exceptions. Any exceptions to the conventions are defined as "mapping overrides." Examples of overrides may be found in MyStore.NHibernateProvider/Overrides. To make things easy, if an override needs to be added, simply implement MyStore.NHibernateProvider.Overrides.IOverride and include your override code. The MyStore.NHibernateProvider.Conventions class looks through the assembly for any classes which implements IOverride and applies them in turn. As a rule, I create one override class for each respective entity which requires an override.

Query Implementations

Lastly, the .NHibernateProvider layer contains any NHibernate-specific queries, which are implementations of respective query interfaces defined in the .Domain layer. 97.6831% of the time, this will not be necessary as querying via LINQ, off of IQueryable<>, is the preferred approach to querying. But in the rare case that you need to implement a query using NHibernate Criteria, HQL, or otherwise, this is the layer to house it. A sample has been included as MyStore.NHibernateProvider.Queries.QueryForProductOrderSummaries.

MyStore.Tests

At the end of our tour of the layers of a S#arp Lite project is the .Tests layer. This layer holds all of the unit tests for the project. S#arp Lite projects are generated with two unit tests out of the box:

MyStore.Tests.NHibernateProvider.MappingIntegrationTests.CanGenerateDatabaseSchema(): This unit tests initializes NHibernate and generates SQL to reflect those mappings. This is a great test to run to verify that a class is being mapped to the database as expected; i.e., you can look at the generated SQL (in the Text Output tab in NUnit) to verify how a class is being mapped. As a side-benefit, you can copy the generated SQL and run it to make modifications to the database. Finally, the unit tests saves the generated SQL into /app/MyStore.DB/Schema/UnitTestGeneratedSchema.sql for additional reference.
MyStore.Tests.NHibernateProvider.MappingIntegrationTests.CanConfirmDatabaseMatchesMappings(): This unit tests initializes NHibernate and verifies that every entity maps successfully to the database. If you have a missing column, this test will let you know. It doesn't test everything, such as many-to-many relationships, but certainly 97.6831% of everything else.

For an introduction to test-driven development, read Kent Beck's Test Driven Development: By Example. Going a step further, go with Gerard Meszaros' xUnit Test Patterns and Michael Feathers' Working Effectively with Legacy Code.

In S#arp Architecture, SQLLite was used to provide an in-memory database for testing custom repository methods. Since custom repositories have been mostly relegated to obsolescence, keeping SQLLite testing built-in to S#arp Lite would have been overkill and has been removed to keep things simpler. (Besides, you can always look at the S#arp Architecture code for that functionality if needed.)

What's in the S#arp Lite Libraries?

Most of what's relevant in the S#arp Lite class libraries has already been discussed while going through the sample project, but let's take a moment to see what all is in the reusalbe, S#arp Lite class libraries.

SharpLite.Domain

This class library provides support for the domain layer of your S#arp Lite project.

ComparableObject.cs: Provides robust, hash code generator for any object which inherits from it. (It's much trickier to implement than you think. ;)
DomainSignatureAttribute.cs: An attribute used to decorate properties which make up a class' domain signature.
EntityWithTypedId (defined in Entity.cs): Provides a (non-mandatory) generic base class for your entities, including an Id property and the inclusion of domain signature properties when comparing like objects. It takes a single, generic parameter declaring the type of the Id property; e.g., int or Guid.
Entity.cs: Provides an EntityWithTypedId base class with an Id assumed to be of type int.
IEntityWithTypedId.cs: Provides an interface which may be used to implement your own entity base class.
/DataInterfaces/IDbContext.cs: Exposes an interface for controlling transactions.
/DataInterfaces/IEntityDuplicateChecker.cs: Exposes an interface for checking if an entity is a duplicate of one already in the database.
/DataInterfaces/IRepository.cs: Exposes a very basic repository including Get, GetAll (returning IQueryable), SaveorUpdate, and Delete.
/Validators/HasUniqueDomainSignatureAttribute.cs: An attribute which may be used to decorate a class to ensure that duplicates do not exist in the database having the same domain signature.

SharpLite.EntityFrameworkProvider

The idea of this library is to provide a pluggable replacement for the NHibernateProvider (discussed next), if the team so chooses. This library has not been fully developed yet. Let me know if you're interested in contributing with this effort.

SharpLite.NHibernateProvider

This infrastructural class library provides everything necessary for S#arp Lite projects to communicate with the database via NHibernate.

DbContext.cs: Implements IDbContext for transaction management.
EntityDuplicateChecker.cs: Implements IEntityDuplicateChecker for checking for entity duplicates.
LazySessionContext.cs: Supports NHibernate open-session-in-view; originally written by Jose Romaniello.
Repository.cs: Provides a minimalist base repository...feel free to extend as needed.
/Web/SessionPerRequestModule.cs: Provides the HTTP module in support of NHibernate open-session-in-view.

SharpLite.Web

This class library provides MVC-specific needs for S#arp Lite projects...which consists entirely of SharpModelBinder.cs. This may be viewed as completely optional for your use in S#arp Lite projects.

/Mvc/ModelBinder/EntityCollectionValueBinder.cs: Used by SharpModelBinder for translating a collection of inputs from the form into a collection of entities bound to the containing object.
/Mvc/ModelBinder/EntityRetriever.cs: Used by SharpModelBinder related classes for retrieving an entity from the database without knowing which repository to use a priori.
/Mvc/ModelBinder/EntityValueBinder.cs: Used by SharpModelBinder for translating a drop-down selection into an entity association bound to the containing object.
/Mvc/ModelBinder/SharpModelBinder.cs: Extends the ASP.NET MVC model binder with additional capabilities for populating associations with entities from the database.

Well, that's it in a nutshell for now. I sincerely hope that S#arp Lite will prove helpful to you and your team in developing well-designed, maintainable ASP.NET MVC applications that scale well as the project evolves. This architectural framework reflects lessons learned from years of experience blood, sweat, & tears and countless ideas shamelessly stolen from those much smarter than I.

Enjoy!
Billy McCafferty

Introducing S#arp Lite ...S#arp Architecture's pompous little sister

Billy McCafferty — Wed, 26 Oct 2011 22:59:00 GMT

[This particular post is intended for people who are at least a little familiar with S#arp Architecture and explains the motivations for S#arp Lite. For a comprehensive 101 on S#arp Lite, read the next post. To get right to the code, visit the S#arp Lite GitHub repository, download SharpLite_0.42.zip and follow README.txt.]

In April of 2008, I announced a project I had been working on called S#arp Architecture. This framework, for developing scalable, ASP.NET MVC applications with NHibernate, strove to adhere to the following key principles:

Facilitate Domain Driven Design in project development
Enforce Loose Coupling in project structure
Preconfigure infrastructure (to take care of all the tedious plumbing)
Support open-ended presentation

I've been pleased to have seen S#arp Architecture downloaded thousands of times and used on many many real world projects. (I can count at least a dozen cradle-to-grave S#arp projects under my own belt.) It has also had some terrific leaders and contributors along the way, including Alec Whittington, Seif Attar, Lee Carter, Jon George, Simone Busoli, Luis Abreu, James Gregory, Martin Hornagold, Jay Oliver, Frank Laub, Geoffrey Smith, Dan Smith, Kyle Baley, Chris Richards, and Howard van Rooijen.

Over the years, I've reflected upon what could be done to make S#arp easier to understand, more tenable for a larger audience, simpler to develop with...all the while, keeping it scalable for larger projects. S#arp Lite reflects this effort...

S#arp Architecture with a Lapband

To create S#arp Lite, I started with S#arp Architecture, grabbed a machete, and went to town to whittle the code down to the absolute essentials. After the dust settled, here's what was left of the S#arp libraries:

ASP.NET MVC 3 with Razor as the underlying framework and initial presentation layer (can be swapped out for ASPX);
A base domain object for providing "domain signature," comparison capabilities;
A minimal, base repository including Get, GetAll (returning IQueryable), SaveOrUpdate, and Delete;
Inclusion of NHibernate ORM suppor for convention-based, data mapping; and
An MVC model binder for binding entity associations to an object from a form.

That's about it. What about Fluent NHibernate you ask? Gone. NHibernate Validator? Gone. MvcContrib and SQLLite? Both gone. AutoMapper, Newtonsoft.Json, WCF, Caliburn.Micro, MEF, PostSharp, RavenDB, Specifications? All gone. Although all of those features are handy, I felt that all the extras began to stand in the way of understanding the core principles of S#arp and should be included at the project level (not in the reusable S#arp Lite libraries).

The Disappearing Repository

One of the biggest pain points that my S#arp Architecture projects (and other teams' projects) have experienced over the years is the explosion of repository interfaces, repositories and "pass through" service classes which do nothing but re-route a request from a controller to a repository method. Because the queries were NHibernate-dependent, the repositories wrapped the NHibernate-gory details in concrete repositories, only exposing their querying capabilities via interfaces. Accordingly, you'd end up with hundreds of repository interfaces on large projects, all in the name of hiding NHibernate from the other layers. Although previously necessary, the game has changed.

With the mature, LINQ provider that NHibernate provides, there is no longer a need to wrap queries within repositories. The LINQ queries, run against the repository's GetAll() method, can now be added directly to application-service/task-layer classes or as stand-alone query objects in the domain layer. They don't necessitate a reference to NHibernate and so your layers still remain cleanly separated while being able to do away with all those annoying repository classes.

Adherence to the Original Architecture

One of the primary goals of this effort was to simplify S#arp while remaining in strict adherence with the original architecture. Let's take a step back to see how the architecture looks on a S#arp Lite project.

This is very similar to the overall architecture of S#arp Architecture. The biggest changes in S#arp Lite, with respect to the architectural approach, are as follows:

Controllers have been moved into the .Web project to better reflect their role as part of the presentation layer. (I believe this may have been done in the most recent S#arp as well.)
An .Init project has been introduced to remove extraneous dependencies from .Web. (Although .Init is very small in size, it helps to eliminate many initialization related dependencies from .Web.)
Since there are no more custom repositories, the data-access layer, .NHibernateProvider is very thin and only has the bare essentials for initializing NHibernate. In fact, its role is so reduced, that it could be swapped out with an .EntityFrameworkProvider (more on this below).

If you've used S#arp Architecture before, you'll quickly notice that the overall architecture is almost identical to before...this is very intentional. The architecture has and always will be a pivotal aspect of keeping the layers cleanly separated. The original, architectural layout of S#arp Architecture worked well and has been preserved, for the most part, in S#arp Lite. (S#arp Lite has just been severely trimmed down to the bare essentials.)

Who is S#arp Lite intended for?

S#arp Lite should be usable to a much wider audience. In fact, because of the switch from custom repositories to using LINQ against IQueryable, you hardly even need to know NHibernate. I hate to imply a one-size-fits-all architecture, but S#arp Lite can be used effectively on small projects (mom & pop stores) up to very large "enterprise" applications (e.g., an application which integrates with half a dozen other systems).

Is S#arp Lite backward compatible with S#arp Architecture?

Hypothetically yes, but it would likely be cost and time prohibitive to migrate a S#arp Architecture project to S#arp Lite. With that said, on our existing S#arp Architecture projects, we've added an IQueryable method to our base repository to help with getting rid of new repositories.

Is S#arp Lite going to change very much over the next year?

Absolutely not. It is going to remain almost exactly as is; with only upgrades to internal components when necessary (NHibernate, ASP.NET MVC, etc.). It is very much intended for S#arp Lite to hardly grow at all for the foreseeable future to avoid any bloat. Sure, contrib projects are welcome, but S#arp Lite will not be growing in size. The only anticipated growth of the project is the inclusion of CRUD scaffolding.

What about S#arp Architecture?

S#arp Architecture is still alive and well under Alec Whittington's guiding light; it's primary advantage now and forever more will be its wide array of options and solutions for various scenarios, such as WCF, SQLLite unit testing, PostSharp, etc. These options simply aren't built into S#arp Lite and won't be...if you use S#arp Lite, it is expected that you'll extend it as necessary to meet the specific needs of your project.

Didn't you say something about .EntityFrameworkProvider?

Because S#arp Lite greatly de-emphasizes the need for custom repositories, NHibernate almost becomes a minor infrastructural concern, something akin to log4net. Well, perhaps not that minor of a role, but certainly closer than before. Because of this, S#arp Lite projects are very lightly coupled to NHibernate....so much so that I feel the SharpLite.NHibernateProvider layer could be (easily?) interchanged with a SharpLite.EntityFrameworkProvider using Entity Framework 4.1. I've put the beginnings of such a layer into the S#arp Lite solution; unfortunately, I'm not versed enough with Entity Framework to know the best way to develop this layer, nor how to properly use it to effectively replace MyStore.NHibernateProvider in the MyStore sample project. If anyone is up for an interesting challenge, I would be interested to see if SharpLite.EntityFrameworkProvider could be built out and used as the data-access infrastructure for the MyStore sample project.

I'm bored with all this chatter...what next?

Alright, let's get to the fun stuff. For starters, the S#arp Lite source (and pretty good sample app) is available at https://github.com/codai/Sharp-Lite. Let's create a S#arp Lite project so you can see what it's all about...

Check out the Sample Project & Create your first S#arp Lite project

Download SharpLite_0.42.zip from the downloads tab of https://github.com/codai/Sharp-Lite
Follow the instructions in README.txt

Subsequent posts will go into more detail about S#arp Lite (for a 101 overview) and will begin providing a cookbook for developing your own S#arp Lite projects. (If you're at all interested in helping with the addition of some ASP.NET MVC scaffolding with MvcScaffolding, you're most welcome to contribute!) I look forward to hearing any and all feedback, for better or worse.

Enjoy!
Billy McCafferty

S#arp Architecture 1.6 Released!

Billy McCafferty — Thu, 19 Aug 2010 15:24:00 GMT

The enthusiastic and dedicated Alec Whittington took over as lead developer of S#arp Architecture earlier this year. Since then, he's worked hard to maintain a clear direction, to further improve S#arp Architecture while adhering to its original principles, and to include additional great talent in the core development team including the guys behind the Who Can Help Me project sample (Howard van Rooijen, James Broome, and Jon George). Through a lot of hard work, the team has released S#arp Architecture 1.6. Alec's announcement:

It has been a while since the last release and we have had some nice features added. The biggest items in this release are:

Fluent NHibernate 1.1 - We are now using version 1.1.0.635. The AutoMappingPersistenceGenerator has been updated to use the new bits as well.

NHibernate Configuration caching has now been added

CRUD forms now enable client side validation

This release was almost 100% done by the community, that is amazing! I would like to thank everyone who contributed to this release. I truly appreciate your help and feel that it benefits the community greatly.

Whats Next?

This will be the last version to support VS 2008 as well as .NET 3.5. We will keep a branch available for the community to maintain, but no further development will be done against this branch. The next version will be tagged 2.0, support only VS 2010 and .NET 4.0.

As you may have heard, we are in the process of a partial rewrite. We are working on increasing the code coverage as well as some new bits. While I will not go into details, lets just say one of our focuses is to make sure the installation and solution creation become easier and less problematic. We are also going with a more Who Can Help Me solution structure for not only the S#arp Architecture project, but also the default template. We all felt that this structure further reinforced the overall project goal of good design practices. We are also going to try and deliver better documentation as well as an expanded line of samples. On that note, the Who Can Help Me application as joined the Northwind sample as an official sample. There are many many other things as well, but they are still in the early stages, so we will leave them there for now. Once they get better defined, we will let everyone know.

I needn't state how pleased I am to see S#arp Architecture not just surviving, but thriving. I have immense appreciation to Alec, the core team, and the S#arp community for the continued support and progress of this project.

If you're new to S#arp Architecture, I invite you to learn more at http://wiki.sharparchitecture.net/.

Billy McCafferty

Are additional layers of abstraction warranted?

Billy McCafferty — Wed, 18 Aug 2010 22:27:00 GMT

I was presented with the following discussion opener concerning the management of complexity via the introduction of abstraction...

How do we handle complexity? I asked a few software architects I know and all of them answered, “Abstraction.” Basically they're right, but being a math major in college, there is a principal I believe that software architects and designers miss - complexity is constant. In other words, if you’re designing a system that is inherently complex, you can’t reduce it. In chemistry I remember learning that energy is constant, it can’t be created or destroyed, it's just there. In software, if the solution to a problem is complex, the complexity is always going to be there.

What about abstraction? Abstraction basically hides complexity. This is good, right? The problem is, in a lot of designs, once abstraction hides complexity the designers tend to forget about it. If you work hard at a good design to hide the complexity and forget about it, it will come back and haunt you some day. So what does one do?

I love this discussion! The heart of this is determining if the introduction of abstractions, for the sake of hiding complexity, is truly an overall benefit. In robotics, we're ever dealing with increasing levels of abstraction for this very reason. For example, in Architectural Paradigms of Robotic Control, I briefly discussed the 3T architecture which has three separate layers, implemented as increasing levels of abstraction. E.g., the skill/servo layer would likely be implemented in C++, the sequencing/execution layer might be implemented as a sequence modeling language, such as ESL or NDDL, while the planning/deliberative layer might be implemented with a higher abstraction yet, such as with the Planning Domain Definition Language or Ontology with Polymorphic Types (Opt).

In response to the concerns put forth, I would tend to agree that abstraction hides complexity and that it does make it more likely that you may be bitten by the hiding of the complexity. With that said, encapsulation of complexity into well formed abstractions is an inevitable step to facilitate taking on increasingly complex problems. For example, in the .NET world of data access, Fluent NHibernate is really just a way of hiding the complexities of NHibernate. NHibernate is really just a way of hiding the complexities of ADO.NET. ADO.NET is really just a way of hiding the complexities of communicating to a database via TCP/IP sockets, or whatever underlying mechanism is employed. Along the same vein, tools such as Herbal, NDDL, and ESL are similarly provided as a means to provide an abstraction for hiding complexity.

Because these layers of complexity have been encapsulated in a manageable fashion, we're now able to take on project work which would be far too complex to manage if we were using a lower level of implementation, e.g., pure C++, or Assembly for that matter. Indeed, there will be times when the added layers of abstraction will make it more difficult to tweak a low level capability, but the improved complexity management that the abstractions provide should far outweigh the sacrifice of losing some low level capabilities.

I think the crux for determining if an abstraction is worthwhile:

Does the abstraction reduce complexity of interacting with the underlying layer that it encapsulates?
Does the abstraction make it easier to tackle increasingly complex problems?
Does the abstraction provide enough tweak points to accommodate the 5-10% of times that more low level control is needed?
Does the abstraction increase maintainability and ease of understanding of the overall goals of the system?

If the answers to the above are yes, then I believe that the encapsulation of complexity, codified as a new layer of abstraction, is pulling its weight. Otherwise, you might not want to throw that Assembly language reference away just yet.

Billy McCafferty

Announcing SharpRobotica.com ...the softer side of robotics

Billy McCafferty — Tue, 13 Jul 2010 03:47:00 GMT

After confidentaly turning over the reigns of S#arp Architecture to the adept and au fait hands of Alec Whittington, I capriciously announced that I'd be turning my writing attentions towards robotics...with the diabolical intentions of taking over the world (obviously).

It became apparent after a few posts that the direction I began taking was diverging far from the subject matter of my esteemed colleagues at devlicio.us. Rather than dilute the theme of devlicio.us, I figured it was time to create the proper context for such topics...

It's my great pleasure to announce the humble beginnings of SharpRobotica.com. This blogging community (with an author population of uno at the moment, being yours truly) emphasizes and explores the software side of robotics. While there are many robotic sites, the focus of SharpRobotica.com, accordingly, is in bringing professional software techniques and insight to this vernal, yet burgeoning, industry.

For years, hardware lagged far behind software in the world of robotics; developers were tirelessly constrained by the performance and capabilities of the available hardware platforms. But with the hardware of robotics becoming startlingly sophisticated, there is now incredible opportunity and demand for solid algorithms and software solutions for balancing the scale once again, by creating software which can adeptly take advantage of available robotics hardware to conquer ever challenging contexts such as underwater, space, entertainment, hazardous environments, and in the home.

So if you have an interest in robotics and/or interested in applying your software skills to a completely new context, many of the posts and discussions might be right up your alley. Also, if you're interested in being an author on SharpRobotica.com, sign up as a user and let me know you're interested!

So while I'll still be showing up now and again on devlicio.us...I invite you to also subscribe for a gander to SharpRobotica.com.

Enjoy!
Billy McCafferty

P.S. Thanks Mr. Tompkins (audacious founder of CodeBetter.com and devlicio.us) for the support of this announcement!

Checklist for Developing Message-Based Systems

Billy McCafferty — Tue, 23 Mar 2010 21:22:00 GMT

Even when developing the most basic CRUD application, we ask ourselves a number of questions - whether we realize it or not - during the initial phases of development concerning the architecture and construction of the project. Where will the data be persisted? What mechanism will be used to communicate with the database? How will data from the database be transformed into business objects? Should separated interfaces and dependency injection be employed to maintain a clean separation of concerns between application logic and data access objects? What UI components will be leveraged to speed development of the UI layer? When developing message-based systems (see Message-Based Systems for Maintainable, Asynchronous Development for an introduction) it's immensely helpful to formalize such questions about how the systems will be designed and developed. Creating a checklist of such items to decide upon, and formalizing answers for the application context, helps to:

Standardize how components (applications participating in the message-based system) will communicate with the messaging middleware;
Provide architectural guidance throughout project development concerning how components will leverage message end-points;
Define the data model that will be used for sending commands and exchanging data among components;
Assist new developers on the team with getting up to speed with maintaining and extending the message-based system; and
Support consistency (and predictable maintainability) amongst the components of the system.

This post provides an addendum, if you will (I'm sure you will), to Message-Based Systems for Maintainable, Asynchronous Development for complementing the article with a checklist of questions and architectural topics that should be discussed, decided upon, and/or spiked before beginning development of your budding message-based system. Accordingly, this does not describe a methodology for developing message-based systems, but simply a checklist of topics which should be taken into consideration before development begins. Many of the checklist items below serve as good starting points for team discussion and for the development of architectural spikes for demonstrating implementation details.

Messaging Middleware

What messaging middleware solution will form the background of the message-based system?
How are channels addressed in the selected middleware? (E.g., as a string or as a port number?)
How does the nomenclature of the middleware match up to standard elements of message-based systems and design patterns? E.g., in Robot Operating System (ROS), a channel is called a "topic." Likewise, is their middleware specific nomenclature for elements such as component, publisher, subscriber, point-to-point channel, publish/subscribe channel, message, command message, request-reply message, router, message end-point, and other messaging design patterns?
What mechanism is available for peeking at messages going over a point-to-point channel?
What utilities are available for viewing active channels, publishers, and subscribers?
What other tools does the middleware provide for debugging and observing the messaging system? Will custom utilities need to be developed to autment development and debugging capabilities?

Integration with Messaging Middleware

How will message end-points publish a messages?
How will message end-points subscribe to a channel for messages?
How will message end-points poll a channel?
What technique will be used to decouple the component (participating application) from the message end-point when sending a message? E.g., dependency injection, separated interface, or other?
What technique will be used to decouple the component from the message end-point when receiving a message or polling a channel? E.g., callback, separated interface, or other?
Should exceptions raised by the messaging middleware be wrapped in component specific exceptions for looser coupling? If so, what will be the standard exceptions? Additionally, how will such exceptions be logged and tracked for debugging; e.g., email alerts, logging, exception message channel?

Channels and Routing

What channels will be created and what data type should each channel carry?
Will a hierarchical naming strategy be leveraged for naming and organizing channels? If so, what are the naming conventions for the project?
Has an invalid message channel been setup with a logger to capture and record invalid messages?
How should a predictive router be implemented, when needed?
How should a reactive router (e.g., a message filter) be implemented, when needed?

Message Elements

What data elements may the header of a message contain and how should each be specified? E.g., return address, message expiration, timestamp, origin, format indicator, etc.
How should a message identifier and correlation identifier be generated and included in the header?
What is the messaging data model supported and/or enforced by the messaging middleware? E.g., ROS supports http://www.ros.org/wiki/msg.
What canonical data model will be used to imbue message content with domain specific semantic meaning? E.g., JAUS.
How will message translators and/or message mappers be leveraged for converting between each component's domain model and the canonical data model?
How will a command, document, event and request-reply message be composed?

Other Topics

How will components maintain information that is associated with a correlation identifier?
Are there existing components which need to interact with the messaging middleware which cannot be modified? If so, how will each communicate with the messaging middleware?
How will each component participate with the message based system; e.g., as a polling consumer, as an event-driven consumer, as strictly an event publisher, as a combination of these?
Should exceptions raised by a component participate in a global exception tracking and debugging mechanism?

As stated previously, this checklist is not intended to provide a methodology for developing message-based systems, but should serve as a good basis to make sure "T's are crossed and I's are dotted" when deciding upon the major architectural aspects and development techniques that will be leveraged throughout the development of your (team's) message-based system.

Enjoy!
Billy McCafferty

A Few Thoughts on DDD, DTOs, View Models, Repositories and Separation of Concerns

Billy McCafferty — Sat, 06 Mar 2010 16:41:00 GMT

What I've loved most about developing an open-source project is the ideas that I get from others who look at the work and either A) validate ideas, B) suggest that something stinks, or C) call a royal WTF and force people (e.g., me) to explain ideas more fully. It's usually during these explanation attempts that light bulbs start to come on and ideas are refactored and become more substantiated (or at least more defendable). The S#arp Architecture forums have been a gold mine, IMO, for the discussion of the practical application of balanced DDD techniques on real-world projects. After feedback that I've already gotten on my recent post concerning application services and CQS and listening to and adding my two cents concerning ideas on the S#arp forums, I'd like to share a few ideas (that I shared on the forum) concerning drawing the line among application layers, the management of DTOs and View Models, and a general discussion of separation of concerns. (As Chris Carter correctly (and a tad bluntly) explained, don't take these ideas at face value...use them as a point of discussion and contemplation to come to your own conclusions with your project team...and let me know what you think.)

Every great developer knows that any problem can be solved by adding another layer of abstraction. ;)

Obviously, with additional layers of abstraction and indirection, we also introduce complexity and additional objects which need to be maintained. Over the past year, I have had struggles concerning just how far the "DTO paradigm" should be taken to effectively separate layers from the domain and "best practices" for how they should be used. I don't think the answer can be found in a best practice (but I also hate to leave it simply at "it depends"). I truly feel that the most important practice is for a project development team to decide, and firmly agree upon, which approach will be taken on a project and for the agreed upon approach to be followed with discipline (and enforced via code reviews) by the team.

On a couple of recent projects, the biggest problems we ran into weren't concerning which approach to leveraging DTO for layer separation was "best," but on coming to a firm layering decision and having the team discipline to adhere to those practices. Because of these "gray" areas, which we didn't come to a firm team agreement on, there were inconsistent practices used by the developers which led to rot, particularly between the controller (or code-behind) and application services layer.

So while we can discuss which approach is best or most effective, I think the greater concern is for the team to pick a direction, to stick to it, and to diverge or refactor as a team decision.

But as I said, I don't want to leave it at "it depends." ;) So after reading everyone's ideas here and contemplating further on past projects and the ideas I recently put forth in http://devlicio.us/blogs/billy_mccafferty/archive/2010/03/05/better-application-services-and-cqs-with-s-arp-architecture-1-0-q3-2009.aspx, here are a couple of ideas that I've been trying to adhere to (and/or will be adhering to on my next project). The layers referenced below may be viewed in a package diagram here for better clarity of what's discussed.

Usually, one object should be used to populate a form while a second should be used to collect data from a form. The former would contain information for default/selected values along with drop-down values, for examples, while the latter would encapsulate which options were selected when the user submitted the form. Personally, I use a <EnityName>FormViewModel to populate the form and use an entity to collect from the form...but I see it equally valid to instead use an intermediary DTO to collect from the form if further separation is warranted. Additionally, I feel it is OK for a FormViewModel object to have references to domain objects to make it easier to transfer data to the form for input population. This is an arguable point and would be just as valid to only pass DTOs in the view model for a cleaner separation between view and domain, with the acceptance that there will be more objects and maintenance that comes with that approach.
I believe that view models should be maintained either in the application services layer OR in a separate ViewModels class library if a project team feels the additional separation is warranted (although I can't envision much reason to do so). (I'll be modifying my sample project post to reflect the former.) Either way, they should not be maintained in .Core to keep a very clean separation from domain concerns. While the name "ViewModel" implies that they are a view concern, they only describe what should be displayed, not how it should be displayed. Accordingly, there is still a very clear separation of concerns between the view layer (which the user interacts with) and the view model classes. Furthermore, the application services expose methods which should be usable by any type of client. In line with this, a ViewModel or FormViewModel class does not impart a particular mechanism for the implementation of the view layer - again, it only describes what data should be available to the view. So even if view model classes are maintained in app services, this does not preclude there use by various client types; e.g., they could be equally consumed by ASP.NET, ASP.NET MVC, Spark, SilverLight, etc. (The only caveat is that if the view model classes maintain references to domain objects, then it will be more difficult to expose them for serialization via web services and WCF. Accordingly, a project team should firmly decide if ViewModels may contain references to domain objects.)
I believe that DTOs should be maintained in a separate assembly and have no reference to .Core or be maintained in .Core if warranted with the acceptance of decreased "shareability." DTOs should almost always be serializable; i.e., transferable via web services and WCF. This allows the DTOs class library to be easily shared with a consumer of web services which may expose these objects, for easier deserialization. Some of the motivations for deciding when to use DTOs would be to provide a clean separation between the view and app services (if a project team feels its warranted), to expose data to web service consumers, and to better enable Command/Query Separation.
Repositories may return Entities, Value Objects or DTOs, in addition to other "basic" types; e.g., primitives, arrays, etc. Ideally, repositories would not be used by domain objects (even via their interfaces) and would have their interfaces maintained in the application services layer. If the repository interfaces are maintained in .Core, then this forces DTOs to be maintained in .Core as well with the possibility that domain references may "leak" into DTOs. If the repository interfaces are maintained in .ApplicationServices, then DTOs may be maintained in a cleanly separated library without worrying about the line between DTOs and domain objects being grayed. But data interfaces (and DTOs) may be maintained in .Core if the team feels the simplification is warranted.

With all this said, I still think the most important decision is deciding a path to take for your project, having the project team agree upon it, and maintaining the discipline to stay on course.

Your feedback on these ideas is most welcome!

Billy McCafferty

Better Application Services and CQS using S#arp Architecture 1.0 Q3 2009

Billy McCafferty — Fri, 05 Mar 2010 18:47:00 GMT

Updated 2010.03.09 to reflect small modifications that were decided through subsequent discussions on S#arp forum and other DDD posts.

Obviously, S#arp Architecture is the bee's knees when it comes to developing ASP.NET MVC applications. ;) But as a project evolves and gets larger, "out of the box" S#arp Architecture 1.0 guidance runs into a few pain points. Particularly, there's poor use of the application services layer, the separation between controllers and application services is not very clear, entity listing pages become performance bottlenecks as the domain model gets sizable, there is no command/query separation (CQS) "out of the box", and unit tests require re-occurring maintenance to deal with changes in the number of constructor parameters to controllers and application services. While the amicable and adroit Alec Whittington (who is taking over the lead role from me on S#arp Architecture) is hard at work upgrading S#arp Architecture to accommodate recent dependency upgrades and accommodating ASP.NET MVC 2, I wanted to take a stab at resolving some of the architectural issues that I've run into, on S#arp projects over the past year.

I've developed and included a sample project, built on S#arp Architecture 1.0 Q3 2009, for the following key reasons:

To resolve some "pain points" that develop as S#arp projects grow to large sizes,
To demonstrate better use of the application services layer,
To demonstrate better command/query separation of the entity listing pages for dramatically better performance as the domain model grows,
To create an architectural spike for a new project I'm working on, and
To collect feedback from the S#arp community to determine if this is an appropriate architectural direction for the next release of S#arp Architecture.

Please post any feedback and/or suggestions you may have in the comments below or, more preferably, to the S#arp Architecture forums at http://groups.google.com/group/sharp-architecture.

Setup instructions

Unzip the sample project to a BetterAppServices folder
Create a new database called BetterAppServices
Using SQL Enterprise Manager, run:
1. /BetterAppServices/db/Schema/CreateBetterAppServicesDb_ChangesWillBeLost.sql
2. /BetterAppServices/db/StoredProcedures/CreateGetProductCategorySummaries.sql
3. /BetterAppServices/db/StoredProcedures/CreateGetProductSummaries.sql
Open /BetterAppServices/BetterAppServices.sln with VS 2008
In VS 2008, open BetterAppServices.Web/NHibernate.config and change the connection string to point to your BetterAppServices database
Right click the BetterAppServices.Web project and "Set as StartUp Project"
Run (F5) the project to see everything in action.

Changes from "out of the box" S#arp Architecture 1.0 Projects

This project is more of an architectural spike more than anything else at the moment. Accordingly, the CRUD scaffolding generator has been removed and non-essential unit tests have been removed to focus on the architecture itself. Many of the changes will be incorporated into the S#arp CRUD scaffolding generator; this will either be available in the next release, or will be provided as an add-on, as this new approach does add complexity and introduces a major breaking change to existing 1.0 projects.

Major changes include:

/db Folder Changes
- Added /db/Schema/CreateBetterAppServicesDb_ChangesWillBeLost.sql. This gets auto-regenerated when unit tests are run. The motivation was to have the DB schema automatically maintained while developing.
- Added /db/StoredProcedures/CreateGetEntityNamePluralSummaries.sql. These SPs provide command/query separation for the entity listing (Index.aspx) pages, which frequently became a performance bottleneck.
Cross-Project Changes
- Moved /BetterAppServices.Core/DataInterfaces/* to /BetterAppServices.ApplicationServices/DataInterfaces/. This is the only major, possible breaking change for existing applications. This was done to further remove the potential of domain objects using data repositories directly, and to allow the repositories to return objects from a new Dtos project for command/query separation, among other benefits (dicussed below). Decided not to do this to support domain services which may require the use of repositories and to make upgrading from previous version much simpler in some cases. Having the interfaces in .Core doesn't necessitate that domain objects use them; in fact, my rule of thumb is for all domain objects to avoid the use of repositories unless an exceptive case exists.
- Replaced all uses of "using BetterAppServices.Core.DataInterfaces;" to "using BetterAppServices.ApplicationServices.DataInterfaces;" to support the above mentioned change.
- Replaced all uses of IRepositoryEntityName with IEntityNameRepository. Be default, all entities now have an explicit repository. This avoids the need to manually change from the generic to the explicit, when the need arose, in unit tests and in constructors; which was frequently occurring.
/BetterAppServices.Core/QueryDtos (/BetterAppServices.Dtos)
- Added a Dtos class library to provide an appropriate location for View Models and DTOs. While this project currently has a dependency on BetterAppServices.Core, so that the View Models can contain references to domain objects, this dependency could be severed for better separation between the view and the domain. The caveat is that much more work would be required maintaining a more complete DTO layer and transferring data via DTOs in all of the CRUD pages. Decided that a separate assembly was overkill. Consequently, added this namespace to contain query DTOs for transferring results of "report" queries into objects. With this example project, only the listing page uses "pure" DTOs for much better performance. The other CRUD pages still communicate directly with domain objects to keep things much simpler. This is something that can be argued either way and it should depend on the project needs to determine if a more separated approach is warranted.
- Added EntityNameDto.cs to act as a summary object to be bound to results from stored procedures (e.g., the entity listing pages), or other DTO needs.
/BetterAppServices.ApplicationServices/ViewModels
- Added this namespace and EntityNameFormViewModel.cs to hold data related to adding and updating the EntityName. This object gets populated and passed to the entity form pages, accordingly.
/BetterAppServices.ApplicationServices
- Added reference to BetterAppServices.Dtos so that app services can return DTOs.
- Added EntityNameManagementService.cs. This app service class removes the CRUD logic from the controllers, makes the logic more reusable, and creates an appropriate class to add additional application service logic to.
- Added IEntityNameManagementService.cs. This app service interface makes unit testing more maintainable because you can mock the service interface and not worry about when the concrete class' constructor arguments change.
- Added /DataInterfaces/IEntityNameRepository.cs. As described above, every entity now has an explicit repository interface to avoid changes down the road when custom interfaces would inevitably be introduced.
/BetterAppServices.Data
- Added /BetterAppServices.Data/EntityNameRepository.cs to implement the associated interface. An important thing to note is that method provided invokes a stored procedure to act as a reporting means in line with command/query separation. This is useful for entity listing pages which would frequently, previously, end up loading a huge portion of the domain model to show summary information.
- Added /BetterAppServices.Data/NamedQuery/GetEntityNameSummaries.hbm.xml and set its compile action to "Embedded Resource." This provides the "short cut" for invoking the stored procedure. It could easily be modified to accept paging parameters.
- Added reference to BetterAppServices.Dtos so that repositories can return DTOs for better command/query separation.
/BetterAppServices.Web
- Added reference to BetterAppServices.Dtos to be able show DTOs on web pages.
- Changed ComponentRegistrar.AddCustomRepositoriesTo "BetterAppServices.Core" to "BetterAppServices.ApplicationServices" to reflect the new location of the data repository interfaces.
/BetterAppServices.Web.Controllers
- Added reference to BetterAppServices.Dtos.
/BetterAppServices.Tests
- Modified MappingIntegrationTests.CanGenerateDatabaseSchema to save DB schema to /db/Schema/CreateBetterAppServicesDb_ChangesWillBeLost.sql every time the unit test is run.

To reiterate, many of the changes above can be incorporated into a CRUD scaffolding generator; the focus with this example project is on providing an archtectural spike of the proposed architectural revisions.

Even if you don't use S#arp Archtiecture, this sample project should serve as a good example of using application services and basic use of command/query separation (CQS). Although the CQS in the sample project could be taken much further, I felt that the sample provides a good balance between practical maintainability and a more austere separation of concerns.

Enjoy!

Billy McCafferty

Message-Based Systems for Maintainable, Asynchronous Development

Billy McCafferty — Tue, 02 Mar 2010 04:22:00 GMT

Preface (you know it’s good if there's a preface)

In Architectural Paradigms of Robotic Control, a number of architectures were reviewed including deliberative, reactive, and hybrid architectures. Each of these exhibit a clean separation of concerns with layering and encapsulation of defined behaviors. When implemented, the various capabilities, such as planners and mobility controllers, are encapsulated into discrete components for better reusability and maintainability. A pivotal aspect not discussed in the previous article is how the various system layers and components communicate with each other, such as reporting sensor feedback and sending commands to actuator controllers. Effectively resolving this communication challenge is not only important to robotic systems but to many other industries and domains for the successful integration of disparate applications.

To give credit where credit is due, this article pulls quite heavily from the patterns, taxonomy, and best practices presented in Enterprise Integration Patterns (Hohpe, 2003). This well organized book is chock full of hard learned lessons and solid guidelines for developing maintainable message-based systems. This article should not be seen as an adequate replacement for that book (it’s more like cliff notes with a spackling of robotics bias); indeed, Enterprise Integration Patterns should have a prominent place on your bookshelf if you're developing message-based systems – so read this post and then browse http://www.eaipatterns.com/ while waiting for your copy to arrive to delve deeper.

A Need for Message-Based Systems

A few industries in particular, such as finance, healthcare and robotics, are demanding integration of an intimidating number of separate technologies that may be spread across computers, networks, and/or built upon a variety of technological platforms. Not only is this integration tricky, it can come with a significant cost to performance and maintainability if not implemented correctly. Accordingly, a solution is needed which facilitates loosely coupled integration while accommodating the performance demands of the task at hand. Taking a message-oriented approach to inter-application communications is one such way to accommodate these demands in a maintainable manner without sacrificing performance. This article gives an introduction to developing message-based systems using messaging middleware, describes taxonomy for discussing messaging topics and patterns, and includes a number of best practices.

Before delving further, it's important to clarify a few terms that will be used frequently:

Messaging Middleware (aka, a message bus): a 3rd party application which provides messaging infrastructure and capabilities (e.g., MSMQ, MS Concurrency and Coordination Runtime (CCR), Robot Operating System (ROS)),
Component: a stand-alone application or piece of executable code which communicates with the messaging middleware,
Message-based system: the entirety of the system including all integrated components and the messaging middleware.

As stated, messaging provides one means of facilitating inter-component communications. But as with any design approach, the project requirements must be carefully considered to determine if messaging is the appropriate mechanism for integration. While messaging is robust and facilitates integration, it also adds complexity and indirection. So before deciding to use messaging as the means of integration, consider all component integration options including (Hohpe, 2003):

File Transfer: wherein a component produces files of shared data which other components consume,
Shared Database: each component stores and retrieves data from a common database,
Remote Procedure Invocation: each component exposes specific procedures to be invoked remotely for exposing behavior and exchanging data,
Messaging: each component connects to a common messaging system, using messages to invoke behavior and exchange data.

Determining which integration approach is most suitable to your project's needs is beyond the scope of this article, focusing instead specifically on messaging. In turn, we'll review important elements of developing a message-based system including: message channels, messages, message routers, and message endpoints.

Message Channels

When a component sends information to another component in a message-based system, it adds the information to a message channel. The receiving component then retrieves the information from the message channel. Different channels are created for each kind of data to be carried; having a separate channel for each datatype better enables receiving components to know what kind of data will be retrieved from a given channel. For using a channel, each channel is addressable for sending and retrieving messages to/from them. How a channel is addressed varies depending on the messaging middleware being leveraged, but it's usually a port number or a unique string identifier. As a good practice for keeping channels organized, if string identifiers are available, a hierarchical naming convention may be employed to label channels by type and name; e.g., a channel carrying laser scans might be called "Perception/LaserScans."

There are two basic kinds of message channels:

Point-to-Point Channels (aka - client/server style): routes messages for components to talk directly with other components; e.g., a remote procedure call to another component. A message over a point-to-point channel only has a single receiver; so while the sender may not necessarily know who the receiver is, the sender can rest assured that the message will only be received by one receiver – it’s a FIFO queue.
Publish-Subscribe Channels (aka - broadcast): routes messages for components to publish data and an arbitrary number of components to subscribe to that data. A copy of the message is generated for each subscriber on the channel.

In addition to channels intended to carry information among components, it is a good practice to setup an invalid message channel that bad-formed or unreadable messages may be forwarded to for logging and to assist with debugging.

Messages

With a function call, a simple parameter or object reference may be passed and retrieved by the invoked method, sharing the same memory space. But when passing data between two processes with separate memory spaces, the data must be packaged into a "message" adhering to an agreed upon format which the receiver will be able to disassemble and understand. The sender of the message passes the message via a message channel. The receiver retrieves the message from the message channel and transforms the message into internal data structures appropriate for the task at hand.

A message is made up of two parts:

Header: describes the data being transmitted and details concerning the message itself; e.g., origin, timestamp information, message expiration (if content is time-sensitive), message identifier, correlation identifier, return address, etc., and
Body: the data content that the receiver is looking to use.

When sending a message, the sender intends for the message to be used, or responded to, in a particular way. The intention of the message may be described as being one of the following:

Command Message: invokes a procedure in another application,
Document Message: passes a set of data to another application,
Event Message: notifies another application of a change in state, and
Request-Reply: requests a reply from another application.

Event messages deserve a bit more discussion. In its simplest form, an event message would simply be informational, letting subscribers know that an event has occurred; e.g., a new laser scan is available. If subscribers would like details concerning the event, they would send a request-reply to the sender of the event to provide further details; e.g., the laser scan details. Alternatively, the event could be a document as an event to inform subscribers that an event has occurred along with the details of that event; e.g., a new laser scan is available with laser scan details included. The size of the event details and the rapidity in which the event occurs should be considered when deciding between publishing simple event messages and document messages as events.

Request-reply messages could also use a bit more describing. A request-reply is usually implemented as two point-to-point channels. The first channel delivers the request as a command message, while the second carries the reply back to the requestor as a document message. To keep the replying component more loosely coupled and reusable, the requestor should include a return address indicating the channel that the replier should use to publish the reply. After receiving the reply, a challenge for the requestor is to then correlate the reply to the original request. If the requestor is sending a number of requests in succession, it will likely be difficult to keep clear – if it matters – which request a reply is associated with. To resolve this, every request may include a unique message identifier that the replier would then include as a correlation identifier. (A message could have both a message Id and a correlation Id.) The requestor uses the correlation identifier to “jog its memory” concerning which request the response is for. But frequently, a request-reply is in context of a particular domain object, such as a terrain map or a bank transaction; but the correlation Id doesn’t include such information. To assist, the requestor can maintain a mapping (e.g., hashtable) between message Ids and relevant domain object Ids which are related to the original request. When the reply is received, the mapping may be used to load the appropriate domain objects and take further action, accordingly.

Obviously, it is important that the senders and receivers of a message system agree upon the format that messages will take for clear interoperability, better reusability of components, and extensibility of the system. Consequently, a canonical data model should be well defined that all applications will adhere to. The canonical data model does not dictate how each application's domain model must be structured, only how each application must format data within messages. Message translators are developed to convert the sending application's domain model into the canonical data model before sending a message; receivers of messages then use their own message translators to translate the message into their own domain. This mechanism allows applications built on completely different technologies (e.g., C#, Lisp, and C++) to communicate with each other and exchange data. Many off the shelf messaging systems define their canonical data model which must be adhered to. For example, the Robot Operating System (ROS), which we'll looking at in more detail in subsequent posts, defines their canonical model at http://www.ros.org/wiki/msg. But the canonical data model need not be limited to defining the types of primitives available and how to include them in messages.

Domain-specific canonical data models may augment message formatting rules, adding semantic meaning to the data within a message. For example, the Joint Architecture For Unmanned Systems (JAUS) is a set of message guidelines for the domain of unmanned systems, such as autonomous vehicles. The JAUS guidelines provide domain specific rules for communicating data, such as propulsion and braking commands, sensor events, pose and location information, etc. To demonstrate, JAUS message types include (Siciliano, 2008):

Command: initiate mode changes or actions,
Query: used to solicit information from a component,
Inform: response to a query,
Event set up: passes parameters to set up an event, and
Event notification: sent when the event happens.

A challenge in dealing with canonical data models is how to handle changes to the model. In order to support backwards compatibility of existing components when the canonical data model changes, new message channels could be created to carry the messages adhering to the model; e.g., "Perception/LaserScans_V1" and "Perception/LaserScans_V2." Alternatively, the existing channels could continue to be leveraged to carry messages adhering to different version of the canonical data model. To do so, the message, within its header, would include a format indicator, such as a version number or format document (e.g., DTD) reference. But if a sender knows that receivers of a particular message are mixed in what format is being used, a component would need to send two messages, one for each version of the canonical data model. Certainly, this is an important consideration when deciding which components should (or even can) be upgraded to newer formats, and in what order.

Message Routers

While the heavy lifting of the message routing is handled by the messaging middleware itself, there are times when it is useful to augment the middleware with custom message routers to support unique scenarios.

Suppose the destination of a message may change based on the number of messages that have already passed over a channel. In this scenario, the sender of a message may not know how many messages have been passed over a channel since other senders may have been publishing messages on the same channel. Consequently, a message router may subscribe to the channel to determine where each message should be forwarded to, based on the described business rules. Once the destination is determined, the router would then place the message on a subsequent channel to be delivered to the appropriate destination. This intermediary routing is described as predictive routing as the message router is aware of every possible destination and the rules for routing, accordingly. If the routing is based on content within the message itself, such as threshold values, then the custom router is known as a content-based router.

A drawback to using message routers is that if the routing rules change frequently, the message router will need to be modified just as often. To help remedy this, if the rules are expected to change frequently, configurable routing rules (e.g., via XML) could be employed to enable easier management of routing rules.

Let's now consider another scenario wherein it's left up to the subscribers to determine which messages they're interested in; i.e., subscribers will be responsible for filtering out the messages they're uninterested in. In this reactive routing scenario, each subscriber would provide a respective message filter which is similar to a message router, but simply forwards, or does not forward, a message onto a subsequent channel that the destination subscriber is listening to. Frequently, message filters decide to forward, or not forward, based on content in the message itself; e.g., only forwarding orders that have a coupon included. While being similar to a message router in basic functionality, a message filter only has one possible channel to forward the message onto.

Deciding between predictive and reactive filtering must take into account a number of considerations. Is the message content sensitive? Do you need to minimize network traffic? Do you need to be able to add and remove subscribers easily? Is the predictive router becoming a bottleneck of message dissemination? For further guidance on selecting among routing options, see (Hohpe, 2003), ppg. 241-242.

Message Endpoints

Each messaging middleware option (e.g., CCR, ROS) has unique requirements for communicating with it. Each has its own API, its own means of addressing channels, and its own rules for packaging messages. Ideally, the components of the system should not be aware of the specifics of communicating with the messaging middleware. Furthermore, while unlikely to occur, the middleware should be able to be replaced with another, requiring little, if any, changes to the components. Accordingly, the components must be loosely coupled to the messaging middleware. Message endpoints provide the bridge between the domain of each component and the API of the messaging middleware.

Message endpoints are similar in nature to repositories when communicating with a database. Repositories encapsulate the code required to communicate with a database to store and retrieve data while being able to convert information from the database into domain objects. If the database changes, or if the mechanism for database communication changes (e.g., ADO.NET to NHibernate), then, ideally, only the repositories are affected. The rest of the application knows little about database communications outside of the repository interfaces. Likewise, components of a message-based system should not be aware of messaging details outside of the message endpoint interfaces, which provide the means to send and receive data. The message endpoint accepts a command or data, converts it into a message, and publishes it onto the correct channel. Additionally, the message endpoint receives messages from a channel, converts the content into the domain of the component, and passes the domain objects to the component for further action. Internally, the message endpoint implements a message mapper to convert between the component domain objects and the canonical data model.

While posting to a channel is rather straight forward, a message endpoint may receive a message by acting as a:

Polling Consumer: wherein the receiver looks at a channel on a regular basis for new messages and/or as soon as it completes the processing of a previous message. This frees the receiver from having to deal with messages as soon as they arrive on the channel in favor of dealing with messages went it’s ready and willing. A consideration to keep in mind is that messages may queue up on a channel while waiting to be retrieved by the polling consumer. Additionally, a polling consumer may take up threads and resources while polling a channel, even if the channel is empty.
Event-Driven Consumer: wherein the message is given to the receiver as soon as it arrives on the channel. The benefits to this include avoiding messages queuing up while being able to process messages asynchronously. But the receiver is no longer in control of the timing in which it processes messages and must handle messages as soon as they arrive on a channel.

Many messaging middleware solutions include support for transactions that the message endpoints may leverage, making the endpoints transactional clients. To illustrate the need for this, suppose the receiver of a request-reply command crashes just moments after the command message is consumed and removed from the channel. When it recovers, the command message is lost and the sender will never receive a reply. Using a transaction, the command message is not removed from the channel until the response is completed and sent. Committing the transaction removes the command message from the channel and adds the reply document message to the reply channel.

There are a few recommendations which should be considered when developing message endpoints. In accordance with the SRP, message endpoints should be able to receive messages or send messages, but not both in the same message endpoint. Furthermore, a message endpoint should only communicate with one message channel. If a component needs to send a message on separate channels, it would leverage multiple message endpoints to do so. If your components are developed in line with DDD, it would be each component's application services layer which would communicate with the message endpoints, preferably via their interfaces instead of concrete instances. This facilitates swapping out the message endpoints with mock objects for unit testing. Leveraging separated interfaces and dependency injection helps enable this approach. While these guidelines introduce more objects and indirection, they are proven practices for increasing maintainability of the component and reusability of the message endpoints.

Performance of Message Based Systems

One would likely be quick to assume that developing a message-based system has a huge cost to performance. Domain objects are converted to messages, messages are passed over channels, routers and filters intercept and forward messages, and messages are converted back into domain objects...this sounds like a heck of a lot going on. But because each component executes in its own thread or process, they need not wait for other components to complete their job before being able to move on to handling another message or task.

In the figure above (adapted from Hohpe, 2003, pg. 73), note that a sequential process requires that each message life cycle is completed in full before moving on to the next message. But in an asynchronous, message-based system, each component can move onto a subsequent message just as soon as it has completed its part in the last one. This greatly compensates for the extra overhead imparted by the messaging infrastructure.

With that said, there are some component responsibilities which take a long time to complete and may impede the speed by which messages are processed. For example, imagine a component which takes images from a web cam and extracts information such as human figures or road signs. This is likely a time consuming process and would impact the turn around time in which the component could process subsequent messages. To alleviate this bottleneck, multiple instances of the same component may subscribe to the same point-to-point channel. Recall that a point-to-point channel ensures that each message only has a single receiver. The instances of the component then become competing consumers of each message. So if one instance of the component is still processing an earlier message, another instance can grab the next message that arrives for concurrent processing. Power in numbers!

Monitoring and Debugging

Due to the widely asynchronous nature of message-based systems, attention must be given to monitoring and debugging techniques to observe system behavior and iron out problems.

Logging message content is an invaluable measure towards getting a clear look at what communications are taking place. To facilitate this, logging could be added directly into sending and receiving components, but logging message content should not be a concern of components; accordingly, using a monitoring utility to log such information is a cleaner separation of concerns. Certainly a benefit of publish-subscribe channels is that a monitoring component may subscribe to all messages and log the information to a file or console window. While it’s just as valuable to monitor messages on point-to-point channels, if a monitor were to consume a message over a point-to-point channel, the message would be noted as consumed and would no longer be available to the intended receiver. Because such monitoring capability is so helpful in developing and debugging, many messaging middleware options include a "peek" option which allows a monitoring utility to review message contents on a point-to-point channel without actually consuming the message. This capability should be taken into consideration when comparing messaging middleware alternatives.

In addition to monitoring message content sent over channels, it’s also assistive to monitor active subscriptions to various channels to accurately determine which components are receiving which messages. This capability is typically built into messaging middleware solutions and is very assistive during development.

It’s possible that even if a message is routed appropriately, the receiving component may not know what to do with the message due to an incorrectly formatted header or body content. Invalid messages such as this should be forwarded to an invalid message channel which an error logging utility would monitor and log, accordingly. An invalid message channel is setup just like any other channel but with the intention of exposing such messages for debugging purposes.

Closing Thoughts

Developing asynchronous, message-based systems requires a paradigm shift from the more traditional, synchronously executed applications that most people are familiar with. It is not appropriate for every application domain and should be seen as one additional architectural option to consider when developing applications. But in some domains, such as robotics or in the integration of disparate applications, this approach to development is absolutely pivotal in providing responsive behavior without sacrificing maintainability of the overall system. Indeed, by splitting responsibilities into discrete components, loosely coupled to each other via messaging middleware, immensely complex problems can be broken down into understandable chunks while being flexible enough to accommodate changes to the underlying middleware or introduction of new components.

In the next couple of posts, we'll look at a checklist for developing message-based systems followed with examples in CCR and ROS.

Enjoy!

Billy McCafferty
http://wiki.sharparchitecture.net

P.S., for S#arp fans, we're trying to get another release together for the coming week...and this ain't no quarterly release. ;)

References

Hohpe, G., Woolf, B. 2003. Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions.

Siciliano, B., Khatib, O. 2008. Springer Handbook of Robotics.

Architectural Paradigms of Robotic Control

Billy McCafferty — Wed, 03 Feb 2010 22:39:00 GMT

Introducing a New Blog Series

Rarely do I make new year's resolution. But I found myself pinky swearing myself a single resolution for this year: write a series of tractable posts concerning the development of software for robotics. Accordingly, I intend to introduce background, techniques, and examples with respect to the following topics and more:

Software Architectural Approaches for Robotics (e.g., Deliberative, Reactive, Hybrid)
Software Design Methodologies (e.g., for Behavior Based Robotics, for Hierarchical)
Integration Patterns (Message Channels, Publish/Subscriber, RPC)
Machine Learning and Adaptation Techniques (Neural Nets, Genetic Algorithms, Fuzzy Logic)
Planning Algorithms (Discrete Planning, Continuous Spaces, Motion Planning)
Simultaneous Localization and Mapping Techniques (Kalman Filters, Particle Filters)
Tools & Frameworks (Microsoft Robotics Studio, Robot Operating System)

I'm writing this series of posts for two reasons. The first, as always, is to bounce ideas off readers as a sanity check. The second is to instill interest in readers in this rapidly evolving field. The sophistication of software algorithms within robotics is beginning to lag behind the potential capabilities of available hardware. Accordingly, if this series acts as motivational tool to get others involved in tackling the current algorithmic challenges involved in robotics, then all the better.

To make this series more interesting and applicable to the development audience that reads http://devlicio.us, I will attempt, when appropriate, to focus on patterns and general techniques that could possibly be applicable to scenarios outside of robotics. Additionally, due to this series' focus on software, I'll concentrate on leveraging simulators over physical, often expensive, robotics hardware to make it easier for readers to duplicate provided examples.

Enjoy!

Introduction to Architectural Paradigms of Robotic Control

While "architecture" is likely one of the least definable terms in software development, it is unavoidably a topic which has one of the greatest impacts on the extensibility and maintainability of an application. Indeed, a frequent cause of an application re-write is due to the architectural decisions that were made early in the project, or the lack thereof. Certainly one of the difficulties in instituting proper architectural practices in a project lies in the fact that architectural decisions must carefully be decided at many different levels of project development. Obviously, the architectural decisions at each level should be made in context of the project requirements in order to facilitate a proper balance of speed of development, scalability of development, and maintainability of the application in the future.

To illustrate, consider a basic eCommerce site vs. a corporate financial management application which integrates with various third party tools. There are two aspects of architecture which must be carefully considered. The first is deciding which project contexts will require architectural consideration. The second is defining the architectural approach to meet the needs of the given project context, and implementing any necessary prefactoring, accordingly.

For a basic eCommerce site, the project contexts requiring architectural consideration would likely include: appropriate separation between presentation and control, separation between control and the domain, separation between the domain and data access, and appropriate integration with a payment gateway. Perhaps the active record pattern might be chosen as the data access mechanism. If inversion of control seems overkill for such a small application, perhaps direct communications to the payment gateway via the domain objects might be chosen as the means of integration. Accordingly, judicious prefactoring would suggest that an architectural spike be created demonstrating appropriate use of the active record pattern with a selected tool along with an example of how and where integration with the payment gateway would take place.

For the more complex and demanding needs of the financial management application, architecture considerations, in addition to those taken into account for the basic eCommerce site, might include: what integration patterns might be leveraged to facilitate client integration, messaging patterns that might facilitate server side integration with third party financial calculators, and where inversion of control is appropriate. Perhaps RESTful services would be included to provide integration support for clients requiring such capabilities. Messaging via a publish/subscribe mechanism using a composed message processor might be selected as the ideal means for coordinating data with third party vendors on the server side. Again, proper refactoring, from an architectural perspective would include developing architectural spikes along with the foundational pieces of such an application to provide appropriate guidance for developers assisting with the project.

Developing software for robotics is no different, in this respect, than developing any other application. One must decide where the key architectural decisions need to be made and then provide adequate decisions and guidance to facilitate development in accordance with those decisions. As stated, these decisions must be made in consideration of the project needs. This post focuses on one project context, the architectural context of determining the overall approach to robotic motor control. We'll delve into three major architectural approaches to motor control along with highlighting a few specific implementations.

Deliberative vs. Reactive Robotics

Certainly the best source of intelligent systems to study and emulate are those found in nature. From the humble ant to the exalted human, evolution has honed a variety of strategies for dealing with the physical world. Accordingly, if we are to create intelligent systems, a good place to start is by emulating the behaviors and responses to stimuli demonstrated by living creatures.

In the early days of robotics, it was presumed that the most effective approach to emulating intelligence was by taking in detailed measurements provided by sensors, creating an internal representation of the world (e.g., grid-based maps), making plans based on that internal representation, and sending pre-planned commands to actuators (devices which convert energy into motion) for moving and interacting with the world. Inevitable, this approach presumes that the internal representation of the world is highly accurate, that the sensor reading are precise, and that there is enough time to carry out a plan before the world changes. In other words, this approach is highly deliberative. Obviously, the world is highly dynamic, sensor readings are sometimes erroneous, and time is sometimes of the essence when interacting with the world. E.g., you probably wouldn't want to spend much time creating an internal representation of the world if there's a Mac truck speeding towards you.

At the other extreme from deliberative is a reactive approach to interacting with the world. Taking a purely reactive approach does away with plans and internal representations of the world altogether; instead, a reactive approach focuses on using the world as its own model and reacts, accordingly. Instead of plans, reactive approaches often rely upon finite state machines to modify behavior as the world changes. Rodney Brooks changed robotics thinking upside down when he introduced this paradigm shift in the 80s. The robots he and his team produced were much faster in their reaction times, when compared to deliberative robots, and exhibited surprisingly complex behavior, appearing quite intelligent in many scenarios. But as is any extreme, a purely reactive approach to the world had it's own drawbacks; its difficulty in managing complex scenarios which demand careful planning is one such example.

The figure at right illustrates some of the differences between deliberative and reactive systems. Adapted from (Arkin, 1998).

While there are certainly pros and cons to either approach, there are some scenarios which are appropriate to one or the other. While still others may require more of a hybrid approach. Let's now take a look at each approach in more detail.

Deliberative/Hierarchical Style

One of the first successful architectural implementations on a mobile robotic platform was the well known robot known as Shakey, created at Stanford University in the 1960s. Of particular note was the robot's architecture which was made up of three predominant layers: a sensing layer, a planning layer, and an execution layer. Accordingly, as information would be made available by sensors, Shakey would make plans on how to react to the perceived world and send those plans on to the execution layer for low level control of actuators. This began the architectural paradigm known as sense-plan-act (SPA). As the SPA approach matured, additional layers were introduced in a hierarchical style, typically all of which having access to a global world model. The upper layers in the hierarchy would use the world model to make plans reaching into the future. After plan development completed at the highest levels, the plans would be broken down into smaller commands and passed to lower levels for execution. Lower layers would then further decompose the commands into more actionable tasks which would ultimately be passed on to actuators at the lowest level. In a hierarchical approach, the sensing layers participate in keeping the internal representation of the world updated. Furthermore, to better accommodate dynamic changes to the world environment, lower planning layers may suggest changes to plans based on recent changes to the world model.

The diagram at right roughly illustrates this hierarchical architectural approach. As should be noted, the sensing layers update the world model while a hierarchy of planning and execution layers formulate plans and breaks those plans down into actionable tasks. Adapted from (Kortenkamp, 1998).

4D/RCS

The current pinnacle of hierarchical architectures may be found in the reference architecture known as 4D/RCS (4 Dimensional / Real-time Control System). 4D/RCS is the latest version of the RCS reference model architecture for intelligent systems that has been evolving for decades. With 4D/RCS, six (more or less) layers are defined for creating and decomposing plans into low level action:

Unit/Vehicle/Mission Layer: this layer decomposes the overall plan into actions for individual modules. Plans are regenerated every 2 hours to account for changes to the world model.
Module Layer: converts actions into tasks to be performed on specific objects and schedules the tasks, accordingly. Plans are regenerated every 10 minutes.
Task Layer: converts actions on a single object into sequences of moves to carry out the action. Plans are regenerated every minute.
Elemental Layer: creates plans for each move, avoiding obstacles and collisions. Plans are regenerated every 5 seconds.
Primitive Layer: determines motion primitives (speed, trajectory) to facilitate smooth movement. Plans are regenerated every 0.5 second.
Servo Layer: sends servo control commands (velocity, force) to actuators. Plans are regenerated every 0.05 seconds.

At the heart of each and every layer are one or more 4D/RCS nodes which contain a behavior generation process which accepts commands from the next higher level and issues subgoals to the behavior generation process at lower levels. Furthermore, each node reports its task execution status up the chain for consideration into further planning. While plans are being solidified and disseminated, a sensory processing process in each node receives sensory input from lower levels, updates the world model, and collates the sensory data into larger units which it passes on to the next higher level; e.g., points get converted to lines which get converted to surfaces which get converted to objects with each successive rise through the 4D/RCS layers A visual summary of a 4D/RCS node quickly shows just how complex such a system can become. But sometimes the demands of a task require a respective amount of sophistication in the architectural approach. A comprehensive review of the 4D/RCS approach is discussed in (Albus, 2006).

The primary advantage to deliberative, hierarchical approaches such as 4D/RCS is that competent plans for managing complex scenarios can be generated and broken down into smaller chunks for lower levels to execute. But an obvious disadvantage to this comes in the form of much added complexity to the overall system while penalizing the speed at which the system can react to a changing environment.

Reactive Style

In the 1980s, Rodney Brooks, in an effort to overcome some of the limitations of sense-plan-act that Shakey and other such robots exhibited, introduced the concept of reactive control. "Simply put, reactive control is a technique for tightly coupling perception and action, typically in the context of motor behaviors, to produce timely robotic response in dynamic and unstructured worlds." (Arkin, 1998). In other words, by eliminating the reliance on maintaining an internal world model and avoiding large amounts of time generating plans, simple responses can be executed in reaction to specific stimuli, thus exhibiting behavior similar to that of living organisms. If there was any doubt in what Brooks was trying to imply, he boldly titled one of his works, Planning is Just a Way of Avoiding Figuring Out What to do Next. This paradigm shift away from planning turned sense-plan-act into a simpler sense-act.

Accordingly, in a sense-act paradigm, the primary focus is in carefully defining behaviors and the environment stimulus which should invoke those behaviors. This focus is deeply rooted in the idea of behaviorism; from a behaviorism perspective, behavior is defined in terms of stimulus and response by observing what an organism does. This approach of developing robotics around such behaviors, unsurprisingly enough, is commonly referred to as Behavior Based Robotics. By leveraging simple state machines, which we'll examine below, to define which behaviors are active for a given stimuli, the overall architectural complexity is greatly reduced while giving rise to responsive, seemingly intelligent behaviors. (The question of whether or not the robot is truly intelligent and/or self-aware is a debate that I'll leave to others such as Searle and Dennett.) While the reactive approach took the spotlight for a number of years, and is still very appropriate in some cases, its limitations in managing complex scenarios and performing sophisticated planning indicated that this approach was not the end all panacea for all situations.

The diagram at right clarifies that reactive systems remove planning and deal with sensory input within the context of each behavior. I.e., "Behavior 1" has no knowledge of the feedback coming from "Sensor 3." Adapted from (Kortenkamp, 1998).

A Design Methodology for Reactive Systems

Due to the simplistic nature of reactive systems, the corresponding design methodology for developing such systems is rather straight-forward (Kortenkamp, 1998):

Choose the tasks to be performed. E.g., find kitchen.
Break down the tasks in terms of specific motor-behaviors necessary to accomplish each task. E.g., find wall, follow wall, recognize kitchen.
For each action, find a condition under which each action should be taken. E.g., wall found.
For each condition, find a set of perceptual measurements that are sufficient to determine the truth of the condition. E.g., long straight line connected to obstacle.
For each measurement, design a custom sensory process to compute it. E.g., bumper sensor activated, edge length over threshold found.

When implemented, we find that the defined behaviors can be realized as states within a finite state machine and that found conditions act as the mechanism for changing from one behavior state to another. What's missing is an arbitration mechanism to determine which behavior wins out in light of competing conditions. Let's briefly look at implementation approaches to reactive systems along with how such arbitration is achieved.

Subsumption & Motor Schemas

A key concern with reactive, behavior based robotics is determining which behavior should take precedence if conditions exist to activate two or more behaviors. An arbitration mechanism needs to be introduced to resolve such competing situation. Brooks dealt with this issue by proposing an architecture known as subsumption. Simply enough, a subsumption architecture still uses a finite state machine to codify behaviors and transitions, but introduces behavioral layers which can provide input to other layers and override behaviors of lower layers. The advantage is in facilitating more sophisticated behaviors as a sum of "lesser" behaviors. It's better understood by reviewing the following example from (Arkin, 1998):

In the example, there are three behavioral layers: Avoid-Objects, Explore, and Back-Out-of-Tight-Situations. The Avoid-Objects Layer's responsibility is to avoid objects by moving away from any perceived obstacles. The Explore Layer's responsibility is to cover large areas of space in the absence of obstacles. The highest level assists the robot in getting out of tight spaces if the lower layers are unable to do so. Each discrete behavior can invoke transitions to other behaviors and/or provide input or advice to subsequent behaviors based on perceived conditions. The "trick" of subsumption is that higher levels can suppress commands between lower level behaviors; consequently, higher layers are able to handle more complex scenarios by manipulating lower level behaviors in addition to its own. To illustrate suppression, note, in the above example that the "Reverse" behavior in the Back-Out-of-Tight-Situations Layer suppresses any commands that the "Forward" behavior is sending to the actuators. By doing so, complex behaviors may emerge using a number of basic behaviors and a relatively simple architectural approach.

A major challenge in using a subsumption architecture is deciding the appropriate hierarchy of behavioral layers. In more complex scenarios, it quickly gets sticky deciding which layer should have the right to suppress which other layers. (Otherwise known as spaghetti-prone.) Additionally, since the layers have bi-directional dependencies amongst each other, in order to provide input and suppression, changes to layers can have large impacts on other layers, often resulting in shotgun surgery with any change. Ronald Arkin introduced a subsequent architecture to address these, and other concerns, with an approach known as Motor-Schema based control which does away with arbitrating competing behaviors. Each active behavior in a Motor-Schema based control system calculates a vector to be carried out by an actuator (e.g., wheels or arm). Using vector addition, a final vector for each actuator is computed and sent for execution. With this approach, the output of behaviors is combined instead of arbitrated. This avoids the need to determine suppression hierarchies and makes a more extensible application, within the limits of behavior based robotics.

Hybrid Style

Most modern architectural approaches to robotics control attempt to combine the planning capabilities of deliberative systems with the responsiveness of reactive systems. Appropriately enough, this is referred to as a hybrid style. While the deliberative approach takes a sense-plan-act perspective and the reactive approach follows with plan-act, a hybrid approach typically takes the form of plan, sense-act. So while the sense-act layers carry out behaviors, the deliberative planning layer can observe the progress of reactive behaviors and suggest direction based on reasoning, planning and problem-solving. The diagram at right crudely demonstrates this combination of the two approaches. Adapted from (Kortenkamp, 1998).

Three-Layer Architecture (3T)

James Firby's thesis proposing Reactive Action Packages (RAPs) (Firby, 1989) provided a solid approach for integrating deliberative planning with reactive behaviors in the form of a three layer architecture. A multitude of subsequent architectures have emulated a similar approach, coming to be known as 3T architecture. From Eran Gat's essay Three Layer Architecture (Kortenkamp, 1998):

Three-layer architectures organize algorithms according to whether they contain no state, contain state reflecting memories about the past, or contain state reflecting predictions about the future. Stateless sensor-based algorithms inhabit the control component. Algorithms that contain memory about the past inhabit the sequencer. Algorithms that make predictions about the future inhabit the deliberator. ... In 3T the components are called the skill layer, the sequencing layer, and the planning [deliberative] layer.

As implied by being a hybrid approach, this 3T architecture is not mutually exclusive to behavior driven robotics. Indeed, the skill layer itself is made up of unique behaviors which resemble that of reactive systems. The sequencing layer then uses the world model to determine when a change in behavior is required. The planning layer can then monitor the lower layers and perform more deliberative, time consuming processes such as path planning. The primary variation amongst 3T implementations lies in the decision between whether the planning layer "runs the show" or if the sequencing layer takes command and invokes the planning layer only when needed. Atlantis is one such example that leaves primary control to the sequencing layer to invoke the planning layer. Other examples of 3T implementations include Bonasso's Integrating Reaction Plans and Layered Competences through Synchronous Control and Standford's Junior which took 2nd place in Darpa's Urban Challenge.

Wrapping Up

The intention of this article has been to provide an introductory overview of paradigms of robotic control including deliberative/hierarchical systems, reactive systems, and hybrid systems. While each approach is appropriate in select contexts, a hybrid architecture is very adaptable for accommodating a large variety of robotic control scenarios with sufficient planning and reactive capabilities. The interested reader is encouraged to dig further into the references and links provided to learn more about these various approaches and design methodologies for implementation.

In the next post, we will examine messaging strategies to facilitate the communications amongst the layers and components of robotic systems.

References

Albus, J., Madhavan, R., Messina, E. 2006. Intelligent Vehicle Systems: A 4D/RCS Approach.

Arkin, R. 1998. Behavior Based Robotics.

Firby, J. 1989. Adaptive Execution in Complex Dynamic Worlds.

Kortenkamp, D., Bonasso, R., Murphy, R. 1998. Artificial Intelligence and Mobile Robots.

Real World S#arp Architecture

Billy McCafferty — Wed, 03 Feb 2010 21:04:00 GMT

With any framework, technology, tutorial, book, idea, <insert pedagogical sources here>, it's often difficult to figure how the technique should be used in the real world. Sure, things sound great when the scope of your project deals cleanly with Customers, Orders and Order Items; and sure, auto-binding works great in a nicely controlled scenarios...but how should things work in the "real world." No, not the Real World in which Brooke has a complete meltdown in season 18. I'm talking about the real world in which many scenarios aren't clean cut in determining where the controller should end and the application services layer should take over; the world in which you're trying to bind from unique data collection mechanisms to appease odd requests from the client; the world in which even the GoF would retort with "heh, that's a tricky one." Yes, we're faced with these kinds of dilemmas on a frequent basis. What I find to be truly helpful is to look at full blown, real-world applications. While a real-world application is never perfect, you can often find great gems for dealing with tricky situations and using patterns and complex scenarios. Howard van Rooijen has released just such a real-world application demonstrating the use of S#arp Architecture, AutoMapper, ELMAH, Spark View Engine, and other great tools.

In Howard's own words... "A few months ago I wrote an email to the community about a site we had just launched – http://fancydressoutfitters.co.uk that used S#arp Architecture at its core along with a whole myriad of other Open Source Frameworks and Tools (Spark, AutoMapper, PostSharp, xVal...).

In the run up to the festive period, myself and two of the development team – Jonathan George & James Broome, decided that in the spirit of giving, we wanted to gift something back to the communities that gave so much to us throughout the year; so we decided to build a new sample web application to showcase the use of these various frameworks & tools called “Who Can Help Me?” which is based on the same architectural style as http://fancydressoutfitters.co.uk.

Who Can Help Me? started out as a small web application I built a few years ago to solve a small and specific business problem within our consulting organisation (and to test out .NET 3.5, LINQ to SQL, ASP.NET WebForms & MS AJAX!). The problem was, that as the organisation grew and new members of staff started, they found it difficult to find the right people who could help them solve specific problems they’d encounter in their consulting gigs. As I have worked for the organisation for a long time (>9 years) I generally knew everyone, had worked with most of them and knew what their areas of expertise were, thus I’d get a few calls every day asking “Do you know anyone who knows about X that could help me?”. The solution was to create a searchable skills matrix that would allow people within an organisation find other people who had specific skills or expertise who could help them solve a particular problem.

So Jonathan, James & I decided to re-write the Who Can Help Me? from scratch, using the architecture style, frameworks and tools we used to build http://fancydressoutfitters.co.uk - it might seem like we've massively over-complicated the architecture for such a simple application - but we really wanted this to demonstrate some of the concepts & techniques we used to build a full scale, public facing enterprise web application.

Who Can Help Me? utilises the following:

The project is currently hosted at Codeplex: http://whocanhelpme.codeplex.com/ and we’ve also released a live demo: http://who-can-help.me. We’ve added some documentation on the Codeplex homepage and will continue to refine this and augment it with blog posts covering some topics in more depth – so if you’re interested – please keep an eye on the following blogs / twitter:

http://howard.vanrooijen.co.uk/blog/ | @HowardvRooijen
http://jonathangeorge.co.uk/ | @jon_george1
http://jamesbroo.me/ | @broomej

Thanks very much for this contribution Howard! Besides myself, I'm sure others have found this project helpful and will continue to do so in the future.

Enjoy!

Billy McCafferty