Rob Reynolds - The Fervent Coder : Development, RoundhousE

RoundhousE–Intelligent Database Migrations And Versioning

Rob Reynolds — Sun, 23 Oct 2011 15:21:00 GMT

“Because everyone wants to kick their database, but sometimes kicking your database is a good thing!”

Many would not argue that you should version your code, and few would argue against versioning your code in a way that can lead back to a specific point in source control history. However, most people don’t really think of doing the same thing with your database. That’s where RoundhousE (RH) comes in.

I have been working on RH for over two years now and people always wander what it is, why and what sets it apart from other migrators. We set out to make a smart tool for migrations that came somewhat close to Ruby’s ActiveRecord Migrations without going the code migrations route (yet). Hopefully this introduction will help you understand why it is different and whether it’s something that is in line with your needs.

What is RoundhousE?

RoundhousE (http://projectroundhouse.org) is a database migrator that uses plain old SQL Scripts to transition a database from one version to another. RoundhousE currently works with Oracle, SQL Server (2000/2005/2008/Express), Access, MySQL, and soon SQLite and PostgreSQL. It comes in the form of a tool, MSBuild, and an embeddable DLL. While someone is working on a GUI, there is no visual tool at the current time.

What sets RoundhousE apart from other migrators?

It subscribes to the idea of convention over configuration, which means you can pass the migrator very few configuration options to get it to work (rh.exe /d dbname), but pass as many options as necessary to meet your conventions. Say you don’t like the tables or folder names that RH uses, you can override those to whatever you want.

RH versions the database how you want it versioned. You can supply it with a DLL path for it to pull the file version from. You can give it an XML file and XPath, or you can use the highest script number in the up folder. You can also just use a sequence based (non-global) form of passive versioning. https://github.com/chucknorris/roundhouse/wiki/Versioning

RH believes in low maintenance and keeping good clean history in your source control. This means that you don’t lump everything into one folder, you put your anytime scripts (views/functions/stored procedures/etc) into their own folders and track history as you go. RH is smart enough to only run these if they are new/different from the current existing scripts in the database.

RH has three modes of operation. Normal, DropCreate, and Restore. Notice none of those are Create like you may see in other migrators. If the intent in the end is to have a database ready to go, why would you want to have to make a step to specify that you want to create the database? RH is smart enough to realize that the database doesn’t exist and it creates it (unless you pass a switch explicitly telling it not to). Normal is just the migration as it is. DropCreate is used during development when you want to continually change the same scripts prior to production. Restore is used when you switch to maintenance mode and want to change the same maintenance script. https://github.com/chucknorris/roundhouse/wiki/RoundhousEModes

RH is environment aware, which means you can have environment specific scripts. If you have scripts or permissions scripts that are different for each environment you can give them a special name. https://github.com/chucknorris/roundhouse/wiki/EnvironmentScripts

RH is an easy to start using on legacy databases. You just take your old DDL/DML scripts and move them into a special folder that RH will only evaluate/run when it is creating a database (say on a new developers machine). You can arrange existing scripts into RH default folders or point RH to the existing folder types. RH splits scripts with the GO batch terminator in them.

RH speeds up your development process. You can use RH with NHibernate to refresh your database without leaving Visual Studio! Entity Framework and FluentMigrator are planned for this feature as well. https://github.com/chucknorris/roundhouse/wiki/Roundhouserefreshdatabasefnh

RH runs on just the .NET framework. This means you don’t need SMO installed like some other migrators require.

While there are probably other features I haven’t mentioned, keep in mind that RH is not a code migrator (yet). If you are looking for a code migrator, there are quite a few good tools out there, including FluentMigrator and Mig#. Entity Framework Code Migrations is really starting to shape up as well (Seriously! Although EF only works for SQL Server).

How do I get RoundhousE?

There are several avenues to get RH. You can use NuGet, Chocolatey, Gems, plain old downloads (still considered official releases), or source (both in git and svn). https://github.com/chucknorris/roundhouse/wiki/Getroundhouse

AppHarbor - Azure Done Right AKA Heroku for .NET

Rob Reynolds — Wed, 16 Feb 2011 12:49:23 GMT

Easy and Instant deployments and instant scale for .NET?

Awhile back a few of us were looking at Ruby Gems as the answer to package management for .NET. The gems platform supported the concept of DLLs as packages although some changes would have needed to happen to have long term use for the entire community. From that we formed a partnership with some folks at Microsoft to make v2 into something that would meet wider adoption across the community, which people now call NuGet. So now we have the concept of package management. What comes next?

Heroku

Instant deployments and instant scaling. Stupid simple API. This is Heroku. It doesn’t sound like much, but when you think of how fast you can go from an idea to having someone else tinker with it, you can start to see its power. In literally seconds you can be looking at your rails application deployed and online. Then when you are ready to scale, you can do that. This is power. Some may call this “cloud-computing” or PaaS (Platform as a Service).

I first ran into Heroku back in July when I met Nick of RubyGems.org. At the time there was no alternative in the .NET-o-sphere. I don’t count Windows Azure, mostly because it is not simple and I don’t believe there is a free version. Heroku itself would not lend itself well to .NET due to the nature of platforms and each language’s specific needs (solution stack). So I tucked the idea in the back of my head and moved on.

AppHarbor Enters The Scene

I’m not sure when I first heard about AppHarbor as a possible .NET version of Heroku. It may have been in November, but I didn’t actually try it until January. I was instantly hooked. AppHarbor is awesome! It still has a ways to go to be considered Heroku for .NET, but it already has a growing community. I created a video series (at the bottom of this post) that really highlights how fast you can get a product onto the web and really shows the power and simplicity of AppHarbor.

Deploying is as simple as a git/hg push to appharbor. From there they build your code, run any unit tests you have and deploy it if everything succeeds. The screen on the right shows a simple and elegant UI to getting things done.

The folks at AppHarbor graciously gave me a limited number of invites to hand out. If you are itching to try AppHarbor then navigate to: https://appharbor.com/account/new?inviteCode=ferventcoder

After playing with it, send feedback if you want more features. Go vote up two features I want that will make it more like Heroku.

Disclaimer: I am in no way affiliated with AppHarbor and have not received any funds or favors from anyone at AppHarbor. I just think it is awesome and I want others to know about it.

From Zero To Deployed in 15 Minutes (Or Less)

Now I have a challenge for you. I created a video series showing how fast I could go from nothing to a deployed application. It could have been from Zero to Deployed in Less than 5 minutes, but I wanted to show you the tools a little more and give you an opportunity to beat my time.

And that’s the challenge. Beat my time and show it in a video response. The video series is below (at least one of the videos has to be watched on YouTube). The person with the best time by March 15th @ 11:59PM CST will receive a prize.

Ground rules:

.NET Application with a valid database connection
Start from Zero
Deployed with AppHarbor or an alternative
A timer displayed in the video that runs during the entire process
Video response published on YouTube or acceptable alternative
Video(s) must be published by March 15th at 11:59PM CST.
Either post the link here as a comment or on YouTube as a response (also by 11:59PM CST March 15th)

From Zero To Deployed In 15 Minutes (Or Less) Part 1

From Zero To Deployed In 15 Minutes (Or Less) Part 2

From Zero To Deployed In 15 Minutes (Or Less) Part 3

Herding Code Talks About Nu

Rob Reynolds — Fri, 27 Aug 2010 06:22:58 GMT

Dru and I were recently featured on Herding Code Podcast. In the podcast we talk about everything from package management in general to Nubular (Nu) to other package management systems (OpenWrap, Bricks, and Horn, although horn was/is slightly different) to the possibility of Microsoft releasing a package management system. It was a good time and I enjoyed doing the podcast, but the herding code guys start recording at an insane 10:30 PM! I have no idea how I made it through the whole thing and was still able to talk afterwards.

Some highlights:

There is a lot of talk in the beginning about package management in general.
Basically, I say the word “basically” way too much.
For the first 16 minutes I think I was teetering on sleep, it was 10:30PM after all. Then after that I start to wake up and sound more articulate.
Big shouts out to all the people in the community that have really been stepping up, and huge shout outs to Nick Quaranto (@qrush), Bil Simser (@bsimser) and Michael Carter (@kiliman).
40:16 – Discussion about what we would do if Microsoft released a package management system
49:34 - “It’s spelled with an N U” - Dru Pimps Nu in the “Pimp Your Code” section of the podcast
Going OT (off topic) 53:11 - “We junk punch audit” - Mentions for RoundhousE and what problems it solves.
55:13 - “At the end of the day it is what it is as long as you bring your A game” - Basically Rehashed and Remixed.

Check out the herding code podcast and notes for more. I didn’t keep track of certain times for interesting bits in the beginning of the podcast, but Ben Griswold does an excellent job on the notes from the podcast.

http://herdingcode.com/?p=272

Gems - Package Management For .NET

Rob Reynolds — Fri, 16 Jul 2010 04:27:00 GMT

The Ruby community has enjoyed a great user experience with a package management system they use called Gems. A gem is a package (or a library), compressed with some additional metadata, and can be either source files or binaries. Let’s focus on binary gems. We have the same concept in .NET (DLLs/EXEs). You may have references to other DLLs. When you want to update a reference you are using on a project, you may also need to update its dependencies as well. And so on and so forth. A package management project is meant to help make that easier. It’s actually really hard to explain what gems or package management without just showing you. So take a look here:

I type:

gem install sidepop

And that’s it. It looks and sees that I have a dependency on log4net. Notice how it nicely just pulls down log4net version 1.2.10 as well?

Background

I can count on one hand all of the package management projects that have been started for .NET. Dru Sellers, Chris Patterson, and I have talked about package management stuff from time to time. Dru and Chris have been a part of one project (Nu) that has been started several times to start to answer this question. We’ve participated on the mailing list for HornGet. At one point I casually asked why we couldn’t just use gems. Other people out there have probably stated the same thing. But no one has really carried the idea forward. Until now.

Yesterday Dru asked for a gem-ify feature for UppercuT. We started talking and looking at how easy it is to create a gem. Then we figured out how to make the executables piece work as well. This is where you can install a gem and then call the executable from the command line anywhere and get output. From the thread on ChuckNorris where we talked about this:

Here is something you might find pretty interesting: http://rubygems.org/gems/roundhouse

If you have ruby installed, you can install roundhouse now from the command line:

gem install roundhouse

This gives you the opportunity to type this anywhere:

rh <options>

Why Should I Care?

If you work with Open Source, you know how much of a pain it can be to update your references. You update one library, say NHibernate, and find out that you also need to update your references to Castle. And possibly, you might then need to update your references to log4net. It can be a painful process. This is the start of answering that question.

Why Should I Get Excited?

It sounds like Jeremy Miller among others are getting excited about this. And why not? We’ve been trying to answer the gems question since Ruby made it so easy.

The implications of this are awesome! I still haven’t fully grokked what we’ve just opened up. But it’s huge!

It doesn’t get us all of the way there to updating the references in our source code folder. That’s where projects, like Nu, are going to start showing up that leverage the idea of using the gems infrastructure to get the libraries from the ruby folders to your source code folder. You are going to see UppercuT come back soon with taking care of getting your gem built with the proper version.

This is the start of something very cool.

Before you comment about “cluttering” the ruby community, please be sure to read this (we’re with you on this): http://devlicio.us/blogs/rob_reynolds/archive/2010/07/19/gems-for-net-community-response.aspx

How To - Gems and .NET and How To – Gems & .NET - Dependencies (References)

Walkthrough – Create Gems Even Easier With a Conventional Build (UppercuT)!

The Future is Now!

Chuck Norris Welcomes You

Rob Reynolds — Sat, 17 Apr 2010 14:57:21 GMT

Chuck Norris doesn’t program with a keyboard. He stares the computer down until it does what he wants.

All things need a name. We’ve tossed around a bunch of names for the framework of tools we’ve been working on, but one we kept coming back to was Chuck Norris. Why did we choose Chuck Norris? Well Chuck Norris sort of chose us. Everything we talked about, the name kept drawing us closer to it. We couldn’t escape Chuck Norris, no matter how hard we tried. So we gave in.

Chuck Norris can divide by zero.

What is the Chuck Norris Framework?

@drusellers and I have been working on a variety of tools:

WarmuP - http://github.com/chucknorris/warmup (Template your entire project/solution and create projects ready to code - From Zero to a Solution with everything in seconds. Your templates, your choices.)
UppercuT - http://projectuppercut.org (Build with Conventions - Professional Builds in Moments, Not Days!) | Code also at http://github.com/chucknorris/uppercut
DropkicK - http://github.com/chucknorris/dropkick (Deploy Fluently)
RoundhousE - http://projectroundhouse.org (Professional Database Management with Versioning) | Code also at http://github.com/chucknorris/roundhouse
SidePOP - http://sidepop.googlecode.com (Does your application need to check email?)
HeadlocK - http://github.com/chucknorris/headlock (Hash a directory so you can later know if anything has changed)

Others – still in concept or vaporware

What’s in a Name?

People ask why we choose such violent names for each tool of our framework? At first it was about whipping your code into shape, but after awhile the naming became, “How can we relate this to Chuck Norris?” People also ask why we uppercase the last letter of each name. Well, that’s more about making you ask questions…but there are a few reasons for it.

Project managers never ask Chuck Norris for estimations…ever.

The class object inherits from Chuck Norris

Chuck Norris doesn’t need garbage collection because he doesn’t call .Dispose(), he calls .DropKick()

Call To Action

So what are you waiting for? Join the Google group today, download and play with the tools.

And lastly, welcome to Chuck Norris. Or should I say Chuck Norris welcomes you…

RoundhousE – ADO.NET over SMO

Rob Reynolds — Wed, 24 Feb 2010 03:43:34 GMT

One thing most database change management tools use is SQL Server Managment Objects (SMO). Most do that because ADO.NET doesn’t allow the batch statement separator keyword GO in sql statements. So most people write off the ability to use a database change tool for sql server without actually having the sql server installed on the machine that is running the tool. This is not a reality in some organizations, especially when licenses for SQL server are limited.

Split the GO

Recently, a few guys have really been helping me get RoundhousE (RH) to the point of not using SMO anymore.I’d like to thank Jochen and Pascal for all of your hard work recently on this and bearing with me while we worked out getting the regex ironed out for splitting sql statements on GO when it isn’t in a comment section.

There only two requirements with GO statements when using RH. The first one (a good practice IMHO) is that GO has to be the last statement on a line. The second is that using GO in between tick marks (‘), like for inserting it into a table, be sure it isn’t the last statement on a given line. This second one is a low possibility, but worth mentioning for those that might structure string insert/updates with GO in them in such a format.

So to recap:

Good Stuff

sql statements GO

something something
GO

some more statement action go

/*
Use some database
GO
*/
--GO
--something something GO

Not Gonna Work

do something GO do something else

insert into sometable (id, description)
VALUES
(1, 
' yep yep. uh huh...ya ya ya ya ya. GO
')

Favor ADO.NET over SMO

Those that have been using RH know that they can use it with SMO for 2005 and 2008. Being able to switch out either one of these has been a lifesaver for some organizations that have not yet upgraded to sql server 2008. Right now RH is in a transition state where SMO is still accessible while ADO.NET is really feeling it’s way out. So if you are using sql2005 or sql2008 now, you can continue doing so. Keep in mind that those will be deprecated in favor of ado.net at some point.

Go check it out! To use the new ado.net version, just change the databaseType (/dt in console) to “sqlserver”. Please let me know if you find any problems with it.

databaseType="sqlserver"

For those that use Oracle, be on the lookout for coming support. Also, Fluent NHibernate lovers, be ready for some schema script generation!

Warmup – Getting Started

Rob Reynolds — Mon, 01 Feb 2010 14:25:00 GMT

What if there was a tool out there that could let you specify a structure for a project (visual studio solution + everything else) and save you up to 3+ hours of work every time you started a new project?

Warmup was an idea by Dru Sellers to remove all of the setup work required every time you set up a new project. You know, create the solution, add projects, put in your references, etc. Then how about getting the infrastructure for your service/website/console set up as well with things like IoC, etc? What about patterns and other pet items that you put in any project?

Yeah – there’s an app for that. And it’s pretty simple to use. Plus you can change your templates when you have new ideas, so it’s totally rockstar!

The first thing you do is to set up templates somewhere in source control (svn or git). Then you specify where that is to the configuration and what type of source control.

<warmup
  sourceControlWarmupLocation="git://github.com/ferventcoder/warmup-templates.git"
  sourceControlType="git"
  />

Then you run a simple command. If base was one of the folders below the directory above my source control, then that is what I would specify as the first argument.

warmup.exe base nameOfProject

I specify nameofProject. That is what I want my project to be named when I am complete.

Templating

Let’s start by taking a look at the base template. The basic idea here is simple. Place __NAME__ everywhere you want to be replaced when running Warmup. In the same way UppercuT does token replacement with ConfigBuilder, DocBuilder, SqlBuilder, and DeployBuilder, Warmup does token replacement for an entire solution.

If we were to look at some of the files you would see the absolute depth of how naming really can be replaced. Let’s take a look at the solution though, that will hold quite a bit of meaning for you.

Now that we have our template all set up, we have one thing to do. Open our .sln file in Notepad and delete the first line (not sure what happens here, but this works). Microsoft should be on the first line when we are done.

And now our template is all ready, so we check our changes into source. If we are using git, we need to push back to the repository we are looking at after we finish committing.

Run Warmup

Now that we’ve seen our template, let’s run Warmup.

Let’s call the new project Alpha.

warmup.exe base Alpha

This is the output:

Hardcore git cloning action to: C:\code\warmup\code_drop\warmup\Alpha
Running: cmd /c git clone git://github.com/ferventcoder/warmup-templates.git C:\code\warmup\code_drop\warmup\Alpha
Initialized empty Git repository in C:/code/warmup/code_drop/warmup/Alpha/.git/

replacing tokens

This project already has UppercuT, RoundhousE, and others already in it. Take a look at the lib folder:

I open a command line and run build.bat. I get a successful build with 31 passing tests! If I were to go to the code_drop folder, I’m all ready to deploy if I had my deployment framework already here.

I can already run RoundhousE and create my database from here:

Conclusion

I have an entire structure that allows me to just concentrate on the stories at hand. Warmup may not be the best thing since sliced bread, but it’s going to save you oodles of time! If you know someone else that has created a template you want to use and it’s shared publicly, you can just edit the config file to point there. We have been using warmup for close to 3 months now and it saves us quite a bit of time. Plus we find more and more things we can put back into the templates to save us time. I believe this aspect of learning and growing your templates over time is the intention of warmup. Plus your template may not be the same as mine and that’s completely cool!

Two Major Milestones for RoundhousE and UppercuT

Rob Reynolds — Fri, 01 Jan 2010 21:04:21 GMT

This last year I had a focus to get more involved in Open Source (OSS). The year before I had been involved a little by submitting a patch here and there, but 2009 was a big year for me in OSS. I now manage 4 open source projects that have solved a need for me and others. Coming up on the end of the year it was important for me to get some final touches into at least two projects to finish out the year.

UppercuT

UppercuT (UC), for those of you who have never heard of it, is an automated build tool that uses conventions to build your code. That means that you can have a professional build for most projects out of the box within 3 minutes! You drop it into a project, update a simple configuration and you’re done. It is also extendable (to the point of steps being completely replaced) so it’s very powerful. The goals we had for UC when we started it are that it’s insanely easy to use, easy to upgrade your builds, and easy to extend. It has met all of those goals and has some features that other build tools do not have (versioning out of the box and templating for configurations, documents, sql, deployment files, etc to name a couple). Check it out at http://projectuppercut.org.

So what milestone did we finish for UC? When the project started, there were plans for support for multiple types of source control for versioning the assemblies. We supported Subversion (SVN) from the beginning, but the priority for other types was lower. I’m pleased to announce that in the last two months we’ve added both TFS and GIT. The GIT one is a little more interesting. It follows what is suggested for versioning by Chris Ortman, GitFu and in the Git documentation. We also go the extra mile to show the SHA1 as well so you could theoretically get to that version.

How easy is it to do versioning with Git? Assuming you have a git repository, you crack open the configuration and change your source control to git:

If you’ve used UppercuT in the past or have been dying to try it out, set a goal to look at the latest verion this year before February.

RoundhousE

RoundhousE (RH), for those of you who have not heard of it, is a database migrations tool that uses SQL Scripts. RH is kind of like Tarantino. We had originally wanted to contribute our ideas to Tarantino, but some of the goals for the two didn’t line up nicely so RoundhousE was born. RoundhousE runs update scripts (the goal is DDL/DML only), but it also looks for other scripts in other folders (like functions, views, and sprocs). The reasoning for the separation is so that versioning and seeing version history is much cleaner in source control. RH also versions databases the way you want. We prefer to version our databases based on source control, but it is extendable to be versioned in any way (or not at all). RH has an MSBuild task, a NAnt task, but most importantly, a command line console (rh.exe). Everything is configurable (including the version and scripts run tables). Check it out at http://projectroundhouse.org. RH has quite an extensive roadmap for where it is going (we are moving it to be a total migration package, not just a migration runner).

So what milestone did we finish for RH? Making RH environment aware so you can execute environment specific scripts. Why would you want to do that? What if you wanted to insert a bunch of test data into your database but didn’t want that to get into production? What if you want to grant permissions to different users in different environments?

Here I have a set of permissions scripts. Notice they are environment specific.

Notice also that they have two things. They have “.ENV.” in the name to let RH know they are environment scripts. Then they also have the name of the specific environment (LOCAL, TEST) they should be run in. By the way, If you use UC and templating, you only maintain one file in source control and the file per environment is generated at build time (I digress). So we have some environment files and now I run RH to migrate my database.

We can see from the log notes below that it only runs the LOCAL environment script in the LOCAL environment.

LOCAL.GrantRobDataReaderDataWriterPermissions.ENV.sql is an environment file. We are in the LOCAL environment. This will run based on this check.
Running LOCAL.GrantRobDataReaderDataWriterPermissions.ENV.sql on (local) - TestRoundhousE.
TEST.GrantRobDataReaderDataWriterPermissions.ENV.sql is an environment file. We are in the LOCAL environment. This will NOT run based on this check.

And to be sure, let’s take a peek in the database:

If you’ve looked at RoundhousE in the past or haven’t yet looked at it, it’s worth downloading and running the sample to see what it can do.

By the way, you’ve probably noticed that have a definite theme with our developer automation tool suite (WarmuP, UppercuT, RoundhousE, DropkicK, HeadlocK, and SidePOP). We are still not solid on the library name, but pretty close. Each of the tools are designed to be enjoyable to use and free. We respond to inquiries and fix problems quickly. Once we have a name for the library, we may offer enterprise support for those that would require that.