Checklists, Automation and Consistency

Check out the post “14 lessons after five years of professional programming” over on siavosh’s blog. It’s a great list, but I’d like to expand on one item: Check lists are your friends.

Most of Siavosh’s list is good advice - you could do a whole lot worse than to follow his guidance. For someone who only admits to five years of professional programming (as of 2013, anyway) it’s clear that he’s learnt a lot from his experiences.

Number #11 on his list is “Check lists are your friends”.

Yes, checklists are useful - I make use of them myself.

That said, the proverbial “47 page Production Release Checklist” is neither mythical nor a good idea. In fact, because they give the impression of consistency without actually providing any support to achieve it, long checklists are dangerous.

Good checklists are short, concise, accurate, and cover the essentials. A good checklist should fit on one page - without printing on the back - and be straightforward to follow.

The key to taking a long and tedious checklist and reducing it to something sane is to simplify, simplify, simplify. One of the most effective techniques is to introduce automation.

Automated processes will do the same thing every time, can (should!) be tested in advance, and greatly reduce the opportunity for error.

Here are some examples. You’ll notice a pretty strong Windows bias simply because that’s the environment I work in day to day - the lessons apply across platform.

When you need to compile a new version to deploy for testing …

Don't	Do an ad-hoc build on your development machine and manually copy it across.
Do	Run an automated build on a dedicated continuous integration server.

This ensures that the all the source code for the version you’re deploying exists in your source code version control system and that the build process is done the same way each time. Jetbrains TeamCity is free for small teams and low numbers of builds.

When you’re deploying a new version …

Don't	Deploy to a server by requiring a sysadmin to manually create a folder and copy across selected files from multiple network shares.
Do	Write a silent msi installer that deploys everything in one go.

For extra credit with your systems administrators, supply a PowerShell script to run that installer with all the right parameters.

When you need to reconfigure production …

Don't	Require your sysadmin to make manual changes to production configuration files using Notepad (or any other editor).
Do	Supply a whole new configuration file to replace the existing one.

Your system administrator can, and should, compare the two configuration files using a good diff tool (Beyond Compare is a good choice) to make sure that only the expected differences are present.

Alternatively, you could write a PowerShell script that makes the required changes after taking a backup (don’t forget to test the script).

When you are using convention based programming …

Don't	Rely on manual checks to ensure your codebase is adhering to the required convention.
Do	Write a unit test to fail your continuous integration build if the convention is violated.

For example, when working with Data Transfer Objects and WCF, I use a unit test that checks all my DTO class aredecorated with [DataContract] and all their properties with [DataMember], thus avoiding any silent failures where necessary properties are not serialised across the connection.

When Checking system health …

Don't	Create a checklist for a system health check that relies on the sysadmin (or dev) understanding all the intracacies of the configuration.
Do	Write a PowerShell script that does all the checks and flags any errors.

Once the script is working, find some way to run it automatically after every deployment so you can detect any issues with the deployment before any of your users. You might want to run it on a schedule in production too.

When using code generation …

Don't	Use a tool to generate a bunch of code and then make the same manual changes every time.
Do	Update the templates used by the code generator so the output is what you need.
Or	Script some additional steps to automatically make the changes you need.

Code generation is supposed to save time, improve consistency and eliminate the need for writing manual boilerplate. Needing to making manual changes to the results of code generation will result in people not regenerating when they should as well as running the risks that the changes are made differently each time.

For example, when generating proxy source using svcutl.exe, I’ll often use a PowerShell script to make surgical changes to the generated source so it meets my needs. (For example, modifying every service implementation to declare the IService interface so I can use dependency injection cleanly.)

Conclusions

Living in a Microsoft world, I’ve found that PowerShell is often the swiss-army-knife that I need for automation. It’s facilities for manipulation of text and xml data are extremely powerful.

As the complexity of your automation grows, introducing Psake can give you a way to manage that complexity without abandoning your existing PowerShell investments.

For deployment of files onto test and production servers it’s hard to go past the WiX tookit, an open source project from Microsoft that compiles msi installers. MSI installations are atomic, reliable and can do more than just copy files - for example, they can customise configuration files as well.

Deployment automation is a big, hairy and audacious goal - the orchestration of multiple components across multiple machines can be daunting. When you get to this point, JetBrains TeamCity and Octopus Deploy might be the tools you need. (Disclaimer - no affiliations, just a very satisfied customer of both.)

The most important thing to add to your toolkit is willingness - automation doesn’t come at zero cost, you do have an up front investment to consider. But the payoffs (saved time, improved consistency, shared knowledge and more) are immense.