Earlier this month I spoke at DevOpsCon in Berlin on the topic of Infrastructure as Code (IaC) and the lessons we can learn from agile software development techniques. Now I’m the first to admit that I’m still a complete novice when it comes to Infrastructure as Code but I like to think that over the past 15 or so years of my software development career that I’ve learned a few things about writing code and delivering value in an incremental and agile way.
My talk at DevOpsCon centred around four agile themes:-
- Simplicity – the art of maximising work not done
- Technical Excellence
- Purpose and a Shared sense of Ownership
The theme that provoked the most conversation and questions was Quality – how can we layer in automated quality checks on our infrastructure code and shift-left on IaC, detecting problems as early as possible?
One way is to make use of static code analysis aka linting. For CloudFormation, cfn-lint and cfn-nag both provide static code analysis of template files. You can run these linting tools on the command line as part of your CI pipeline or there are plugins available for common IDEs that can help you spot code style issues and common mistakes well before you check your code into a code repository. Cfn-nag is designed to check security settings, for example, it can check to see if you’re about to make a change that will unnecessarily open up a security group. Cnf-nag is also customisable and extensible for you to perform your own code checks.
Tflint is a similar tool for Terraform (in fact, most infrastructure and configuration management tools have open source linting tools available). When used with the -deep option, tflint can actually check the state of the resources you’re about to apply to. Checks such as ensuring that an IAM profile your template references actually exists, or that an ELB name is unique, are things that Terraform Plan alone wouldn’t be able to check for you and the first thing you’d know about it was when you were attempting to apply.
But the most valuable type of infrastructure testing I’ve seen to date is Terraform “unit-testing”. OK, its not strictly unit testing and it would be very tricky to run locally on your own machine before you check in the code but Terraform tests can be performed on a sandbox environment as part of a full CI pipeline.
Kitchen Terraform is a plugin for Kitchen CI (which we most commonly associate with Chef). Kitchen Terraform can manage the application of your Terraform configuration and then run a series of controls – I like to think of these as rules that declare what the desired state of your system should be after the code is applied. These controls can then apply a set of assertions and are written in ruby using the RSpec syntax. Kitchen can run RSpec, Inspec or bash automation scripts. Inspec is another layer on top of RSpec and in fact, when you write a Terraform test you can use RSpec syntax for checking, or you can use Inspec Universal Matchers.
In the example below of a very basic Inspec test, the control checks the operating system of a provisioned instance and asserts that it is Ubuntu.
The great thing about Inspec tests as they spit out a Junit-esque test report which can then be rendered in beautiful colourful glory by your CI tool of choice to make quality levels visible.
To close my section of Quality in Iac, I talked about how, for the development teams I work with, unit testing is now a given; it is something they do in their sleep. It has taken 25 years to get to that point but now unit testing is (almost) universally embraced for application development. When I started my career, 15 years ago, people tried to resist the level of investment of time and effort needed to test code and yet now we see that the investment pays off – high test coverage actually helps tech teams move faster because it dramatically increased quality. So when you’re questioning whether or not the time and effort you have to invest in testing infrastructure is worth it, remember that it really is an investment and may take time to pay back its reward.