Deploying to prod
IaC tools like Terraform or Cloud Formation allow us to make out environments more consistent but does not necessarily make changes to prod less error prone.
The same IaC file can be used in dev, test, and prod but it doesn’t mean the environments match. Things might be turned off or scaled down in non-production environments. Whereas production environments may have hidden dependencies to other services that are simply out of the scope of IaC.
When deploying, you might find the 10 applies you did to dev in 10 small steps is not the same as the 1 big apply to prod. Meaning the actual work done in a given apply is much greater in prod and therefore riskier.
Which means you find yourself without the confidence to know wether your change to prod is going to have any unintended consequences.
Building confidence
To deploy confidently there’s two questions that you will need to answer:
- Before making a change: What’s going to be impacted?
- After the change: What’s the difference?
Before making a change
A Terraform Plan will tell you what it’s going to touch but then it’s on you to work out what the impact might be. You can have a look through a CMDB or documentation, provided that it is actively maintained and up-to-date. Or ask colleague if they’re not away on vacation.
With Overmind's risks you can surface incident-causing config changes as part of your pull request. When a pull request is opened and a Terraform plan is executed you can calculate the potential impact (or blast radius) of your change. By parsing the Terraform plan output and then using only read-only AWS credentials it can map out your infrastructure. It queries AWS directly and discovers relationships automatically, working out what the actual impact of your change is. Even for things not managed under Terraform.
From this you are then able to check the affected items to see if there is anything unexpected. If you notice that the change might affect more than you thought, you can modify either your code, or the way you plan to roll out and monitor the change to account for it. You can then share this change or graph with your team or the change advisory board.
From the blast radius it also provides a list of human readable risks that can be reviewed prior to running Terraform apply. These risks can either be commented back as part of your CI / CD pipeline or viewed in the app. Using our Github action you can combine this as part of your workflow. The action will comment back on the pull request telling you the blast radius (everything that might be affected by the given change).
Inside the app you can see the full blast radius in a interactive graph along with any metadata Overmind was able to get from AWS. When you're ready to start the change, Overmind will take a snapshot before and after to validate that the change went through as intended.
Don’t just take our word for it…
We want to make it as easy as possible to get started, because of this we have created an example repository. It shows how to run terraform on GitHub Actions and automatically submit each PR's changes to Overmind and report back the blast radius as a comment on the PR. This way you can get started easily with either your personal or org AWS account.
.gif)