aws
Confidently Automating AWS Infrastructure Changes
James Lane
October 12, 2023
London, UK
team@overmind.tech
Our Beta is live - Join Now for Free
IaC tools like Terraform or Cloud Formation allow us to make out environments more consistent but does not necessarily make changes to prod less error prone.
The same IaC file can be used in dev, test, and prod but it doesn’t mean the environments match. Things might be turned off or scaled down in non-production environments. Whereas production environments may have hidden dependencies to other services that are simply out of the scope of IaC.
When deploying, you might find the 10 applies you did to dev in 10 small steps is not the same as the 1 big apply to prod. Meaning the actual work done in a given apply is much greater in prod and therefore riskier.
Which means you find yourself without the confidence to know wether your change to prod is going to have any unintended consequences.
To deploy confidently there’s two questions that you will need to answer:
A Terraform Plan will tell you what it’s going to touch but then it’s on you to work out what the impact might be. You can have a look through a CMDB or documentation, provided that it is actively maintained and up-to-date. Or ask colleague if they’re not away on vacation.
With Overmind, when a pull request is opened and a Terraform plan is executed you can calculate the potential impact (or blast radius) of your change. By parsing the Terraform plan output and then using only Read-only AWS credentials it can map out your infrastructure. It queries AWS directly and discovers relationships automatically, working out what the actual impact of your change is. Even for things not managed under Terraform.
Using our Github action you can combine this as part of your workflow. The action will comment back on the pull request telling you the blast radius (everything that might be affected by the given change).
From this you are then able to check the affected items to see if there is anything unexpected. If you notice that the change might affect more than you thought, you can modify either your code, or the way you plan to roll out and monitor the change to account for it. You can then share this change or graph with your team or the change advisory board.
After making a change Terraform will tell you that it is done, but nothing else. You can spend more time watching your monitoring tools for any alerts on logs, metrics or traces. Only to receive a phone call to tell you that something you didn’t even know existed has gone down.
With Overmind you’ll soon be able to see the diff of your changes. It takes a snapshot of your infrastructure before and after you make a Terraform apply giving you a diff of your configuration that allows you to immediately identify unexpected changes and revert them.
We want to make it as easy as possible to get started with Overmind, because of this we have created an example repository. It shows how to run terraform on GitHub Actions and automatically submit each PR's changes to Overmind, reporting back the blast radius as a comment on the PR. This way you can get started easily with either your personal or org AWS account.
Stay updated and network with other Overmind users.
