Why you can't use Chatgpt to tell you if your next Terraform or AWS change will break something

Making infrastructure changes

When deploying to production you need to know the impact of your changes. As Loom found out in their March 2023 incident, even deploying a change to dev, test & staging for 10 days doesn’t guarantee that when you press deploy to prod it all goes smoothly.

If you are using IaC tools like Terraform, then a plan output will tell you what it’s going to change, but it’s still on you to work out what that impact could be. Looking through your CMDB / docs may help you to get a more detailed picture of the impact, provided that they’re up-to-date…

Could the explosion in popularity in AI & LLM tools be used to give you that much needed context? Let’s take a look.

Without context

Firstly lets look at a example of just pasting in a Terraform plan output of a AWS infra change in a ChatGPT playground session and see if it can give us any useful context.

Just asking what the impact will be doesn’t give us anything not already in the plan.

Asking for specific resource names gives us a little more context but we are still missing what other related resources will be impacted. This makes sense though because ChatGPT is only going off whats in the provided Terraform plan so it’s not a limitation of the tool but rather the plan output.

What we need to do is give it some further context of whats in your AWS, it’s links and dependencies and see if that helps to improve the output.

With context (Overmind)

With Overmind it’s possible to get this context as a output. It parses the Terraform plan output and then using read-only AWS credentials can calculate the impact (blast radius) of your change. Even for resources not managed under Terraform.

Overmind also parses any sensitive data from your Terraform plan and doesn't store any of your AWS config in a database or cache as it queries the API in real-time

Have a go yourself…

We’d love for you to have a go yourself and let us know what you think. Is this something you would use or like to see added as a feature in Overmind?

The best way to get started is using the Overmind example repository. It shows how to run terraform on GitHub Actions and automatically submit each PR's changes to Overmind, reporting back the blast radius as a comment on the PR which you can then provide to ChatGPT.

• Check out the example Terraform example repo.
• Get started by creating your free Overmind account here.
• Or join our Discord to discuss the next wave of Devops.