Could Chatgpt confidently tell you the impact of your next AWS change?

Making infrastructure changes

When deploying to production you need to know the impact of your changes. As Loom found out in their March 2023 incident, even deploying a change to dev, test & staging for 10 days doesn’t guarantee that when you press deploy to prod it all goes smoothly.

If you are using IaC tools like Terraform, then a plan output will tell you what it’s going to change, but it’s still on you to work out what that impact could be. Looking through your CMDB / docs may help you to get a more detailed picture of the impact, provided that they’re up-to-date…

Could the explosion in popularity in AI & LLM tools be used to give you that much needed context? Let’s take a look.

Without context

Firstly lets look at a example of just pasting in a Terraform plan output of a AWS infra change in a ChatGPT playground session and see if it can give us any useful context.

Just asking what the impact will be doesn’t give us anything not already in the plan.

Asking for specific resource names gives us a little more context but we are still missing what other related resources will be impacted. This makes sense though because ChatGPT is only going off whats in the provided Terraform plan so it’s not a limitation of the tool but rather the plan output.

What we need to do is give it some further context of whats in your AWS, it’s links and dependencies and see if that helps to improve the output.

With context (Overmind)

With Overmind it’s possible to get this context as a output. It parses the Terraform plan output and then using read-only AWS credentials can calculate the impact (blast radius) of your change. Even for resources not managed under Terraform.

To copy the blast radius output in Overmind use the "Konami code" in app when viewing a graph (create a free account here.)

You’ll get a notification saying copied to clipboard. Here’s a snippet from of the output pasted in a code editor.

By providing the blast radius output in ChatGPT you can see the response contains more useful information.

Would this be enough to confidently press deploy? Probably not, you’d still want to review/test changes. But could it be a useful tool in your arsenal? Certainly.

Have a go yourself…

We’d love for you to have a go yourself and let us know what you think. Is this something you would use or like to see added as a feature in Overmind?

The best way to get started is using the Overmind example repository. It shows how to run terraform on GitHub Actions and automatically submit each PR's changes to Overmind, reporting back the blast radius as a comment on the PR which you can then provide to ChatGPT.

• Check out the example Terraform example repo here.
• Get started by creating your free Overmind account here.
• Or join our Discord to discuss the next wave of Devops.

Note: Some caveats we found testing:

• When copying in a larger Overmind blast radius / Terraform plan you can reach the token limit (GPT3.5 16k in the Open AI API playground.) We got round this by removing the IP / DNS items from the blast radius in a code editor.
• Reminder, never share any sensitive personal/ company information on an AI platform like ChatGPT.
• Using some prompt engineering can help improve responses. Here’s an example:

“You are an experienced DevOps engineer. Your job is to review the provided terraform plan output, along with the details of potentially affected items and determine the top three risks for this change. When responding you should be as specific as possible in what the risks might be and how the user should mitigate them. Include things like the names of the resources and relevant AWS CLI commands that might help. Focus on the specific changes that are mentioned in the Terraform plan, and how these changes specifically might cause problems. Do not include any text before or after the top three risks.”