How to change a ClusterRole (without breaking the cluster)

In Kubernetes, a ClusterRole and a Role define authorisation rules to access Kubernetes resources either on a cluster level (ClusterRole) or on a particular namespace (Role).

A ClusterRole is a non-namespaced resource that allows you to define permissions across the entire cluster. It can grant access to resources like nodes, namespaces, or persistent volumes that exist at the cluster level, independent of any particular namespace. On the other hand, a Role operates within the boundary of a particular namespace and is used to grant permissions on resources such as Pods, Services, or ConfigMaps that exist within that namespace.

This ClusterRole for example allows read access to all pods in the cluster:

kind: ClusterRoleapiVersion: rbac.authorization.k8s.io/v1metadata:  name: pod-readerrules:- apiGroups: [""]  resources: ["pods"]  verbs: ["get", "watch", "list"]

And this Role allows writing (creating, updating, deleting) to pods in a particular namespace (my-namespace for this instance):

kind: RoleapiVersion: rbac.authorization.k8s.io/v1metadata:  namespace: my-namespace  name: pod-writerrules:- apiGroups: [""]  resources: ["pods"]  verbs: ["create", "update", "delete"]

In order for a workload (Pod) to be able to use these permissions:

That pod must have a ServiceAccount
The ServiceAccount must have a RoleBinding/ClusterRoleBinding that reference the Role/ClusterRole

However Pods aren’t used directly in Kubernetes (usually), they are instead controlled by Deployments, ReplicaSets, DaemonSets, Jobs or CronJobs and therefore in order to truly understand the impact of changing a Role/ClusterRole, we need to link all the way back to these controlling resources. This means that the whole relationship looks like this:

This means that in order to work out the true potential impact of changing a Role/ClusterRole, we must follow all of the relationships in this diagram. Here are the commands you need:

‍

Get the details of a role:

kubectl describe clusterrole {role-name}
Name:         srcman-proxy-role
Labels:       {none}
Annotations:  {none}
PolicyRule:
  Resources                                  Non-Resource URLs  Resource Names  Verbs
  ---------                                  -----------------  --------------  -----
  tokenreviews.authentication.k8s.io         []                 []              [create]
  subjectaccessreviews.authorization.k8s.io  []                 []              [create]

‍

Get the bindings that refer to that role

kubectl get clusterrolebinding -o custom-columns=NAME:.metadata.name,ROLE:.roleRef.name --all-namespaces | grep {role-name}srcman-proxy-rolebinding                               srcman-proxy-role

‍

Get the ServiceAccounts that use a given ClusterRoleBinding bindings

kubectl get clusterrolebinding  -o=jsonpath='{range .subjects[?(@.kind=="ServiceAccount")]}{@.name}{"\n"}{end}' --all-namespacessrcman-controller-manager

‍

Get the Pods that use that ServiceAccount

❯ kubectl get pods --all-namespaces -o=jsonpath='{range .items[?(@.spec.serviceAccountName=="")]}namespace={.metadata.namespace} pod={.metadata.name} ownerKind={.metadata.ownerReferences[0].kind} ownerName={.metadata.ownerReferences[0].name}{"\n"}{end}'
namespace=srcman-system pod=srcman-controller-manager-767496f48-fvbgh ownerKind=ReplicaSet ownerName=srcman-controller-manager-767496f48

Note that in the above command we produce the ownerKind and ownerName columns. These show what type of resource owns this pod, e.g. a ReplicaSet, DaemonSet, Job or CronJob

‍

Get the CronJob that controls a given Job:

kubectl get job  -n  -o=jsonpath='{.metadata.ownerReferences[0].name}{"\n"}'srcman-controller-cronjob

‍

Get the CronJob that controls a given Job:

kubectl get job  -n  -o=jsonpath='{.metadata.ownerReferences[0].name}{"\n"}'srcman-controller-cronjob

‍

Get the Deployment that controls a ReplicaSet:

kubectl get rs  -n  -o=jsonpath='{.metadata.ownerReferences[0].name}{"\n"}'srcman-controller-manager

‍

Get the details of a deployment:

❯ kubectl describe deployment  -n 
Name:                   srcman-controller-manager
Namespace:              srcman-system
CreationTimestamp:      Fri, 16 Jun 2023 15:23:36 +0100
Labels:                 control-plane=controller-manager
Annotations:            deployment.kubernetes.io/revision: 2
Selector:               control-plane=controller-manager
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:           control-plane=controller-manager
  Service Account:  srcman-controller-manager
  Containers:
   kube-rbac-proxy:
    Image:      gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0
    Port:       8443/TCP
    Host Port:  0/TCP
    Args:
      --secure-listen-address=0.0.0.0:8443
      --upstream=http://127.0.0.1:8080/
      --logtostderr=true
      --v=10
    Environment:  
    Mounts:       
   manager:
    Image:      ghcr.io/overmindtech/srcman:0.13.0
    Port:       
    Host Port:  
    Command:
      /manager
    Args:
      --health-probe-bind-address=:8081
      --metrics-bind-address=127.0.0.1:8080
      --leader-elect
    Limits:
      cpu:     100m
      memory:  30Mi
    Requests:
      cpu:      100m
      memory:   20Mi
    Liveness:   http-get http://:8081/healthz delay=15s timeout=1s period=20s #success=1 #failure=3
    Readiness:  http-get http://:8081/readyz delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment Variables from:
      tracing-keys  Secret  Optional: true
    Environment:    
    Mounts:         
  Volumes:          
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Progressing    True    NewReplicaSetAvailable
  Available      True    MinimumReplicasAvailable
OldReplicaSets:  srcman-controller-manager-8485cff9d6 (0/0 replicas created)
NewReplicaSet:   srcman-controller-manager-767496f48 (1/1 replicas created)
Events:

With Overmind

Overmind allows you to determine the potential blast radius of changes to ClusterRoles, Deployments and any other Kubernetes or AWS resources before you make them. Using current application config it acts as a second pair of eyes, analysing your config changes to calculate any dependencies. From the blast radius it can provide a list of human readable risks that can be reviewed prior to deploying the changes.These risks can either be commented back as part of your CI / CD pipeline or viewed in the app.

To get started calculating the impact of a change to a ClusterRole, firstly create a free Overmind account.

Next you will need to create a change. This can be done by:

Get started quickly using the Overmind Github action. The action will automatically create a new change with the resources and items parsed from your Terraform plan output.
Integrate Overmind into any CI /CD tool using the CLI.
Coming soon or by using the Overmind CLI to run a plan and apply locally.

Select the relevant ClusterRole as the expected change. From there Overmind automatically determines the potential impact in real time:

Interested in learning more, join our Discord or check out our Terraform example repository.

How to change a ClusterRole (without breaking the cluster)

With Overmind

What the CrowdStrike Outage Can Teach Us About Preparing for the Unexpected

Why Our Startup Runs on Discord and the Channels That Make It Work

5 Terraform Tools You Should Know About in 2024

The State of Terraform (mini) Report

How to change a ClusterRole (without breaking the cluster)

With Overmind

Latest blogs

What the CrowdStrike Outage Can Teach Us About Preparing for the Unexpected

Why Our Startup Runs on Discord and the Channels That Make It Work

5 Terraform Tools You Should Know About in 2024

The State of Terraform (mini) Report