Preventing Tag Mutation With kbld And Argo CD

You’re probably aware that it is best practice not to use the latest tag when deploying to Kubernetes because that tag can be changed to point at a different image. Yet the same problem actually exists with any image tag. This creates the possibility that the image you’re running in your cluster is not the image you expect.

There is a way to ensure that the container image being used as a part of a deployment to Kubernetes is the exact version you expect and that it won’t change post deployment. While tags can be moved to point to a different version of an image, each version of this tag corresponds to a unique digest. These digests provide an immutable reference to an image that can provide assurance about what image is deployed to your cluster.

Digests are not the most human readable format, and they aren’t as commonly used as tags, which makes using digests instead of tags a challenge for developers. However, there is a tool that helps to address this challenge that can be added to existing Argo CD workflows.

How to Pin Your Image Tags

kbld is a tool that can help resolve image tags to a digest format. kbld takes Kubernetes manifests as input and converts image tags in these manifests to a digest format as output. Consider the example Deployment below with an image using a tag:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: go-web-server
spec:
  replicas: 2
  selector:
    matchLabels:
      app: go-web-server
  template:
    metadata:
      labels:
        app: go-web-server
    spec:
      containers:
      - name: go-web-server
        image: danielhelfand/go-web-server:1.0.0
        imagePullPolicy: Always
        ports:
        - containerPort: 8080

Running kbld -f deployment.yml against this Deployment gives us the following:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    kbld.k14s.io/images: |
      - origins:
        - resolved:
            tag: 1.0.0
            url: danielhelfand/go-web-server:1.0.0
        url: index.docker.io/danielhelfand/go-web-server@sha256:d6854e11026cff8c89b5f81a975ff226907e8d1c61bddf75f03bd33b065a534f
  name: go-web-server
spec:
  replicas: 2
  selector:
    matchLabels:
      app: go-web-server
  template:
    metadata:
      labels:
        app: go-web-server
    spec:
      containers:
      - image: index.docker.io/danielhelfand/go-web-server@sha256:d6854e11026cff8c89b5f81a975ff226907e8d1c61bddf75f03bd33b065a534f
        imagePullPolicy: Always
        name: go-web-server
        ports:
        - containerPort: 8080

Notice how the go-web-server:1.0.0 image reference was changed to a digest format (i.e. sha256:d6854e11026cff8c89b5f81a975ff226907e8d1c61bddf75f03bd33b065a534). kbld also adds annotations with details on the original image tag.

Now any subsequent pushes to the 1.0.0 tag will not change what is deployed out to your Kubernetes cluster. So you can still gain the benefits of tags being human readable, easy to reference, and the ability to change what is pushed to a tag without needing to use this tag during deployment.

This is because running kbld against image references will always fetch the latest digest by default so updates can be gracefully introduced and remain stable until your next deployment.

Deploying Securely With kbld And Argo CD

With kbld handling the conversion process of making sure manifests are using this digest format for image references, the next step is deploying these converted manifests to Kubernetes using Argo CD.

Argo CD can use the Kubernetes manifests generated by kbld -f like any other Kubernetes manifests. An example of a manifest converted by kbld is available here and can be used with Argo CD using the argocd CLI commands below:

argocd app create kbld-example --repo https://github.com/danielhelfand/argocd-kbld-plugin-example --path kbld-example-app-manifests --dest-name in-cluster --dest-namespace default --directory-include sample-app.yml
argocd app sync kbld-example

You can verify the deployment is using the digest by running the commands below:

kubectl get deployment/go-web-server -n default -o yaml | grep -e "- image"

You should see the following image reference used:

- image: index.docker.io/danielhelfand/go-web-server@sha256:d6854e11026cff8c89b5f81a975ff226907e8d1c61bddf75f03bd33b065a534f

Conclusion

Argo CD and kbld are a great combination for more securely deploying to Kubernetes. The ability to assure what is deployed to an environment should be top of mind for development teams, and kbld and Argo CD can make this difficult topic much simpler.

If you would like to watch a video of the walkthrough in this blog post to see everything in action, please watch at the link below. Thanks for reading!