Argo CRDs and Kustomize: The Problem of Patching Lists

Published in

Argo Project

6 min readAug 25, 2022

Current Problem

As an end user of any Argo Project or a Kubernetes custom resource and also a user of Kustomize, I want to be able to use strategic merge patches (SMPs) for environment variables or any array type field within the resource definition.

Here is an example:

I have a partially defined Argo Rollout resource as follows and I want to change the environment variable SOMECONFIG_1 to new_config_value. I also want to add a new environment variable called ADD_NEW_CONFIG with the value of newconfig.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: rollout-canary
spec:
  template:
    spec:
      containers:
      - name: rollouts-demo
        image: argoproj/rollouts-demo:blue
        imagePullPolicy: Always
        env:
          - name: SOMECONFIG_1
            value: config_value_1
          - name: KEEP
            value: "true"
        ports:
        - containerPort: 8080

This would be the kustomization.yaml file containing a strategic merge patch:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomizationresources:
- rollout.yamlpatchesStrategicMerge:
- |-
  apiVersion: argoproj.io/v1alpha1
  kind: Rollout
  metadata:
    name: rollout-canary
  spec:
    template:
      spec:
        containers:
        - name: rollouts-demo
          env:
            - name: SOMECONFIG_1
              value: new_config_value
            - name: ADD_NEW_CONFIG
              value: newvalue

With the final output from kustomize build . being:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: rollout-canary
spec:
  template:
    spec:
      containers:
      - env:
        - name: SOMECONFIG_1
          value: new_config_value
        - name: ADD_NEW_CONFIG
          value: newvalue
        name: rollouts-demo

As we can see, it did not properly merge the environment variables. Instead, it did a replace and resulted in us losing the KEEP environment variable from our base resource.

However, if we go and add these two lines to our kustomization.yaml configuration:

openapi:
  path: https://github.com/argoproj/argo-schema-generator/raw/main/schema/argo_all_k8s_kustomize_schema.json

We will get the correct output that we intended. However there are a few issues with this approach explained below.

Current State of Affairs: Kustomize End Users

Kustomize supports adding your own openapi schema definition so it can figure out how properly merge arrays via strategic merging patch but has some big user interface issues. The main one being it currently only supports the replacement of the schema definition and not the addition to the pre-bundled Kubernetes resources. What this means is that currently Kustomize bundles an openapi schema with all the native Kubernetes resources like Deployments, Statefulsets, Secrets, etc. However, this causes issues for end users who use CRD’s with native types within a single Kustomize application. The user only has a few options:

If the custom resource they are using provides an open api schema definition with merge key information, they can choose to use that as their open api schema within Kustomize and merging will only work for that particular resource. However, they then give up being able to use strategic merge patching for native types like Ingress and ServiceAccounts etc.
If the custom resource does not provide an openapi schema the end user is forced to use Json6902 patches. The issue with these patches is that they can be very fragile . This is because patching arrays requires knowing the array index to replace or add to which can cause issues with anything but the most simple patches.

Current State of Affairs: CRD Developers

As a developer of a Kubernetes controller, I would like to be able to provide an openapi schema file that my Kustomize users are able to use without much effort. This becomes a bit complicated however as we will soon see. I will go a bit into the technical details next that a developer would have to do in order to provide strategic merge patch support for users of their CRDs. I will assume the controller’s codebase is written in golang and that we can leverage upstream Kubernetes code generation tools.

Kubernetes has a tool called openapi-gen that is used to generate an openapi_generated.go file (example). This auto generated code has one exported function called GetOpenAPIDefinitions that we can use to help us generate an openapi schema definition json file that is compatible with Kustomize with some minor changes. You can see the usage of this here and a small snippet of output from a single generated definition below. The main idea behind GetOpenAPIDefinitions is too generate the $reflinks you see in the output.

{
 "definitions": {
  "io.argoproj.rollouts.v1alpha1.RolloutSpec": "...",
  "io.argoproj.rollouts.v1alpha1.Rollout": {
   "description": "Rollout is a specification for a R...",
   "type": "object",
   "required": [
    "spec"
   ],
   "properties": {
    ...
    "metadata": {
     "default": {},
     "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta"
    },
    "spec": {
     "default": {},
     "$ref": "#/definitions/io.argoproj.rollouts.v1alpha1.RolloutSpec"
    },
    "status": {
     ...
    },
    "x-kubernetes-group-version-kind": [{
     "group": "argoproj.io",
     "kind": "Rollout",
     "version": "v1alpha1"
    }]
   }
  }
 }
}

Let’s step back just a bit and talk about what these $ref links are and how Kustomize uses this information. When Kustomize is going to do a strategic merge patch to a particular resources like a Rollout object, it first looks through all the definitions within the schema file and finds a match based on an extension to the openapi spec. This is shown in the example above as x-kubernetes-group-version-kind. Once Kustomize has found the resource that it needs to patch, it then walks the reference links, and as you can see from the example the references start with #/definitions/ which is the top level definitions object followed by a resource key such as io.argoproj.rollouts.v1alpha1.RolloutSpec. It goes through each of these items looking for snippets like those below, which is the env section from a container spec from the native Kubernetes type. You can see that it has two bits of metadata x-kubernetes-patch-merge-key and x-kubernetes-patch-strategy that it uses to do strategic merge patching of arrays.

"env": {
    "description": "List of environment variables to set in the container. Cannot be updated.",
    "type": "array",
    "items": {
        "default": {},
        "$ref": "#/definitions/io.k8s.api.core.v1.EnvVar"
    },
    "x-kubernetes-patch-merge-key": "name",
    "x-kubernetes-patch-strategy": "merge"
},

Putting It All Together

So how did we generate a schema file with everything we needed? The openapi-gen tool I mentioned has a feature that lets you include a list of input-dirs, however the name of this flag is a bit confusing because it is actually a list of dirs that are found within your go environment or vendor folder. This feature is what we use to create a schema definition of all the Kubernetes native resources and include all the Argo project resources into one schema definition. There is however one minor issue the openapi_generated.go file that is generated by openapi-gen does not include the metadata for x-kubernetes-group-version-kind.

This was not too hard of an issue to solve for Argo projects resources because we can infer them quite easily from the auto generated names. However, it gets a little trickier for the native Kubernetes types but there is a pretty clever workaround. We end up using the upstream provided schema file found here and match the dictionary keys pulling out just the group version kind information and adding it to our generated definitions file. This final bit allows us to generate a fully compatible schema definition file that Kustomize can use to allow strategic merge patching of all Argo projects resources while still maintain support for native Kubernetes kinds.

Looking Forward

There are two possible improvements to this situation being discussed upstream one in Kubernetes and the other in Kustomize. The fairly big improvement that Kustomize can make is allowing openapi configuration become a list of resource definitions that Kustomize would then do all the hard work of merging into one global schema. This is talked about in this upstream issue.

The other even greater improvement would be if Kubernetes allow this merge key metadata to become part of a custom resource definition. This would solve everyone's problem very conveniently because Kubernetes has an open api endpoint that tools like Kustomize could just use to get a definition configuration of all the CRDs installed on the end users cluster. The issue tracking this can be found here.

Taking Advantage of Strategic Merge Patches Today

If you are currently using an Argo project and would like to take advantage of strategic merge patches. You can now easily do this by including this openapi configuration in your top level Kustomization configuration file. Happy Merging!

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomizationresources:
- rollout.yamlopenapi:
  path: https://github.com/argoproj/argo-schema-generator/raw/main/schema/argo_all_k8s_kustomize_schema.json#To track a specific git tag which is suggested use:
#openapi:
#  path: https://github.com/argoproj/argo-schema-generator/raw/2022-08-15-1660539267/schema/argo_all_k8s_kustomize_schema.json