New sync and diff strategies in ArgoCD
ArgoCD 2.3 will be shipping with a new experimental sync option that will verify diffing customizations while preparing the patch to be applied in the cluster. It also includes a new diff strategy that leverages managedFields, allowing users to trust specific managers.
The Problem
The diffing customization feature allows users to configure how ArgoCD behaves during the diff stage which is the step that verifies if an Application is synced or not. However, diffing configurations weren’t considered during the sync step, which sometimes leads to undesirable behavior. One classic example is creating a Deployment with a predefined number of replicas and later on configuring an Horizontal Pod Autoscaler (HPA) to manage the number of replicas of your application. In this case we have two controllers, argocd and kube-controller-manager, competing for the same replicas field. The main direction, in this case, is removing the replicas field from the desired state (git) to avoid conflicts with HPA configurations.
Unfortunately, there are some challenges with this approach that could lead to application downtime if not executed properly. In some other cases, this approach isn’t an option as users are deploying Helm charts that don’t provide the proper configuration to remove the replicas field from the generated manifests. This is common example but there are many other cases where some fields in the desired state will be conflicting with other controllers running in the cluster.
A New Sync Option
Users can now configure the Application resource to instruct ArgoCD to consider the ignore difference setup during the sync process. In order to do so, add the new sync option RespectIgnoreDifferences=true in the Application resource.
The example below shows a configuration to ignore a Deployment’s replicas field from the desired state during the diff and sync stages:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
...
spec:
... ignoreDifferences:
- group: "apps"
kind: "Deployment"
jsonPointers:
- /spec/replicas syncPolicy:
syncOptions:
- RespectIgnoreDifferences=true
This is particularly useful for resources that are incompatible with GitOps because a field value is required during resource creation and is also mutated by controllers after being applied to the cluster. This causes a conflict between the desired and live states that can lead to undesirable behavior. Istio VirtualService configured with traffic shifting is one example of a GitOps incompatible resource. See this issue for more details.
Configuring Diffs with managedFields
Users are already able to customize ArgoCD diffs using jsonPointers and jqPathExpressions. Both approaches require the user to have a deep understanding of the exact fields that should be ignored on each resource to have the desired behavior. Now it is possible to leverage the managedFields metadata to instruct ArgoCD about trusted managers and automatically ignore any fields owned by them. A new diff customization (managedFieldsManagers) is now available allowing users to specify managers the application should trust and ignore all fields owned by them.
See the Argo Rollout example below:
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
...
managedFields:
- apiVersion: argoproj.io/v1alpha1
fieldsType: FieldsV1
fieldsV1:
'f:metadata':
'f:annotations':
'f:rollout.argoproj.io/revision': {}
'f:spec':
'f:replicas': {}
'f:template':
'f:spec':
'f:containers': {}
'f:initContainers': {}
...
manager: rollouts-controller
operation: Update
time: '2022-01-13T05:10:02Z'
spec:
...Examining the managedFields above, we can see that the rollouts-controller manager owns some fields in the Rollout resource. We can configure the ArgoCD Application so it will ignore all of these fields during the diff stage. The example below shows how this can be achieved:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
...
spec:
...ignoreDifferences:
- group: "*"
kind: "*"
managedFieldsManagers:
- rollouts-controller
Conclusion
Diff customization is a useful feature to address some edge cases especially when resources are incompatible with GitOps or when the user doesn’t have the access to remove fields from the desired state. Custom diffs configured with the new sync option deviates from a purist GitOps approach and the general approach remains leaving room for imperativeness whenever possible and use diff customization with caution for the edge cases.
