Breaking Changes in Argo CD 2.4

Michael Crenshaw
Argo Project
Published in
5 min readJun 6, 2022

Argo CD 2.4 includes some awesome improvements and also gave the Argo CD team an opportunity to clean up some tech debt.

This clean-up introduced some breaking changes, so make sure to read up on these changes below to ensure a smooth upgrade process.

Update your Helm 2 or Ksonnet manifests

Argo CD 2.4 removes Helm 2 and Ksonnet support, as these have reached end of life.

If you use Ksonnet, convert your manifests to a supported config management tool. If you use Helm 2, follow their migration guide before upgrading to Argo CD 2.4.

Update your RBAC to handle Web Terminal

Argo CD 2.4 introduces a new Web Terminal feature. This allows users to “exec into” Pods via the Argo CD UI.

It’s an awesome feature, and it’s also powerful. If you use RBAC wildcards, you might unintentionally enable this feature for users that shouldnt have access to it when upgrading to 2.4.

The third column in your Argo CD RBAC config is the “resource” column. If you have a wildcard (*) in that column, the new exec resource is automatically included. For example:

p, role:org-admin, *, create, my-proj/*, allow

To continue allowing all the non-exec resources, enumerate them instead of using *. You would replace the example above with the following:

p, role: org-admin, clusters, create, my-proj/*, allow
p, role: org-admin, projects, create, my-proj/*, allow
p, role: org-admin, applications, create, my-proj/*, allow
p, role: org-admin, repositories, create, my-proj/*, allow
p, role: org-admin, certificates, create, my-proj/*, allow
p, role: org-admin, accounts, create, my-proj/*, allow
p, role: org-admin, gpgkeys, create, my-proj/*, allow

Test the repository sever with its new Service Account

As a security enhancement, Argo CD 2.4’s install manifests include a dedicated Service Account for the repository server Deployment.

If your setup relies on the repository server using the default Service Account (for example, if a plugin uses the default Service Account for auth), upgrading to Argo CD 2.4 could cause problems. You may need to replicate customizations from the default Service Account to the argocd-repo-server Service Account.

We recommend testing Argo CD 2.4 on a non-production instance before upgrading.

Update your plugins’ configuration

The introduction of config management plugins has allowed people to customize Argo CD in many incredible ways! The plugin system is designed for flexibility, and by necessity plugins are granted a certain level of trust in the Argo CD architecture. This also means that Argo CD can’t validate or guarantee that an external plugin is secure! Design your custom plugins carefully and always audit any third-party plugins before installing them.

The following breaking changes in 2.4 are designed to help plugin authors keep their implementation secure.

If you use config management plugins, it’s critical to read the following three sections carefully. If you do not use config management plugins, feel free to skip to the Conclusion!

Update your plugins to use newly-prefixed environment variables

Config management plugins accept user-defined variables from the Application manifest. For example:

apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
source:
plugin:
env:
- name: FOO
value: bar

Before 2.4, those environment variables were injected in the init and generate commands configured in the plugin.

Since many plugins use third-party tools whose behavior is configurable via environment variables (such as git), we do not want to allow users to set arbitrary environment variables.

Starting with Argo CD 2.4, all user-supplied environment variables will be prefixed with ARGOCD_ENV_ before being sent to the plugin’s commands. The plugin author can still choose to remove the prefix (export FOO=$ARGOCD_ENV_FOO), but they must do so explicitly.

If you have written a custom config management plugin which relies on user-supplied environment variables, update your plugin logic to handle the new prefix before upgrading to 2.4.

If you use a third-party plugin, make sure the plugin is compatible with Argo CD 2.4. If the project doesn’t explicitly advertise support for 2.4, open an issue to verify whether they have added support.

(Note: If you don’t need a new 2.4 feature, it’s safe to stay on the 2.3.x series until all your plugins support 2.4. We’ll continue publishing security patches for 2.3.x until 2.6.0 is released.)

Confirm that your sidecar plugin environment variables are set on the sidecar container

If you use a plugin that’s installed as a sidecar (instead of in the argocd-cm ConfigMap), a bug fix in 2.4 could prevent the plugin from receiving all its required environment variables.

Before 2.4, environment variables on the main repo-server container were sent to the plugin’s commands, taking precedence over environment variables set on the sidecar.

Starting with 2.4, environment variables (besides the standard build environment and user-supplied variables) must be set directly on the sidecar container in order to be picked up by the configured plugin commands.

Before upgrading to Argo CD 2.4, check your sidecar plugins’ configurations to see what environment variables they require. If any of them is set on the main repo-server container instead of the sidecar container, you might need to also set the environment variable on the sidecar container.

Replace the shared /tmp volume in plugin sidecars

If you use a plugin that’s installed as a sidecar (instead of in the argocd-cm ConfigMap), it mounts a volume at /tmp. Before 2.4, that volume needed to be the same as the temp volume mounted by the main repo-server container.

Starting with Argo CD 2.4, each sidecar plugin should mount its own temp volume. This helps mitigate the risk of directory traversal attacks. For example:

apiVersion: apps/v1
kind: Deployment
metadata:
name: argocd-repo-server
spec:
template:
spec:
containers:
- name: your-plugin-name
volumeMounts:
- mountPath: /tmp
name: your-plugin-name-tmp
volumes:
# Add this volume.
- name: your-plugin-name-tmp
emptyDir: {}

Extra credit: prepare for 2.5 by enabling logs RBAC enforcement

Argo CD 2.5 will include finer-grained RBAC control of logs access. You can opt in to log RBAC control in 2.4 to make the 2.4-to-2.5 upgrade process faster and easier.

Conclusion

The Argo CD team takes breaking changes seriously. We’ll only include a breaking change if we’re certain it’s critical for the health of the project and to the users.

The items above are also listed in the 2.3 to 2.4 upgrade guide. If you are upgrading from an older version, read the other upgrade guides before skipping versions. If you follow all the instructions, the upgrade process should go smoothly.

If you have questions or encounter issues, reach out on Slack or GitHub, and we’ll be happy to help!

Free

Distraction-free reading. No ads.

Organize your knowledge with lists and highlights.

Tell your story. Find your audience.

Membership

Read member-only stories

Support writers you read most

Earn money for your writing

Listen to audio narrations

Read offline with the Medium app

Responses (1)

What are your thoughts?