Best Practices for Multi-tenancy in Argo CD
This blogpost is co-authored by Dan Garfield and Jesse Suen. Special thanks to Alexander Matyushentsev, Jann Fischer, Henrik Blixt, and the amazing community for all their hard work in making Argo CD secure.
Argo CD’s ease-of-use has led many teams to view it as the easiest gateway to GitOps. With that, we’ve seen many different ways of using Argo arise with more and more questions around Multi-tenancy as DevOps teams look at how best to enable their engineers to take advantage of Argo CD. When it comes to multi-tenancy there are a few key things teams should consider when architecting and understanding what multi-tenancy in Argo CD is designed to do, and where limitations exist.
First, Argo CD supports RBAC and SSO which allows admins to specify and scope permissions based on user or role. This can limit which resources, apps, clusters, and actions can be taken by different users. Team A and Team B can have distinct access to write or read different applications within a cluster. These permissions and their scoping are generally considered safe.
Argo CD Security Scope
Most Argo CD adopters looking for multi-tenant support do so with the intention of only bringing employees and other at least semi-trusted actors into their systems. But there are additional considerations to make when users may include untrusted actors. When that may be the case, we generally advise that each team operate in their own Argo CD instance and not in a multi-tenant environment. There are several reasons for this.
Firstly, Argo CD operates with the privileges necessary to manage resources on clusters and deploy new ones. It is beyond the scope of Argo CD to validate that when users deploy a container that the container they specify not be malicious. Argo CD’s job is to validate that a user is deploying to a resource they have access to. Beyond that, users should take special care of their software supply chain to understand where containers originate, who built them, and run security tools to prevent merging configuration into their git repos that have security issues.
Likewise, connected git repositories should have their own security protocols to prevent malicious code or merges. Argo CD’s role within the software supply chain does not involve validating that connected git repos are free of malicious code.
Config Management Tools
Ultimately Argo CD needs YAML that it can apply against the Kubernetes API. Beyond supporting plain manifests, config management tools like Helm, Kustomize, and others are supported. Those tools bring convenience but have features beyond simply generating YAML. For example, Helm allows symlinking resources outside of its own directories. JSonnet allows for full code execution. Kustomize can reference a single external base that has many external references beyond itself. Any of these external references can be a potential vector to either execute malicious code within Argo CD or deploy malicious code into Kubernetes clusters.
For Argo CD, this risk is relative depending on implementation choices users are expected to make. Do any of your applications put secrets into manifests? Do users have permission to modify git repositories or their contents? Is any sensitive data stored on the Argo CD instance? These are all implementation choices that carry their own risks.
Some mitigations have been made and additional features are planned for future releases to improve security in these areas. However, these simple precautions should not be understood to prevent all possible traversals though they make those traversals more difficult. For example, In the future, Argo CD may run manifest generation tools in their own sandbox, but that would require configuring Argo CD with more permissions than it has today.
Today, there are two choices when using Argo CD with untrusted users:
- Admins can carefully control the contents of connected git repositories via permissions in git and set limitations on which repositories can be added to Argo CD.
- Each set of users can operate on independent instances of Argo CD. Either approach will prevent users from accessing secrets or components that are beyond their expected scope.
Conclusion
Teams considering multi-tenant setups of Argo CD should very carefully consider which tools they allow to be used and what their security implications are. RBAC should not be considered a silver bullet. The manner in which you operate Argo CD instances is highly dependent on the organization’s policies on tenancy as well as assumptions about access. For example, running a single Argo CD instance might work for a small engineering team with a high degree of mutual trust. On the other hand, a large organization with hundreds of engineers with write privileges to git deployment repos will likely prefer to run multiple instances to provide stronger isolation and smaller blast radiuses. If admins decide to go into a fully multi-tenant model they will need to take extra steps and precautions to maintain security across their instances.
For more information about running Argo CD securely please review the operator’s manual which always has the latest information.
