Hassle-free multi-tenant K8S clusters management using Argo CD

One of the mains goals of Argo CD is to provide a great experience of automating day-to-day tasks that required to manage Kubernetes clusters. The GitOps functions, a convenient web and command-line interface make it the perfect tool for developers who deploy applications to Kubernetes. Besides developers, there are operators: members of the platform team who run Kubernetes clusters and tools like Argo CD for the whole organization. How about them?

Argo CD can help them too! Keep reading to learn which Argo CD features allow building a multi-tenant platform on top of a fleet of multi-tenant Kubernetes clusters.

Shared Kubernetes Cluster

A key feature of Kubernetes is a single API that allows managing any piece of your infrastructure. It significantly lowers the barrier and allows developers to manage infrastructure independently without constantly asking for help from the platform team. At the same time platform team have to protect developers from shooting themselves in foot and stepping on each other in a shared cluster. RBAC is a built-in Kubernetes feature that allows controlling access in the shared cluster. The Argo CD fits very well into that model. You can just run one Argo CD instance per tenant and rely solely on Kubernetes RBAC.

This setup is very secure and guarantees that cluster tenants don’t get any extra privileges. However, it also requires managing a lot of RBAC rules and does not allow to get maximum out of Argo CD.

Multiple Shared Kubernetes Clusters

Things quickly get even more complex once you have more than one cluster which is a common case in real life. The same team often host applications in different Kubernetes clusters and multiple namespaces. That is a lot of RBAC settings. Instead, you can use one Argo CD instance to manage all clusters in your organization and leverage features like SSO integration, RBAC, and Projects to enforce necessary security boundaries.

To make it more interesting, let’s learn and practice at the same time! For a start, we need a test Kubernetes cluster with Argo CD installed. If you have both, feel free to move to the next paragraph.

Install Argo CD and Minikube

Let’s use minikube as a sample cluster. The installation process is described at https://kubernetes.io/docs/setup/ . As soon as use have cluster, go ahead and install Argo CD as described at https://argoproj.github.io/argo-cd/getting_started/

TL;DR:

If you are a Mac user, you can install minikube and Argo CD CLI using the following command:

brew install minikube argocd

Create a new minikube cluster and install Argo CD:

minikube start -p argocd-multi-tenancy-demo
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

By default, the Argo CD API and UI are not exposed outside of the cluster. Change Kubernetes service type and run minikube tunnel using the following commands:

kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "LoadBalancer"}}'
minikube tunnel -p argocd-multi-tenancy-demo

Done! Now we have a Kubernetes cluster with Argo CD running on it! If you navigate to https://localhost/ you should see the Argo CD login page.

SSO Integration

The Argo CD has built-in accounts support however main use case of that feature is the ability to create a token for API access. Instead of manually registering accounts for your teammates you are welcome to configure SSO integration and immediately onboard everyone in your team. Argo CD speaks OpenID Connect (OIDC) — modern authentication protocol based on the OAuth 2.0. If your organization is using OIDC then you are ready to go. No problem if you are using something else — we still got you covered. Default Argo CD installation bundles Dex. Dex is a federated OpenID Connect provider that acts as a proxy and allows connecting Argo CD to other identity providers such as LDAP, SAML, GitHub, Google, and many others.

For the sake of simplicity let’s use Github as an identity provider. Please navigate to https://github.com/settings/applications/new and create Github OAuth application:

Once the application is created, use the following command to patch argocd-cm ConfigMap and configure Github as an identity provider:

export CLIENT_ID=<clientID>
export CLIENT_SECRET=<clientSecret>
kubectl apply -n argocd -f - << EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  labels:
    app.kubernetes.io/part-of: argocd
data:
  url: https://localhost
  dex.config: |
    connectors:
      - type: github
        id: github
        name: GitHub
        config:
          clientID: $CLIENT_ID
          clientSecret: $CLIENT_SECRET
          loadAllGroups: true
EOF

The <clientID> and <clientSecret> are Github application client id and secret which can be obtained using the Github application profile page. Done? After refreshing the https://localhost/ you should be able to login usingLOGIN VIA GITHUB button.

RBAC

The above Dex configuration allows logging our Argo CD anything with zero Github account but with zero permissions. So you won’t be able to make any configuration changes and won’t see any Argo CD applications. To fix it we need to change the RBAC configuration and grant admin permissions to our account. The RBAC configuration is defined in argocd-rbac-cm ConfigMap. For the sake of this exercise, let configure admin access based on Github account email. Run the following command to apply the configuration:

export ADMIN_EMAIL='your@email.com'
kubectl apply -n argocd -f - << EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-rbac-cm
  namespace: argocd
  labels:
    app.kubernetes.io/name: argocd-rbac-cm
    app.kubernetes.io/part-of: argocd
data:
  policy.csv: |
    g, $ADMIN_EMAIL, role:admin
  scopes: '[groups, email]'
EOF

Let’s take a look at the policy.csv field of a ConfigMap. The Argo CD is not reinventing the wheel and uses an existing open source project Casbin to implement RBAC management. We’ve configured the policy that grants a built-in admin role to the user with the specified email. You can add more Casbin policies to define additional roles and grant more permissions.

The next setting is scopes . It specifies which OIDC scopes to examine during RBAC enforcement. By default Argo CD analyzes groups scope, so we’ve added email scope to enable Email-based authorization.

You are an Argo CD administrator now! As the next step, you can configure fine-grained access for all users or your Argo CD instance by adding more Casbin policies. However, this approach won’t scale if you have hundreds of uses and thousands of applications. Instead, let learn how you can leverage Argo CD projects.

Projects

Projects provide a logical application grouping and help to separate users from each other in the multitenant Argo CD instance. As the Argo CD operator, you can create one project per team and use project settings to specify where and what members of your team can deploy. Argo CD comes with a bundled default project that provides full access. Let’s create a sample project that provides limited access to one team:

kubectl apply -n argocd -f - << EOF
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: team1
spec:
  destinations:
  - namespace: my-team-1-*
    server: '*'
  sourceRepos:
  - https://github.com/my-team1/*
  roles:
  - name: my-team-1-admin
    groups:
    - my-team1
    policies:
    - p, proj:team1:my-team-1-admin, applications, *, team1/*, allow
EOF

The first two interesting fields on our project are sourceRepos and destinations:

• sourceRepos field specifies the list of repository URLs that are allowed for deployment. You can use it to enforce usage of only internally hosted Git provider or, like in our example, only repositories that belong to a specific Github organization.
• destinations field specifies which Kubernetes clusters and namespaces can be used for deployment. Here we are allowing deployment to any Kubernetes namespace in any cluster as long as the namespace name has my-team-1- prefix.

Finally, the sample project has a role my-team-1-admin that provides full project access to any member of OIDC group my-team1.

As soon as the project is created Argo CD operator no longer needs to curate each and every user action. The members of my-team1 OIDC group will be able to help themself and manage all applications of team1 project. At the same time, they won’t see applications of other teams and won’t be able to accidentally modify Kubernetes resources that don’t belong to them.

What is Next?

Argo CD projects have more features. In addition to sources and destinations, you can fine-tune the list of Kubernetes resources that can be managed in that project. You can generate tokens and automate project creation and management.

Finally, the next version of the Argo CD is getting more features that make multi-tenancy easier. The ApplicationSet is one of such features. Stay tuned and don’t hesitate to share your ideas of how Argo CD can be improved!