Multi-Tenancy Architecture

This repository provides a starting point for managing multi-tenant clusters with Git and Flux v2. https://github.com/fluxcd/flux2-multi-tenancy/

Tenant Roles

The Flux multi-tenancy architecture involves two primary roles: Platform Admin and Tenant.

Platform Admin

• Has cluster admin access to the fleet of clusters
• Has maintainer access to the fleet Git repository
• Manages cluster-wide resources (CRDs, controllers, cluster roles, etc)
• Onboards the tenant’s main GitRepository and Kustomization
• Manages tenants by assigning namespaces, service accounts, and role binding to the tenant’s apps

Tenant

• Has admin access to the namespaces assigned to them by the platform admin
• Has maintainer access to the tenant Git repository and apps repositories
• Manages app deployments with GitRepositories and Kustomizations
• Manages app releases with HelmRepositories and HelmReleases

Tenant Creation

The Flux CLI provides commands to generate the Kubernetes manifests needed to define tenants.

Assuming a platform admin wants to create a tenant named dev-team with access to the apps namespace:

1. Create the tenant base directory:

mkdir -p ./tenants/base/dev-team
          

2. Generate the namespace, service account, and role binding for the dev-team:

flux create tenant dev-team --with-namespace=apps \
          --export > ./tenants/base/dev-team/rbac.yaml
          

3. Create the sync manifests for the tenant Git repository:

flux create source git dev-team \
          --namespace=apps \
          --url=https://github.com// \
          --branch=main \
          --export > ./tenants/base/dev-team/sync.yaml
          

Cluster Bootstrap

On cluster bootstrap, you need to configure Flux to deploy the validation webhook and its policies before reconciling the tenants repositories.

Inside the clusters directory, we define the order in which the infrastructure items and the tenant workloads are going to be reconciled on the staging and production clusters:

./clusters/
          ├── production
          │   ├── infrastructure.yaml
          │   └── tenants.yaml
          └── staging
          ├── infrastructure.yaml
          └── tenants.yaml
          

First, we setup the reconciliation of custom resource definitions and their controllers. For this example, we’ll use Kyverno:

apiVersion: kustomize.toolkit.fluxcd.io/v1
          kind: Kustomization
          metadata:
          name: kyverno
          namespace: flux-system
          spec:
          interval: 10m
          sourceRef:
          kind: GitRepository
          name: flux-system
          path: ./infrastructure/kyverno
          

Policy Enforcement

Any change to the Kubernetes manifests or the repository structure should be validated in CI before a pull request is merged into the main branch and synced on the cluster. This repository contains the following GitHub CI workflows:

• The test workflow validates the Kubernetes manifests and Kustomize overlays with kubeconform.
• The e2e workflow starts a Kubernetes cluster in CI and tests the staging setup by running Flux in Kubernetes Kind. https://fluxcd.io/flux/installation/configuration/multitenancy/

Tenant Repository Authentication

You can configure Flux to connect to a tenant repository using SSH or token-based authentication. The tenant credentials will be stored in the platform admin repository as a Kubernetes secret.

Repository Access Restrictions

Some deployments may require that all sources come from a specific GitHub Organisation, as the verify-git-repositories policy demonstrates:

apiVersion: kyverno.io/v1
          kind: ClusterPolicy
          metadata:
          name: verify-git-repositories
          spec:
          validationFailureAction: audit # Change to 'enforce' once the specific org url is set.
          rules:
          - name: github-repositories-only
          exclude:
          resources:
          namespaces:
          - flux-system
          match:
          resources:
          kinds:
          - GitRepository
          validate:
          message: ".spec.url must be from a repository within the organisation X"
          anyPattern:
          - spec:
          url: "https://github.com/fluxcd/?*" # repositories in fluxcd via https
          - spec:
          url:  # repositories in fluxcd via ssh
          

Tenant Onboarding Process

The tenant onboarding process involves several steps:

1. Create the tenant namespace, service account, and role binding.
2. Decrypt the tenant Git credentials using the GPG private key.
3. Create the tenant Git credentials Kubernetes secret in the tenant namespace.
4. Clone the tenant repository using the supplied credentials.
5. Apply the ./staging directory from the tenant’s repo using the tenant’s service account.

Custom Resource Definitions

The clusters/staging/flux-system/kustomization.yaml file defines a patch that modifies the service account used by the Flux controllers:

target:
          kind: Deployment
          name: "(kustomize-controller|helm-controller|notification-controller|image-reflector-controller|image-automation-controller)"
          - patch: |
          - op: add
          path: /spec/template/spec/containers/0/args/-
          value: --no-remote-bases=true
          target:
          kind: Deployment
          name: "kustomize-controller"
          - patch: |
          - op: add
          path: /spec/template/spec/containers/0/args/-
          value: --default-service-account=default
          target:
          kind: Deployment
          name: "(kustomize-controller|helm-controller)"
          - patch: |
          - op: add
          path: /spec/serviceAccountName
          value: kustomize-controller
          target:
          kind: Kustomization
          name: "flux-system"
          

Cluster Bootstrap Process

The flux bootstrap command commits the manifests for the Flux components in clusters/staging/flux-system directory and creates a deploy key with read-only access on GitHub, so it can pull changes inside the cluster. Wait for the staging cluster reconciliation to finish:

$ flux get kustomizations --watch
          NAME            	READY  	MESSAGE
          flux-system     	True   	Applied revision: main/616001c38e7bc81b00ef2c65ac8cfd58140155b8
          kyverno         	Unknown	Reconciliation in progress
          kyverno-policies	False  	Dependency 'flux-system/kyverno' is not ready
          tenants         	False  	Dependency 'flux-system/kyverno-policies' is not ready
          

Examples

Tenant Manifests

tenants/base/dev-team/sync.yaml

apiVersion: source.toolkit.fluxcd.io/v1
          kind: GitRepository
          metadata:
            name: dev-team
            namespace: apps
          spec:
            interval: 1m
            url: https://github.com/fluxcd/flux2-multi-tenancy
            ref:
              branch: dev-team
          ---
          apiVersion: kustomize.toolkit.fluxcd.io/v1
          kind: Kustomization
          metadata:
            name: dev-team
            namespace: apps
          spec:
            serviceAccountName: dev-team
            interval: 5m
            sourceRef:
              kind: GitRepository
              name: dev-team
            prune: true
          

tenants/base/dev-team/rbac.yaml

---
          apiVersion: v1
          kind: Namespace
          metadata:
            labels:
              toolkit.fluxcd.io/tenant: dev-team
            name: apps
          
          ---
          apiVersion: v1
          kind: ServiceAccount
          metadata:
            labels:
              toolkit.fluxcd.io/tenant: dev-team
            name: dev-team
            namespace: apps
          
          ---
          apiVersion: rbac.authorization.k8s.io/v1
          kind: RoleBinding
          metadata:
            labels:
              toolkit.fluxcd.io/tenant: dev-team
            name: gotk-reconciler
            namespace: apps
          roleRef:
            apiGroup: rbac.authorization.k8s.io
            kind: ClusterRole
            name: cluster-admin
          subjects:
          - kind: User
            name: gotk:apps:reconciler
          - kind: ServiceAccount
            name: dev-team
            namespace: apps
          

tenants/base/dev-team/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
          kind: Kustomization
          namespace: apps
          resources:
            - rbac.yaml
            - sync.yaml
          

Cluster Manifests

clusters/staging/tenants.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1
          kind: Kustomization
          metadata:
            name: tenants
            namespace: flux-system
          spec:
            dependsOn:
              - name: kyverno-policies
            interval: 5m
            serviceAccountName: kustomize-controller
            sourceRef:
              kind: GitRepository
              name: flux-system
            path: ./tenants/staging
            prune: true
          

clusters/production/tenants.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1
          kind: Kustomization
          metadata:
            name: tenants
            namespace: flux-system
          spec:
            dependsOn:
              - name: kyverno-policies
            interval: 5m
            serviceAccountName: kustomize-controller
            sourceRef:
              kind: GitRepository
              name: flux-system
            path: ./tenants/production
            prune: true
          

tenants/staging/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
          kind: Kustomization
          namespace: apps
          resources:
            - ../base/dev-team
          patches:
            - path: dev-team-patch.yaml
          

tenants/production/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
          kind: Kustomization
          namespace: apps
          resources:
            - ../base/dev-team
          patches:
            - path: dev-team-patch.yaml
          

tenants/staging/dev-team-patch.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1
          kind: Kustomization
          metadata:
            name: dev-team
            namespace: apps
          spec:
            path: ./staging
          

tenants/production/dev-team-patch.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1
          kind: Kustomization
          metadata:
            name: dev-team
            namespace: apps
          spec:
            path: ./production
          

Kyverno Policies

infrastructure/kyverno-policies/verify-git-repositories.yaml

apiVersion: kyverno.io/v1
          kind: ClusterPolicy
          metadata:
            name: verify-git-repositories
          spec:  
            # This provides users a working example of how an admin
            # would be able to enforce git repository sources across
            # all tenants.
            validationFailureAction: Audit # Change to 'Enforce' once the specific org url is set.
            rules:
              - name: github-repositories-only
                match:
                  any:
                  - resources:
                      kinds:
                        - GitRepository
                exclude:
                  any:
                  - resources:
                      namespaces:
                        - flux-system
                validate:
                  message: ".spec.url must be from a repository within the organisation X"
                  pattern:
                    spec:
                      url: https://github.com/fluxcd/?* | 
          

Infrastructure Manifests

infrastructure/kyverno/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
          kind: Kustomization
          resources:
            - source.yaml
            - sync.yaml
          

clusters/staging/infrastructure.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1
          kind: Kustomization
          metadata:
            name: kyverno
            namespace: flux-system
          spec:
            interval: 720m0s
            sourceRef:
              kind: GitRepository
              name: flux-system
            serviceAccountName: kustomize-controller
            path: ./infrastructure/kyverno
            prune: true
            wait: true
            timeout: 10m
          ---
          apiVersion: kustomize.toolkit.fluxcd.io/v1
          kind: Kustomization
          metadata:
            name: kyverno-policies
            namespace: flux-system
          spec:
            dependsOn:
              - name: kyverno
            interval: 15m0s
            sourceRef:
              kind: GitRepository
              name: flux-system
            serviceAccountName: kustomize-controller
            path: ./infrastructure/kyverno-policies
            prune: true
          

clusters/production/infrastructure.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1
          kind: Kustomization
          metadata:
            name: kyverno
            namespace: flux-system
          spec:
            interval: 10m
            sourceRef:
              kind: GitRepository
              name: flux-system
            serviceAccountName: kustomize-controller
            path: ./infrastructure/kyverno
            prune: true
            wait: true
            timeout: 5m
          ---
          apiVersion: kustomize.toolkit.fluxcd.io/v1
          kind: Kustomization
          metadata:
            name: kyverno-policies
            namespace: flux-system
          spec:
            dependsOn:
              - name: kyverno
            interval: 5m
            sourceRef:
              kind: GitRepository
              name: flux-system
            serviceAccountName: kustomize-controller
            path: ./infrastructure/kyverno-policies
            prune: true
          

Top-Level Directory Explanations

clusters/ - This directory contains configuration and scripts for managing Kubernetes clusters.

clusters/production/ - This directory contains configuration and scripts for managing the production Kubernetes cluster.

clusters/production/flux-system/ - This directory contains configuration and scripts for the FluxCD system in the production cluster.

clusters/staging/ - This directory contains configuration and scripts for managing the staging Kubernetes cluster.

clusters/staging/flux-system/ - This directory contains configuration and scripts for the FluxCD system in the staging cluster.

infrastructure/ - This directory contains infrastructure-related configuration files and scripts.

infrastructure/kyverno-policies/ - This directory contains the actual policy files for Kyverno.

infrastructure/kyverno/ - This directory contains configuration files and scripts for Kyverno, an open-source Kubernetes policy engine.

scripts/ - This directory contains scripts used for various tasks, such as automation and deployment.

tenants/ - This directory contains configuration and scripts for managing tenants, which are separate namespaces or projects within the Kubernetes cluster.

tenants/base/ - This directory contains configuration and scripts for the base tenant.

tenants/base/dev-team/ - This directory contains configuration and scripts for the development team within the base tenant.

tenants/production/ - This directory contains configuration and scripts for the production tenant.

tenants/staging/ - This directory contains configuration and scripts for the staging tenant.