fluxcd / flux2-multi-tenancy

Platform and Tenant Roles

This document outlines the roles and responsibilities of both Platform Administrators and Tenants in the multi-tenancy setup of the Flux project.

Platform Admin

  • Has cluster admin access to the fleet of clusters.
  • Has maintainer access to the fleet Git repository.
  • Manages cluster-wide resources (CRDs, controllers, cluster roles, etc.).
  • Onboards the tenant’s main GitRepository and Kustomization.
  • Manages tenants by assigning namespaces, service accounts and role binding to the tenant’s apps.

Tenant

  • Has admin access to the namespaces assigned to them by the platform admin.
  • Has maintainer access to the tenant Git repository and apps repositories.
  • Manages app deployments with GitRepositories and Kustomizations.
  • Manages app releases with HelmRepositories and HelmReleases.

Creating a Tenant

The Flux CLI offers commands to generate the Kubernetes manifests needed to define tenants. Assuming a platform admin wants to create a tenant named dev-team with access to the apps namespace:

  1. Create the tenant base directory:

    mkdir -p ./tenants/base/dev-team

  2. Generate the namespace, service account and role binding for the dev-team:

    flux create tenant dev-team --with-namespace=apps \
          --export > ./tenants/base/dev-team/rbac.yaml

  3. Create the sync manifests for the tenant Git repository:

    flux create source git dev-team \
          --namespace=apps \
          --url=https://github.com// \
          --branch=main \
          --export > ./tenants/base/dev-team/sync.yaml

Tenant Credentials

Flux can connect to a tenant repository using SSH or token-based authentication. The tenant credentials are stored in the platform admin repository as a Kubernetes secret.

Tenant Onboarding

  1. Create the tenant namespace, service account and role binding.
  2. Decrypt the tenant Git credentials using the GPG private key.
  3. Create the tenant Git credentials Kubernetes secret in the tenant namespace.
  4. Clone the tenant repository using the supplied credentials.
  5. Apply the ./staging directory from the tenant’s repo using the tenant’s service account.

Default Service Account

When Flux is bootstrapped, both kustomize-controller and helm-controller will impersonate the default service account in the tenant namespace when applying changes to the cluster. The default service account exists in all namespaces and should always be kept without any privileges. To enable a tenant to operate, a service account must be created with the required permissions and its name set to the spec.serviceAccountName of all Kustomization and HelmRelease resources the tenant has.

Default Service Account Patch

The kustomization.yaml file in both clusters applies a patch that sets a default service account via --default-service-account to kustomize-controller and helm-controller. If a tenant does not specify a service account in a Flux Kustomization or HelmRelease, it would automatically default to the default service account. It is recommended that the default service account has no privileges and each named service account used observes the least privilege model.

Tenant Repository Configuration

apiVersion: kustomize.config.k8s.io/v1beta1
          kind: Kustomization
          namespace: apps
          resources:
          - ../base/dev-team
          patches:
          - path: dev-team-patch.yaml

With this configuration, the Flux instance running on the staging cluster will clone the dev-team’s repository and reconcile the ./staging directory from the tenant’s repo using the dev-team service account. Since this service account is restricted to the apps namespace, the dev-team repository must contain Kubernetes objects scoped to the apps namespace only.

Security Considerations

The Platform Admin may impose additional policies to enforce specific behaviours. Some considerations include:

  • Expand policies to HelmRepository and Bucket.
  • Restrict protocols allowed for HelmRepository and GitRepository.
  • Restrict providers and regions for Bucket.

Policy Validation Webhook

On cluster bootstrap, Flux needs to be configured to deploy the validation webhook and its policies before reconciling the tenants’ repositories. The clusters directory defines the order in which the infrastructure items and the tenant workloads will be reconciled on the staging and production clusters:

./clusters/
          ├── production
          │   ├── infrastructure.yaml
          │   └── tenants.yaml
          └── staging
          ├── infrastructure.yaml
          └── tenants.yaml

First, custom resource definitions and their controllers are set up. For example, using Kyverno:

apiVersion: kustomize.toolkit.fluxcd.io/v1
          kind: Kustomization
          metadata:
          name: kyverno
          namespace: flux-system
          spec:
          interval: 10m
          sourceRef:
          kind: GitRepository
          name: flux-system
          path: ./infrastructure/kyverno

Tenant Repository Verification

Verify that the tenant Git repository has been cloned:

$ flux -n apps get sources git
          NAME    	READY	MESSAGE
          dev-team	True 	Fetched revision: dev-team/ca8ec25405cc03f2f374d2f35f9299d84ced01e4

Verify that the tenant Helm repository index has been downloaded:

$ flux -n apps get sources helm
          NAME   	READY	MESSAGE
          podinfo	True 	Fetched revision: 2022-05-23T10:09:58.648748663Z

Wait for the demo app to be installed:

$ watch flux -n apps get helmreleases
          NAME   	READY	MESSAGE                         	REVISION	SUSPENDED
          podinfo	True 	Release reconciliation succeeded	5.0.3   	False

To expand on this example, check the enforce tenant isolation for security related considerations.

Repository Structure

The platform admin repository contains the following top directories:

  • clusters dir contains the Flux configuration per cluster.
  • infrastructure dir contains common infra tools such as admission controllers, CRDs and cluster-wide polices.
  • tenants dir contains namespaces, service accounts, role bindings and Flux custom resources for registering tenant repositories.
├── clusters
          │   ├── production
          │   └── staging
          ├── infrastructure
          │   ├── kyverno
          │   └── kyverno-policies
          └── tenants
          ├── base
          ├── production
          └── staging

A tenant repository contains the following top directories:

  • base dir contains HelmRepository and HelmRelease manifests.
  • staging dir contains HelmRelease Kustomize patches for deploying pre-releases on the staging cluster.

Alternative Approach

Kyverno’s resource generation feature can be leveraged to create tenants instead of using the flux create tenant approach.

Default Patch Examples

patches:
          - patch: |
            - op: add
              path: /spec/template/spec/containers/0/args/-
              value: --no-cross-namespace-refs=true
            target:
              kind: Deployment
              name: "(kustomize-controller|helm-controller|notification-controller|image-reflector-controller|image-automation-controller)"
          - patch: |
            - op: add
              path: /spec/template/spec/containers/0/args/-
              value: --no-remote-bases=true
            target:
              kind: Deployment
              name: "kustomize-controller"
          - patch: |
            - op: add
              path: /spec/template/spec/containers/0/args/-
              value: --default-service-account=default
            target:
              kind: Deployment
              name: "(kustomize-controller|helm-controller)"
          - patch: |
            - op: add
              path: /spec/serviceAccountName
              value: kustomize-controller
            target:
              kind: Kustomization
              name: "flux-system"

Top-Level Directory Explanations

clusters/ - This directory contains configuration and scripts for managing Kubernetes clusters.

clusters/production/ - This directory contains configuration and scripts for managing the production Kubernetes cluster.

clusters/production/flux-system/ - This directory contains configuration and scripts for the FluxCD system in the production cluster.

clusters/staging/ - This directory contains configuration and scripts for managing the staging Kubernetes cluster.

clusters/staging/flux-system/ - This directory contains configuration and scripts for the FluxCD system in the staging cluster.

infrastructure/ - This directory contains infrastructure-related configuration files and scripts.

infrastructure/kyverno-policies/ - This directory contains the actual policy files for Kyverno.

infrastructure/kyverno/ - This directory contains configuration files and scripts for Kyverno, an open-source Kubernetes policy engine.

scripts/ - This directory contains scripts used for various tasks, such as automation and deployment.

tenants/ - This directory contains configuration and scripts for managing tenants, which are separate namespaces or projects within the Kubernetes cluster.

tenants/base/ - This directory contains configuration and scripts for the base tenant.

tenants/base/dev-team/ - This directory contains configuration and scripts for the development team within the base tenant.

tenants/production/ - This directory contains configuration and scripts for the production tenant.

tenants/staging/ - This directory contains configuration and scripts for the staging tenant.