fluxcd / flux2-multi-tenancy

GitOps and Kustomize

This repository serves as a starting point for managing multi-tenant clusters with Git and Flux v2.

Infrastructure

  • clusters dir: Contains the Flux configuration per cluster
  • infrastructure dir: Contains common infra tools such as admission controllers, CRDs and cluster-wide polices
  • tenants dir: Contains namespaces, service accounts, role bindings and Flux custom resources for registering tenant repositories

Tenants

  • base dir: Contains HelmRepository and HelmRelease manifests
  • staging dir: Contains HelmRelease Kustomize patches for deploying pre-releases on the staging cluster

GitOps Implementation

  • Git as the source of truth: All Kubernetes configurations are stored in Git.
  • Flux as the operator: Flux is used to automatically reconcile the cluster state with the Git repository.
  • Kustomize for overlays: Kustomize is used to apply overlays on base configurations, allowing for environment-specific customizations.

Kustomization files

  • clusters/staging/flux-system/kustomization.yaml: Contains the Flux components for the staging cluster.

    apiVersion: kustomize.config.k8s.io/v1beta1
          kind: Kustomization
          resources:
            - gotk-components.yaml
            - gotk-sync.yaml
          patches:
            - patch: |
                - op: add
                  path: /spec/template/spec/containers/0/args/-
                  value: --no-cross-namespace-refs=true
              target:
                kind: Deployment
                name: "(kustomize-controller|helm-controller|notification-controller|image-reflector-controller|image-automation-controller)"
            - patch: |
                - op: add
                  path: /spec/template/spec/containers/0/args/-
                  value: --no-remote-bases=true
              target:
                kind: Deployment
                name: "kustomize-controller"
            - patch: |
                - op: add
                  path: /spec/template/spec/containers/0/args/-
                  value: --default-service-account=default
              target:
                kind: Deployment
                name: "(kustomize-controller|helm-controller)"
            - patch: |
                - op: add
                  path: /spec/serviceAccountName
                  value: kustomize-controller
              target:
                kind: Kustomization
                name: "flux-system"

  • clusters/production/flux-system/kustomization.yaml: Contains the Flux components for the production cluster. 

    apiVersion: kustomize.config.k8s.io/v1beta1
          kind: Kustomization
          resources:
            - gotk-components.yaml
            - gotk-sync.yaml
          patches:
            - patch: |
                - op: add
                  path: /spec/template/spec/containers/0/args/-
                  value: --no-cross-namespace-refs=true
              target:
                kind: Deployment
                name: "(kustomize-controller|helm-controller|notification-controller|image-reflector-controller|image-automation-controller)"
            - patch: |
                - op: add
                  path: /spec/template/spec/containers/0/args/-
                  value: --no-remote-bases=true
              target:
                kind: Deployment
                name: "kustomize-controller"
            - patch: |
                - op: add
                  path: /spec/template/spec/containers/0/args/-
                  value: --default-service-account=default
              target:
                kind: Deployment
                name: "(kustomize-controller|helm-controller)"
            - patch: |
                - op: add
                  path: /spec/serviceAccountName
                  value: kustomize-controller
              target:
                kind: Kustomization
                name: "flux-system"

  • clusters/staging/infrastructure.yaml: Contains Kyverno configurations for the staging cluster.

    apiVersion: kustomize.toolkit.fluxcd.io/v1
          kind: Kustomization
          metadata:
            name: kyverno
            namespace: flux-system
          spec:
            interval: 720m0s
            sourceRef:
              kind: GitRepository
              name: flux-system
            serviceAccountName: kustomize-controller
            path: ./infrastructure/kyverno
            prune: true
            wait: true
            timeout: 10m
          ---
          apiVersion: kustomize.toolkit.fluxcd.io/v1
          kind: Kustomization
          metadata:
            name: kyverno-policies
            namespace: flux-system
          spec:
            dependsOn:
              - name: kyverno
            interval: 15m0s
            sourceRef:
              kind: GitRepository
              name: flux-system
            serviceAccountName: kustomize-controller
            path: ./infrastructure/kyverno-policies
            prune: true

  • clusters/production/infrastructure.yaml: Contains Kyverno configurations for the production cluster. 

    apiVersion: kustomize.toolkit.fluxcd.io/v1
          kind: Kustomization
          metadata:
            name: kyverno
            namespace: flux-system
          spec:
            interval: 10m
            sourceRef:
              kind: GitRepository
              name: flux-system
            serviceAccountName: kustomize-controller
            path: ./infrastructure/kyverno
            prune: true
            wait: true
            timeout: 5m
          ---
          apiVersion: kustomize.toolkit.fluxcd.io/v1
          kind: Kustomization
          metadata:
            name: kyverno-policies
            namespace: flux-system
          spec:
            dependsOn:
              - name: kyverno
            interval: 5m
            sourceRef:
              kind: GitRepository
              name: flux-system
            serviceAccountName: kustomize-controller
            path: ./infrastructure/kyverno-policies
            prune: true

  • clusters/staging/tenants.yaml: Contains the configuration for tenants on the staging cluster.

    apiVersion: kustomize.toolkit.fluxcd.io/v1
          kind: Kustomization
          metadata:
            name: tenants
            namespace: flux-system
          spec:
            dependsOn:
              - name: kyverno-policies
            interval: 5m
            serviceAccountName: kustomize-controller
            sourceRef:
              kind: GitRepository
              name: flux-system
            path: ./tenants/staging
            prune: true

  • clusters/production/tenants.yaml: Contains the configuration for tenants on the production cluster. 

    apiVersion: kustomize.toolkit.fluxcd.io/v1
          kind: Kustomization
          metadata:
            name: tenants
            namespace: flux-system
          spec:
            dependsOn:
              - name: kyverno-policies
            interval: 5m
            serviceAccountName: kustomize-controller
            sourceRef:
              kind: GitRepository
              name: flux-system
            path: ./tenants/production
            prune: true

  • infrastructure/kyverno/sync.yaml: Defines the synchronization of the Kyverno controller.

    apiVersion: kustomize.toolkit.fluxcd.io/v1
          kind: Kustomization
          metadata:
            name: kyverno-controller
            namespace: flux-system
          spec:
            interval: 720m0s
            sourceRef:
              kind: OCIRepository
              name: kyverno-controller
            serviceAccountName: kustomize-controller
            path: ./
            prune: true
            wait: true
            timeout: 5m

  • tenants/base/dev-team/kustomization.yaml: Defines the resources for the dev-team namespace.

    apiVersion: kustomize.config.k8s.io/v1beta1
          kind: Kustomization
          namespace: apps
          resources:
            - rbac.yaml
            - sync.yaml

  • infrastructure/kyverno-policies/kustomization.yaml: Defines the Kyverno policies.

    apiVersion: kustomize.config.k8s.io/v1beta1
          kind: Kustomization
          resources:
            - verify-flux-images.yaml
            - verify-git-repositories.yaml

  • infrastructure/kyverno/kustomization.yaml: Defines the Kyverno resources.

    apiVersion: kustomize.config.k8s.io/v1beta1
          kind: Kustomization
          resources:
            - source.yaml
            - sync.yaml

  • tenants/staging/kustomization.yaml: Defines the staging overlay for the dev-team tenant.

    apiVersion: kustomize.config.k8s.io/v1beta1
          kind: Kustomization
          namespace: apps
          resources:
            - ../base/dev-team
          patches:
            - path: dev-team-patch.yaml

  • tenants/production/kustomization.yaml: Defines the production overlay for the dev-team tenant.

    apiVersion: kustomize.config.k8s.io/v1beta1
          kind: Kustomization
          namespace: apps
          resources:
            - ../base/dev-team
          patches:
            - path: dev-team-patch.yaml

  • tenants/staging/dev-team-patch.yaml: Contains the patch for the dev-team tenant in staging.

    apiVersion: kustomize.toolkit.fluxcd.io/v1
          kind: Kustomization
          metadata:
            name: dev-team
            namespace: apps
          spec:
            path: ./staging

Security

  • Secret encryption: Secrets are encrypted with SOPS CLI using OpenPGP, Age or KMS.
  • Multi-tenancy lockdown: Flux’s multi-tenancy lockdown feature enforces tenant isolation at the control plane level.

CI/CD

  • Test workflow: Validates Kubernetes manifests and Kustomize overlays with kubeconform.
  • E2E workflow: Tests the staging setup by running Flux in Kubernetes Kind.

Bootstrapping Flux

  1. Install the Flux CLI.
  2. Fork the repository to your personal GitHub account.
  3. Set the GITHUB_USER and GITHUB_REPO environment variables.
  4. Verify prerequisites with flux check --pre.
  5. Bootstrap Flux with flux bootstrap github.

Top-Level Directory Explanations

clusters/ - This directory contains configuration and scripts for managing Kubernetes clusters.

clusters/production/ - This directory contains configuration and scripts for managing the production Kubernetes cluster.

clusters/production/flux-system/ - This directory contains configuration and scripts for the FluxCD system in the production cluster.

clusters/staging/ - This directory contains configuration and scripts for managing the staging Kubernetes cluster.

clusters/staging/flux-system/ - This directory contains configuration and scripts for the FluxCD system in the staging cluster.

infrastructure/ - This directory contains infrastructure-related configuration files and scripts.

infrastructure/kyverno-policies/ - This directory contains the actual policy files for Kyverno.

infrastructure/kyverno/ - This directory contains configuration files and scripts for Kyverno, an open-source Kubernetes policy engine.

scripts/ - This directory contains scripts used for various tasks, such as automation and deployment.

tenants/ - This directory contains configuration and scripts for managing tenants, which are separate namespaces or projects within the Kubernetes cluster.

tenants/base/ - This directory contains configuration and scripts for the base tenant.

tenants/base/dev-team/ - This directory contains configuration and scripts for the development team within the base tenant.

tenants/production/ - This directory contains configuration and scripts for the production tenant.

tenants/staging/ - This directory contains configuration and scripts for the staging tenant.