fluxcd / flux2-multi-tenancy

Helm and HelmRelease

This document outlines how Helm is integrated into the multi-tenancy setup and how HelmReleases are utilized for application deployments.

HelmRelease Manifest Structure

  • HelmRelease resources are defined in the base directory of each tenant repository. This directory contains the basic HelmRelease configuration that is common to both staging and production environments.

  • staging and production directories contain Kustomize patches that modify the HelmRelease configurations for each environment.

    • staging directory contains patches specific to the staging cluster.
    • production directory contains patches specific to the production cluster.

Tenant Isolation and Service Accounts

  • The platform admin repository contains a patch that sets a default service account for kustomize-controller and helm-controller.
  • If a tenant does not specify a service account in a Kustomization or HelmRelease, it automatically defaults to the default service account.
  • It is recommended that the default service account has no privileges.
  • Each named service account should adhere to the least privilege model.

HelmRelease Examples

  • podinfo-release.yaml: This file defines the HelmRelease for deploying the podinfo application.
  • podinfo-repository.yaml: This file defines the HelmRepository for the podinfo chart.
  • podinfo-values.yaml: This file contains environment-specific values for the podinfo HelmRelease.

Enforcing Tenant Isolation

  • To enforce tenant isolation, the platform admin must configure Flux to reconcile Kustomization and HelmRelease kinds by impersonating a service account from the namespace where these objects are created.
  • Flux has built-in multi-tenancy lockdown features that enable tenant isolation at the Control Plane level.
  • These features prevent cross-namespace references and deny access to remote Kustomize bases.

Cluster Bootstrap and Infrastructure Setup

  • During cluster bootstrap, the platform admin configures Flux to deploy the validation webhook and its policies before reconciling the tenants repositories.
  • The clusters directory defines the order in which infrastructure items and tenant workloads are reconciled on the staging and production clusters.
  • Kyverno, an admission controller, is used to enforce policies and manage the reconciliation process.

Policies and Security

  • The verify-flux-images policy ensures that all Flux images used are built and signed by the Flux team.
  • Kyverno custom resources can be used to enforce a specific GitHub organization for container images.
  • Additional policies can be implemented to restrict repository access and align image policies with pod security contexts.

References

Top-Level Directory Explanations

clusters/ - This directory contains configuration and scripts for managing Kubernetes clusters.

clusters/production/ - This directory contains configuration and scripts for managing the production Kubernetes cluster.

clusters/production/flux-system/ - This directory contains configuration and scripts for the FluxCD system in the production cluster.

clusters/staging/ - This directory contains configuration and scripts for managing the staging Kubernetes cluster.

clusters/staging/flux-system/ - This directory contains configuration and scripts for the FluxCD system in the staging cluster.

infrastructure/ - This directory contains infrastructure-related configuration files and scripts.

infrastructure/kyverno-policies/ - This directory contains the actual policy files for Kyverno.

infrastructure/kyverno/ - This directory contains configuration files and scripts for Kyverno, an open-source Kubernetes policy engine.

scripts/ - This directory contains scripts used for various tasks, such as automation and deployment.

tenants/ - This directory contains configuration and scripts for managing tenants, which are separate namespaces or projects within the Kubernetes cluster.

tenants/base/ - This directory contains configuration and scripts for the base tenant.

tenants/base/dev-team/ - This directory contains configuration and scripts for the development team within the base tenant.

tenants/production/ - This directory contains configuration and scripts for the production tenant.

tenants/staging/ - This directory contains configuration and scripts for the staging tenant.