Flux v2 Fundamentals

This repository acts as a foundational framework for managing multi-tenant Kubernetes clusters through Git and Flux v2. https://github.com/fluxcd/flux2-multi-tenancy/

Key Concepts:

• Platform Admin: This user manages the overall cluster resources and tenants. They have cluster admin access, manage the fleet Git repository, and onboard new tenants.
• Tenant: Tenants have admin access to their assigned namespaces and manage their applications within those namespaces. They utilize GitRepositories and Kustomizations for deployment, and HelmRepositories and HelmReleases for managing application releases.

Workflow:

1. Bootstrapping:

   • Install the Flux CLI and fork this repository on your GitHub account. https://github.com/fluxcd/flux2-multi-tenancy/
   • Set up environment variables for your GitHub username and repo name:

     export GITHUB_USER=
               export GITHUB_REPO=
               

   • Verify the staging cluster prerequisites using:

     flux check --pre
               

   • Bootstrap Flux using the flux bootstrap github command:

     flux bootstrap github \
               --context=your-staging-context \
               --owner=${GITHUB_USER} \
               --repository=${GITHUB_REPO} \
               --branch=main \
               --personal \
               --path=clusters/staging
               

   • Provide your GITHUB_TOKEN (a.k.a. Personal Access Token) during the bootstrapping process. https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token
   • The bootstrap process generates a deploy key with read-only access to your GitHub repository and commits the Flux component manifests in clusters/staging/flux-system. https://github.com/fluxcd/flux2-multi-tenancy/

2. Tenant Onboarding:

   • The Platform Admin provides a namespace, service account, and role binding to the tenant.
   • They then onboard the tenant’s primary GitRepository and Kustomization in the tenants directory. https://github.com/fluxcd/flux2-multi-tenancy/
   • The tenants directory contains namespaces, service accounts, role bindings, and Flux custom resources for registering tenant repositories. https://github.com/fluxcd/flux2-multi-tenancy/

3. Tenant Management:

   • Tenants deploy applications using their assigned GitRepository and Kustomizations. https://github.com/fluxcd/flux2-multi-tenancy/
   • They manage application releases using HelmRepositories and HelmReleases. https://github.com/fluxcd/flux2-multi-tenancy/
   • Any changes to Kubernetes manifests or repository structure are validated in CI before merging and syncing to the cluster. https://github.com/fluxcd/flux2-multi-tenancy/

Repository Structure:

• clusters: Contains Flux configuration per cluster.
• infrastructure: Contains common infrastructure tools like admission controllers, CRDs, and cluster-wide policies.
• tenants: Contains namespaces, service accounts, role bindings, and Flux custom resources for registering tenant repositories. https://github.com/fluxcd/flux2-multi-tenancy/

Tenant Repository Structure:

• base: Contains HelmRepository and HelmRelease manifests.
• staging: Contains HelmRelease Kustomize patches for deploying pre-releases on the staging cluster.
• production: Contains HelmRelease Kustomize patches for deploying stable releases on the production cluster.

CI/CD:

• The repository includes GitHub CI workflows:
  • test: Validates Kubernetes manifests and Kustomize overlays using kubeconform. https://github.com/fluxcd/flux2-multi-tenancy/
  • e2e: Tests the staging setup by running Flux in Kubernetes Kind. https://github.com/fluxcd/flux2-multi-tenancy/

Security Considerations:

• Multi-tenancy Lockdown: Enforces tenant isolation through security measures like RBAC and admission control. https://fluxcd.io/flux/installation/configuration/multitenancy/
• Verify Flux Images: Ensures all Flux images used are signed by the Flux team. https://github.com/fluxcd/flux2-multi-tenancy/

Examples:

• clusters/staging/flux-system/gotk-components.yaml: clusters/staging/flux-system/gotk-components.yaml
• tenants/base/dev-team/sync.yaml: tenants/base/dev-team/sync.yaml
• scripts/validate.sh: scripts/validate.sh
• infrastructure/kyverno-policies/verify-flux-images.yaml: infrastructure/kyverno-policies/verify-flux-images.yaml
• clusters/production/flux-system/kustomization.yaml: clusters/production/flux-system/kustomization.yaml
• clusters/production/tenants.yaml: clusters/production/tenants.yaml
• infrastructure/kyverno/source.yaml: infrastructure/kyverno/source.yaml
• clusters/staging/tenants.yaml: clusters/staging/tenants.yaml
• clusters/production/infrastructure.yaml: clusters/production/infrastructure.yaml

Top-Level Directory Explanations

clusters/ - This directory contains configuration and scripts for managing Kubernetes clusters.

clusters/production/ - This directory contains configuration and scripts for managing the production Kubernetes cluster.

clusters/production/flux-system/ - This directory contains configuration and scripts for the FluxCD system in the production cluster.

clusters/staging/ - This directory contains configuration and scripts for managing the staging Kubernetes cluster.

clusters/staging/flux-system/ - This directory contains configuration and scripts for the FluxCD system in the staging cluster.

infrastructure/ - This directory contains infrastructure-related configuration files and scripts.

infrastructure/kyverno-policies/ - This directory contains the actual policy files for Kyverno.

infrastructure/kyverno/ - This directory contains configuration files and scripts for Kyverno, an open-source Kubernetes policy engine.

scripts/ - This directory contains scripts used for various tasks, such as automation and deployment.

tenants/ - This directory contains configuration and scripts for managing tenants, which are separate namespaces or projects within the Kubernetes cluster.

tenants/base/ - This directory contains configuration and scripts for the base tenant.

tenants/base/dev-team/ - This directory contains configuration and scripts for the development team within the base tenant.

tenants/production/ - This directory contains configuration and scripts for the production tenant.

tenants/staging/ - This directory contains configuration and scripts for the staging tenant.