fluxcd / flux2-kustomize-helm-example

Multi-Cluster Deployment for Flux2 Kustomize Helm Example

This repository demonstrates how to use Flux and Kustomize to manage multiple Kubernetes clusters with minimal duplication of configurations.

Motivations:

  • Scaling Applications Across Multiple Environments: Deploy and manage applications consistently across different environments (e.g., staging, production) with a centralized approach.

Prerequisites:

  • Kubernetes Cluster: Version 1.28 or newer (e.g., Kubernetes Kind). README.md
  • GitHub Account: With a personal access token that can create repositories (all permissions under repo). README.md
  • Flux CLI: Install on macOS or Linux using Homebrew or by downloading precompiled binaries. README.md

Deployment Steps:

1. Repository Setup:

  • Fork the repository on your personal GitHub account. README.md
  • Export your GitHub access token, username, and repository name. README.md

2. Cluster Configuration:

  • Create a directory for each cluster inside the clusters directory. README.md
  • Copy the infrastructure.yaml and apps.yaml manifests from the staging directory into the new cluster directory. README.md
  • Optionally create a cluster-specific overlay in the apps directory. README.md
  • Change the spec.path in the clusters/<cluster-name>/apps.yaml to point to the correct overlay directory. README.md
  • Push the changes to the main branch. README.md

3. Flux Bootstrap:

  • Set the kubectl context and path to your cluster. README.md
  • Bootstrap Flux using the flux bootstrap github command, specifying the required parameters (context, owner, repository, branch, path). README.md
  • The bootstrap command commits the manifests for Flux components in clusters/<cluster-name>/flux-system and creates a deploy key with read-only access on GitHub. README.md

4. Cluster Reconciliation:

  • Tell Flux to deploy the workloads on the cluster using the flux reconcile kustomization command. README.md

5. CI/CD Validation:

  • Validate changes to the Kubernetes manifests and repository structure in CI before merging pull requests. README.md

Example Scenario (Staging and Production):

  • Directory Structure: The repository includes the following directories:

    • apps: Contains Helm releases with custom configuration per cluster.
    • infrastructure: Contains common infrastructure tools (e.g., ingress-nginx, cert-manager).
    • clusters: Contains the Flux configuration per cluster. README.md

  • Flux Kustomization Definitions: The clusters/<cluster-name> directories contain Flux Kustomization definitions. README.md

  • Staging Cluster:

    • The staging directory includes apps.yaml and infrastructure.yaml. README.md
    • The infrastructure.yaml configures Let’s Encrypt with a staging API server. README.md

  • Production Cluster:

    • The production directory also includes apps.yaml and infrastructure.yaml. README.md
    • The infrastructure.yaml configures Let’s Encrypt with the production API server. README.md

  • Application Configuration:

    • The apps directory contains base, production, and staging subdirectories. README.md
    • The base subdirectory contains common Helm release definitions for both clusters. README.md
    • The production and staging subdirectories contain cluster-specific Helm release values. README.md

CI Workflows:

  • The repository includes GitHub CI workflows for:
    • Test workflow: Validates Kubernetes manifests and Kustomize overlays using kubeconform. README.md
    • E2E workflow: Starts a Kubernetes cluster in CI and tests the staging setup by running Flux in Kubernetes Kind. README.md

Code Examples:

  • clusters/staging/infrastructure.yaml: Configures Let’s Encrypt with the staging API server. README.md
  • clusters/production/infrastructure.yaml: Configures Let’s Encrypt with the production API server. README.md
  • clusters/staging/apps.yaml: Defines the staging cluster Kustomization with dependencies and overlays. README.md
  • clusters/production/apps.yaml: Defines the production cluster Kustomization with dependencies and overlays. README.md
  • apps/base/podinfo/kustomization.yaml: Contains common Helm release definitions for the podinfo application. README.md
  • apps/staging/kustomization.yaml: Contains staging-specific values for the podinfo application. README.md
  • apps/production/kustomization.yaml: Contains production-specific values for the podinfo application. README.md

Key Features:

  • Centralized Configuration: Manage multiple clusters through a single Git repository.
  • Kustomize for Overlays: Leverage Kustomize for cluster-specific overrides and configurations.
  • Flux Integration: Use Flux for automatic deployment, updates, and reconciliation.
  • CI/CD Integration: Automated testing and validation for deployments.

Additional Information:

Top-Level Directory Explanations

apps/ - This directory contains the application definitions and configurations. It includes base, production, staging directories.

apps/base/ - This directory contains the base application configurations. It includes podinfo directory.

apps/production/ - This directory contains the production application configurations.

apps/staging/ - This directory contains the staging application configurations.

clusters/ - This directory contains the Kubernetes cluster configurations. It includes production and staging directories.

clusters/production/ - This directory contains the production Kubernetes cluster configurations. It includes flux-system directory.

clusters/staging/ - This directory contains the staging Kubernetes cluster configurations. It includes flux-system directory.

infrastructure/ - This directory contains the infrastructure configurations and scripts.

infrastructure/configs/ - This directory contains the configuration files for the infrastructure.

infrastructure/controllers/ - This directory contains the Kubernetes controllers for managing the infrastructure.

scripts/ - This directory contains the scripts used for automating tasks.