fluxcd / flux2-kustomize-helm-example

Flux for Multi-Cluster Management

This repository demonstrates a multi-cluster setup leveraging Flux and Kustomize. It outlines the steps required to manage a staging and a production environment.

Motivation

The goal is to manage two clusters (staging and production) with minimal duplicated declarations using Flux and Kustomize. This ensures a seamless deployment process and simplifies the application lifecycle management.

Repository Structure

The repository is organized into three primary directories:

  • apps: This directory contains Helm releases with custom configurations for each cluster.
  • infrastructure: This directory contains common infra tools like ingress-nginx and cert-manager.
  • clusters: This directory holds the Flux configuration for each cluster.

Setting up Flux in Clusters

  1. Prerequisites:

    • Kubernetes cluster version 1.28 or newer.
    • A GitHub account.
    • A personal access token with “repo” permissions.
    • Flux CLI installed via Homebrew or downloaded binaries.

  2. Staging Cluster Setup:

    • Fork the repository.
    • Set environment variables for GitHub access token, username, and repository name.
    • Verify staging cluster prerequisites with flux check --pre.
    • Bootstrap Flux on the staging cluster. The command commits manifests for Flux components in clusters/staging/flux-system and creates a deploy key with read-only access on GitHub for pulling changes.
    flux bootstrap github \
          --context=staging \
          --owner=${GITHUB_USER} \
          --repository=${GITHUB_REPO} \
          --branch=main \
          --personal \
          --path=clusters/staging

  3. Production Cluster Setup:

    • Bootstrap Flux on the production cluster by modifying the context and path to your production cluster.
    • Monitor the production reconciliation using:
    $ flux get kustomizations --watch

Defining Flux Kustomization

The clusters directory contains Flux configurations for each cluster. For example, in the clusters/staging directory, the apps.yaml file defines the Kustomization definition:

apiVersion: kustomize.toolkit.fluxcd.io/v1
          kind: Kustomization
          metadata:
            name: apps
            namespace: flux-system
          spec:
            interval: 10m0s
            dependsOn:
              - name: infra-configs
            sourceRef:
              kind: GitRepository
              name: flux-system
            path: ./apps/staging
            prune: true
            wait: true
            timeout: 5m0s

This configuration instructs Flux to:

  • Sync the staging Kustomize overlay from the ./apps/staging path.
  • Depend on the infra-configs Kustomization, ensuring infrastructure items are created before deploying apps.

Managing Infrastructure

The infrastructure directory manages common infrastructure components:

  • controllers: Contains manifests for controllers like cert-manager and ingress-nginx.
  • configs: Includes configuration files for these controllers, potentially tailored for each cluster.

For example, in clusters/production/infrastructure.yaml, the dependsOn field ensures that controllers are installed before configs, guaranteeing CRDs are registered on the cluster before applying custom resources.

Deploying Applications with Helm

Flux supports Helm deployments using the HelmRelease custom resource. In the apps directory, you can define application deployments for each cluster.

For instance, in the apps/production directory, podinfo-values.yaml defines the Helm release:

apiVersion: helm.toolkit.fluxcd.io/v2
          kind: HelmRelease
          metadata:
            name: podinfo
            namespace: podinfo
          spec:
            chart:
              spec:
                version: ">=1.0.0"
            values:
              ingress:
                hosts:
                  - host: podinfo.production
                    paths:
                      - path: /
                        pathType: ImplementationSpecific

This configuration instructs Flux to:

  • Automatically upgrade the HelmRelease to the latest stable chart version (alpha, beta, and pre-releases are ignored).
  • Define an ingress host for the podinfo application.

CI/CD Workflow

This repository includes GitHub CI workflows:

  • test workflow: Validates Kubernetes manifests and Kustomize overlays using kubeconform.
  • e2e workflow: Starts a Kubernetes cluster in CI and tests the staging setup using Flux in Kubernetes Kind.

Adding New Clusters

To add a new cluster to your fleet:

  1. Clone the repository locally.
  2. Create a directory inside clusters with the new cluster name.
  3. Copy the sync manifests from the staging cluster to the new directory.
  4. Optionally create a new overlay for the cluster in the apps directory.
  5. Modify the spec.path in the new cluster’s apps.yaml file to point to the corresponding overlay.
  6. Push the changes to the main branch.
  7. Bootstrap Flux on the new cluster using the appropriate context and path.

Summary

This repository provides a comprehensive example of how to leverage Flux and Kustomize for multi-cluster management. By defining separate configurations for different environments and utilizing CI/CD workflows, developers can ensure consistent and automated deployments across their clusters.

Top-Level Directory Explanations

apps/ - This directory contains the application definitions and configurations. It includes base, production, staging directories.

apps/base/ - This directory contains the base application configurations. It includes podinfo directory.

apps/production/ - This directory contains the production application configurations.

apps/staging/ - This directory contains the staging application configurations.

clusters/ - This directory contains the Kubernetes cluster configurations. It includes production and staging directories.

clusters/production/ - This directory contains the production Kubernetes cluster configurations. It includes flux-system directory.

clusters/staging/ - This directory contains the staging Kubernetes cluster configurations. It includes flux-system directory.

infrastructure/ - This directory contains the infrastructure configurations and scripts.

infrastructure/configs/ - This directory contains the configuration files for the infrastructure.

infrastructure/controllers/ - This directory contains the Kubernetes controllers for managing the infrastructure.

scripts/ - This directory contains the scripts used for automating tasks.