fluxcd / flux2-kustomize-helm-example

Kubernetes for https://github.com/fluxcd/flux2-kustomize-helm-example/

Motivation: The Foundation

This repository aims to provide a robust, scalable, and efficient way to manage multi-cluster Kubernetes deployments using Flux and Kustomize. The core goal is to leverage these tools to automate the deployment, testing, and upgrades of applications across different environments, while minimizing redundancy and maximizing consistency.

Workflows:

  • The test workflow within the GitHub CI ensures the validity of Kubernetes manifests and Kustomize overlays through kubeconform.
  • The e2e workflow sets up a Kubernetes cluster using Kind and tests the staging setup by running Flux within the CI environment.

Cluster Management:

  1. Adding a New Cluster:

    • Clone the Repository: git clone https://github.com/${GITHUB_USER}/${GITHUB_REPO}.git && cd ${GITHUB_REPO}
    • Create a Cluster Directory: mkdir -p clusters/dev
    • Copy Staging Manifests: cp clusters/staging/infrastructure.yaml clusters/dev
    • Copy Staging Manifests: cp clusters/staging/apps.yaml clusters/dev
    • Create a Custom Overlay: Create a ‘dev’ overlay within the apps directory and adjust the spec.path in clusters/dev/apps.yaml to point to the overlay, e.g., path: ./apps/dev.
    • Push Changes: git add -A && git commit -m "add dev cluster" && git push
    • Set Kubectl Context: Configure your kubectl context for the dev cluster.
    • Bootstrap Flux: flux bootstrap github --context=dev

  2. Production Deployment:

    • Configure Let’s Encrypt: Within the clusters/production/infrastructure.yaml file, update the Let’s Encrypt server value to point to the production API, e.g.:
    apiVersion: kustomize.toolkit.fluxcd.io/v1
          kind: Kustomization
          metadata:
          name: infra-configs
          namespace: flux-system
          spec:
          # ...omitted for brevity
          dependsOn:
          - name: infra-controllers
          patches:
          - patch: |
          - op: replace
          path: /spec/acme/server
          value: https://acme-v02.api.letsencrypt.org/directory
          target:
          kind: ClusterIssuer
          name: letsencrypt
    • Dependency Management: The dependsOn property in infrastructure.yaml ensures that Flux installs or upgrades the controllers before the configs. This ensures CRDs are registered on the cluster before Flux applies any custom resources.

Prerequisites:

  • Kubernetes Cluster: Requires a Kubernetes cluster version 1.28 or newer. You can use Kubernetes Kind for local testing.
  • GitHub Account: A GitHub account is required for managing the Git repository.
  • GitHub Personal Access Token: Requires a personal access token with repo permissions for creating repositories.
  • Flux CLI: Install the Flux CLI using Homebrew on macOS or Linux: brew install fluxcd/tap/flux or by downloading precompiled binaries: curl -s https://fluxcd.io/install.sh | sudo bash

Repository Structure:

├── apps
          │   ├── base
          │   ├── production
          │   └── staging
          ├── infrastructure
          │   ├── configs
          │   └── controllers
          └── clusters
          ├── production
          └── staging
  • apps: Contains Helm releases with cluster-specific configurations.
  • infrastructure: Contains common infrastructure tools like ingress-nginx and cert-manager.
  • clusters: Contains the Flux configuration for each cluster.

Application Management:

  • Helm Repository: Defines the Helm repository for cert-manager in infrastructure/controllers/cert-manager.yaml:

    apiVersion: source.toolkit.fluxcd.io/v1
          kind: HelmRepository
          metadata:
          name: cert-manager
          namespace: cert-manager
          spec:
          interval: 12h
          url: https://charts.jetstack.io
    • interval: 12h: Flux pulls the Helm repository index every 12 hours to check for updates.
    • version: "1.x": Defines the semver range for the Helm chart.

  • Let’s Encrypt Issuer: Creates a Let’s Encrypt issuer within the infrastructure/configs/ directory:

    apiVersion: cert-manager.io/v1
          kind: ClusterIssuer
          metadata:
          name: letsencrypt
          spec:
          acme:
          # Replace the email address with your own contact email
          email: 
          server: https://acme-staging-v02.api.letsencrypt.org/directory
          privateKeySecretRef:
          name: letsencrypt-nginx
          solvers:
          - http01:
          ingress:
          class: nginx

  • Helm Release: Defines the Helm release for cert-manager in infrastructure/controllers/cert-manager.yaml:

    apiVersion: helm.toolkit.fluxcd.io/v2
          kind: HelmRelease
          metadata:
          name: cert-manager
          namespace: cert-manager
          spec:
          interval: 30m
          chart:
          spec:
          chart: cert-manager
          version: "1.x"
          sourceRef:
          kind: HelmRepository
          name: cert-manager
    • interval: 30m: Flux checks the Helm chart for updates every 30 minutes.

  • Podinfo Demo App: The demo app utilizes a Helm release for deployment. You can verify its deployment and access it using the following steps:

    • Helm Release Definition: The apps/base/podinfo/release.yaml file contains the Helm release configuration for Podinfo.
    • Verification: Check the release status using the flux get helmreleases command.
    • Access via Ingress: Use kubectl -n ingress-nginx port-forward svc/ingress-nginx-controller 8080:80 & and access the app at http://localhost:8080.

Cluster Synchronization:

  1. Deploying to a New Cluster:

    • Bootstrap: flux bootstrap github --context=production-clone --owner=${GITHUB_USER} --repository=${GITHUB_REPO} --branch=main --personal --path=clusters/production-clone
    • Pull Changes: git pull origin main
    • Create Kustomization:
    apiVersion: kustomize.config.k8s.io/v1beta1
          kind: Kustomization
          resources:
          - flux-system
          - ../production/infrastructure.yaml
          - ../production/apps.yaml
    • Reconcile: flux reconcile kustomization flux-system --context=production-clone --with-source

  2. Production Cluster:

    • Bootstrap: flux bootstrap github --context=production --owner=${GITHUB_USER} --repository=${GITHUB_REPO} --branch=main --personal --path=clusters/production
    • Watch Reconciliation: flux get kustomizations --watch

Infrastructure Organization:

  • infrastructure/controllers: Contains namespaces and Helm release definitions for Kubernetes controllers.
  • infrastructure/configs: Contains Kubernetes custom resources, such as cert issuers and network policies.
  • HelmRelease Definitions: infrastructure/controllers/cert-manager.yaml and infrastructure/controllers/ingress-nginx.yaml contain the Helm release definitions for various controllers.

Configuration:

  • flux-system Kustomization: The clusters/production/flux-system/kustomization.yaml and clusters/staging/flux-system/kustomization.yaml files define the kustomization for the flux-system namespace. These files include various patches for customizing the Flux components.

  • Infrastructure Kustomizations:

    • infrastructure/configs/kustomization.yaml: Defines the kustomization for the infrastructure configurations.
    • infrastructure/controllers/kustomization.yaml: Defines the kustomization for the infrastructure controllers.

  • App Kustomizations:

    • apps/staging/kustomization.yaml: Defines the kustomization for the staging environment.
    • apps/production/kustomization.yaml: Defines the kustomization for the production environment.
    • apps/base/podinfo/kustomization.yaml: Defines the kustomization for the Podinfo application.

  • Ingress-nginx Definition: The infrastructure/controllers/ingress-nginx.yaml file defines the deployment of ingress-nginx using a Helm release.

  • Cluster-Specific Kustomizations: The clusters/production/apps.yaml and clusters/staging/apps.yaml files define the kustomizations for the apps namespace in each cluster. These files include dependencies on the infrastructure components.

  • Production Infrastructure: The clusters/production/infrastructure.yaml file defines the kustomization for the infrastructure configurations. This file includes the Let’s Encrypt configuration.

  • Staging Infrastructure: The clusters/staging/infrastructure.yaml file defines the kustomization for the infrastructure configurations. This file includes the staging Let’s Encrypt configuration.

  • Podinfo Values: The apps/staging/podinfo-values.yaml and apps/production/podinfo-values.yaml files define the values for the Podinfo Helm release in the respective environments.

Key Concepts:

  • GitOps: This repository implements GitOps principles, where the infrastructure and application configurations are stored in Git and changes are propagated to the cluster through reconciliation.
  • Flux: Flux is a GitOps operator that automatically deploys, updates, and manages Kubernetes applications based on the configurations in Git.
  • Kustomize: Kustomize provides a mechanism to customize and apply configurations to multiple environments, reducing redundancy and promoting consistency.
  • Helm: Helm is a package manager for Kubernetes, used to manage the deployment of applications in a structured and reproducible manner.
  • Let’s Encrypt: Let’s Encrypt is used for obtaining and managing TLS certificates for the applications.

Top-Level Directory Explanations

apps/ - This directory contains the application definitions and configurations. It includes base, production, staging directories.

apps/base/ - This directory contains the base application configurations. It includes podinfo directory.

apps/production/ - This directory contains the production application configurations.

apps/staging/ - This directory contains the staging application configurations.

clusters/ - This directory contains the Kubernetes cluster configurations. It includes production and staging directories.

clusters/production/ - This directory contains the production Kubernetes cluster configurations. It includes flux-system directory.

clusters/staging/ - This directory contains the staging Kubernetes cluster configurations. It includes flux-system directory.

infrastructure/ - This directory contains the infrastructure configurations and scripts.

infrastructure/configs/ - This directory contains the configuration files for the infrastructure.

infrastructure/controllers/ - This directory contains the Kubernetes controllers for managing the infrastructure.

scripts/ - This directory contains the scripts used for automating tasks.