Kustomize for Multi-Environment Deployments

Motivation: This example demonstrates how to leverage Kustomize and Flux to manage deployments for multiple environments (staging and production) while minimizing duplicate declarations.

Structure:

• apps directory: Contains Helm releases with custom configurations for each environment.
• infrastructure directory: Contains common infrastructure tools such as ingress-nginx and cert-manager.
• clusters directory: Contains Flux configuration for each cluster.

Kustomize Configuration

• apps/base/podinfo/kustomization.yaml:

  • Defines the base resources for the podinfo Helm release, including the namespace, repository, and release definition.
  • Source

• infrastructure/configs/kustomization.yaml:

  • Defines the resources for the ClusterIssuer which manages Let’s Encrypt certificates.
  • Source

• apps/production/kustomization.yaml:

  • Defines the Kustomization for the production environment of the podinfo Helm release.
  • Includes the base resources and applies a patch to the podinfo-values.yaml file.
  • Source

• apps/staging/kustomization.yaml:

  • Defines the Kustomization for the staging environment of the podinfo Helm release.
  • Includes the base resources and applies a patch to the podinfo-values.yaml file.
  • Source

• infrastructure/controllers/kustomization.yaml:

  • Defines the resources for the cert-manager and ingress-nginx controllers.
  • Source

• clusters/production/flux-system/kustomization.yaml:

  • Defines the Kustomization for Flux in the production cluster.
  • Includes the necessary Flux components and patches for custom configuration.
  • Source

• clusters/staging/flux-system/kustomization.yaml:

  • Defines the Kustomization for Flux in the staging cluster.
  • Includes the necessary Flux components and patches for custom configuration.
  • Source

Flux Configurations

• clusters/production/apps.yaml:

  • Defines the Kustomization for the production apps environment.
  • Specifies the source, path, and dependencies for the production deployment.
  • Source

• clusters/staging/apps.yaml:

  • Defines the Kustomization for the staging apps environment.
  • Specifies the source, path, and dependencies for the staging deployment.
  • Source

• clusters/production/infrastructure.yaml:

  • Defines the Kustomizations for the production infrastructure environment.
  • Specifies the source, path, dependencies, and a patch for the ClusterIssuer to use the production Let’s Encrypt API.
  • Source

• clusters/staging/infrastructure.yaml:

  • Defines the Kustomizations for the staging infrastructure environment.
  • Specifies the source, path, dependencies, and a patch for the ClusterIssuer to use the staging Let’s Encrypt API.
  • Source

Example Usage:

• Deploying to a new cluster:

  • Create a new directory in clusters for the new cluster.
  • Copy the infrastructure.yaml and apps.yaml files from the staging cluster.
  • Modify the spec.path in the apps.yaml file if necessary.
  • Bootstrap Flux to the new cluster using the flux bootstrap github command.

• Adding a new environment:

  • Create a new directory within apps for the new environment.
  • Copy the base resources from apps/base.
  • Apply any necessary patches specific to the new environment.
  • Update the spec.path in the corresponding clusters/environment/apps.yaml file.
  • Reconcile Flux to apply the changes.

• Cloning a cluster:

  • Bootstrap the new cluster using the flux bootstrap github command, specifying the path to the desired environment configuration.
  • Create a kustomization.yaml file within the new cluster directory, including the resources from the source environment.

Important Notes:

• The dependsOn field in the Kustomization manifests ensures that resources are deployed in the correct order.
• The patches field allows for environment-specific configuration changes without modifying the base resources.
• Flux continuously monitors the Git repository and automatically updates the cluster based on the defined Kustomizations.
• The scripts/validate.sh script validates the Kubernetes manifests and Kustomize overlays with kubeconform.

Top-Level Directory Explanations

apps/ - This directory contains the application definitions and configurations. It includes base, production, staging directories.

apps/base/ - This directory contains the base application configurations. It includes podinfo directory.

apps/production/ - This directory contains the production application configurations.

apps/staging/ - This directory contains the staging application configurations.

clusters/ - This directory contains the Kubernetes cluster configurations. It includes production and staging directories.

clusters/production/ - This directory contains the production Kubernetes cluster configurations. It includes flux-system directory.

clusters/staging/ - This directory contains the staging Kubernetes cluster configurations. It includes flux-system directory.

infrastructure/ - This directory contains the infrastructure configurations and scripts.

infrastructure/configs/ - This directory contains the configuration files for the infrastructure.

infrastructure/controllers/ - This directory contains the Kubernetes controllers for managing the infrastructure.

scripts/ - This directory contains the scripts used for automating tasks.